• what is the scope of this talk?
• linear fixed effects (possibly with a link function)
• linear random effects/latent variables (ditto)
• known clusters with independent sets of latent variables
• multivariate Normal Gaussian distribution of latent variables

\[ \begin{split} \underbrace{Y_i}_{\text{response}} & \sim \overbrace{\text{Distr}}^{\substack{\text{conditional} \\ \text{distribution}}}(\underbrace{g^{-1}(\eta_i)}_{\substack{\text{inverse} \\ \text{link} \\ \text{function}}},\underbrace{\phi}_{\substack{\text{scale} \\ \text{parameter}}}) \\ \underbrace{\boldsymbol \eta}_{\substack{\text{linear} \\ \text{predictor}}} & = \underbrace{\boldsymbol X\boldsymbol \beta}_{\substack{\text{fixed} \\ \text{effects}}} + \underbrace{\boldsymbol Z\boldsymbol b}_{\substack{\text{random} \\ \text{effects}}} \\ \underbrace{\boldsymbol b}_{\substack{\text{conditional} \\ \text{modes}}} & \sim \text{MVN}(\boldsymbol 0, \underbrace{\Sigma(\boldsymbol \theta)}_{\substack{\text{covariance} \\ \text{matrix}}}) \end{split} \]

… possibly allow zero-inflation/hurdle component, dispersion (scale) model (e.g. \(\phi = \exp(\boldsymbol X_d \beta_d (? + \boldsymbol Z_d \boldsymbol b_d ?))\))

depends on:

• data size
• computation time/cost, memory requirements
• availability of reliable software
• complexity of the statistical model
• ANOVA → linear models → GLMs → mixed models → ???

• “goods … that are distinguishable in terms of attributes (uniqueness or brand) end up becoming simple commodities in the eyes of the market or consumers” (Wikipedia)
• basic components become part of a stack/utilities for higher-level features
• operating system → linear algebra packages (BLAS, LAPACK) → regression tools → generalized linear models → mixed models → … ?
• build on top of well-tested frameworks

• as long as people are educated enough
• ¿can we afford enough statistical consultants?
• Efron: “recommending that scientists use [X] is like giving the neighborhood kids the key to your F-16” (Gelman 2008)
  (X = “Bayes’ Theorem” here but could be “mixed models”?)
• make easy things easy and hard things possible
  (and warn about inadvisable things?)
• importance of sensible defaults

• limited set of requirements
• R (“a free software environment”): also SAS …
• Wilkinson-Rogers-Bates formulas (Wilkinson et al. 1973)
  • initial target: designed experiments
  • GENOTYPE*POL(SITE, 1,SITEMEAN)*POLND(DENSITY)
  • GENSTAT, expanded in S/R (Chambers et al. 1992)
  • expanded further to include random effects (Pinheiro et al. 2000; Tanaka et al. 2019)
  • further expansions … (brms package etc.)
• too compact (McElreath 2015)?
  who really knows what y ~ x1*x2 + (x1*x2|g) means?

• Mixed models task view: currently mentions 169 packages …

Open source software is:

• powerful
• equitable
• reproducible
• transparent
• cutting-edge
• cheap!

from sandserifcomics

• emphasis on algorithms and scalability
• elevate prediction over inference (Breiman 2001)
• (modern) nonparametric models
• automatic differentiation

• METIS/SuiteSparse/CHOLMOD
  (Davis, Karypis: used in lme4, TMB)
• INLA meshes (Lindgren, Rue: INLA, sdmTMB)
• spline basis construction (Wood/mgcv)
• autodifferentiation

• magic algorithm: automatic gradients with cost <5\(\times\) cost of objective function (Griewank et al. 2003)
• CppAD, TMB (Kristensen), Stan (Carpenter/Betancourt), PyTorch, Tensorflow (greta), NIMBLE (de Valpine) (Valpine et al. 2017)
• memory-intensive

• any ‘parameter’ of a model is an expectation of a change in the response
  • partial dependence (marginal effects), conditional dependence:
• users are almost always interested in counterfactuals
• … and estimates of uncertainty (or they should be)
• uncertainty in a counterfactual effect == inference ???
• connections to causal inference

• Revisit this layer: \(\boldsymbol b\sim \text{MVN}(\boldsymbol 0, \Sigma(\boldsymbol \theta))\)
• if \(\Sigma\) is diagonal then this is equivalent to putting a ridge penalty (\(\boldsymbol b\boldsymbol b^\top = \sum b_i^2\)) on the conditional modes
• we could use structured covariance matrices (compound symmetry, AR1, etc.)
• … penalizing by \(\boldsymbol b\Sigma \boldsymbol b^T\) is equivalent to using a smoothing matrix \(\Sigma^{-1}\)
• Gaussian processes, splines, Markov random fields …
• mgcv package provides implementations that can be used anywhere (Wood 2017)
  • gamm4, glmmTMB, sdmTMB, brms …

• dimensions of a mixed model
  • number of observations (\(10^2\) - \(10^8\))
  • number of latent variables (\(10\) - \(10^4\))
  • number of clusters (\(10\) - \(10^4\))
  • number of top-level parameters (\(\beta\), \(\theta\)): \(10\) - \(1000\)
• MLE scaling typically \({\cal O}(N^{3/2})\); method-of-moments \({\cal O}(N)\)
  (stochastic gradient descent???)
• Recent work on scalable MM (Hillary M. Heiling et al. 2024a; Hillary M. Heiling et al. 2024b; Gao et al. 2017; Gao et al. 2020; Ghosh et al. 2022a; Ghosh et al. 2022b; Ghandwani et al. 2023)

• linear algebra (decompositions) (Bates et al. 2015)
• EM algorithm
• brute-force (RE)ML with autodiff (Brooks et al. 2017)
• MCEM
• ??

• computational trick or statement about the world?
• combining mixed models with sparsity-inducing penalization (e.g. glmmLasso)
• INLA meshes leverage conditional independence of spatial points to handle spatial problems efficiently (sdmTMB)

• mgcv smooths use rank reduction techniques instead of relying on sparsity
• factor-analytic models: glmmTMB, gllvm

• regularization/priors
  • Bayes: justified by belief
  • frequentism: justified by
    • decreased bias (e.g. Firth logistic regression)
    • decreased error (Stein’s paradox, ridge regression)
• empirical Bayes: regularization/priors only on intermediate layers
• maximum a posteriori (MAP) estimation: INLA, blme
• for good confidence intervals, MCMC is faster than parametric bootstrapping (e.g. pharmacokinetic models)

• concern about subjectivity, ease of use (Efron 1986)
• Bayes is still slower