Priors

Includes material from Ian Dworkin and Jonathan Dushoff, but they bear no responsibility for the contents.

library(brms)
library(lme4)
library(broom.mixed)
library(tidybayes)

baseline prior choices

See distribution explorer (backend)

real-valued parameters: Gaussian (or Student-\(t\))
variances: half-Student-t on the standard deviation scale or log-Normal
other positive-valued: Gamma or log-Normal
- older: inverse-Gamma for variances, for conjugacy
bounded: Beta
correlation/covariance matrices
- LKJ or “onion” (Lewandowski, Kurowicka, and Joe 2009). \(\eta=1\) \(\to\) uniform, \(\eta>1\) favours a stronger diagonal (smaller correlations)
- Wishart [older] (‘mean’ = typical corr matrix, usually diagonal; shape ‘nu’ determines variance [1 is equivalent to an exponential dist for 1-dim, so <1 is ‘weak’]

independent vs multivariate priors

we usually set priors on parameters one at a time
this assumes they’re independent (!)
hard to do very much about this, but keep it in mind
means that reparameterizing a model (e.g. by centering a variable) can change the priors, if we’re not careful

prior problems

scale dependence
improper priors (scale dependence, identifiability/computational issues)

issues with proper uniform priors

Cromwell’s rule (Lindley 1980): don’t exclude possible values

squashing problems

‘wide’ on one scale \(\neq\) noninformative on a transformed scale
especially for intervals, e.g. probabilities
for positive distributions, uninformative often leads to a spike at zero (inverse-Gamma)

stacking problems

when uniform priors that are supposed to be uninformative are actually informative … (Carpenter 2017)
posterior piles up at the edge

priors for variances

people used to use inverse-Gamma a lot (because conjugate): uninformative/wide → big spike at zero (Gelman 2006)
half-Normal, half-\(t\) are better
- parameter-expanded priors in MCMCglmm Gelman and Hill (2006)
regularizing priors (Chung et al. 2013) prevent ‘singularity’
- singularity doesn’t really matter when fully Bayesian (not MAP!) because we integrate over the whole posterior

priors for covariance/correlation matrices

a single correlation is easy (e.g. a Beta distribution)
multivariate correlation/covariance matrix is hard (Wishart)
separate priors for (log)-SDs, correlations
- LKJ or “onion” (Lewandowski, Kurowicka, and Joe 2009)

choosing priors

Singmann et al. (2023) (“using priors over inherently meaningful units instead of default priors on standardized scales”)
unitless (log, logit-transformed, standardized) parameters make it easier to choose (Schielzeth 2010)
choose a ‘wide range’ as e.g. \(\pm 2\sigma\) or \(\pm 3\sigma\) of a Normal (or \(t\)) prior: Inchausti (2023), Wan et al. (2014)
use prior information but dilute it: Ibrahim et al. (2015)

prior sensitivity

compare results with different priors: Crome, Thomas, and Moore (1996)
more general refs: Banner, Irvine, and Rodhouse (2020); ; Edwards (1996); Nicenboim, Schad, and Vasishth (n.d.) (chap 3); Gelman et al. (2020); Xie and Carlin (2006); Kallioinen et al. (2022); Fink (1997); Sarma and Kay (2020)
lots here (kind of a grab bag): https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations
https://www.monicaalexander.com/posts/2020-28-02-bayes_viz/

definitions (from Banner, Irvine, and Rodhouse (2020))

type	definition
default	Commonly used non-informative priors that are often left unjustified by the user. Examples include, normal priors for regression coefficients with variances as large as \(10^6\), Uniform(0,1) on probabilities or proportions, and other ‘non-informative’ priors used without justification in software tutorials (e.g. WinBugs manual)
vague, flat, diffuse	A non-informative prior that is used to reflect the prior knowledge that not much is known about the parameter of interest, but is well justified and hyper-parameter values are set to reflect a reasonable range of values for the parameter in the context of the problem.
Jeffreys’	A prior for a single-parameter that, when the parameter is transformed to a different scale (via a 1:1 transformation), the resulting prior for the transformed parameter is exactly the same as the prior for the parameter on the original scale. This approach was introduced by Jeffreys (Jeffreys, 1946), and is often used to define a non-informative prior for a single-parameter that is invariant to transformations, or scale-invariant
weakly informative	Often refers to prior distributions that are used to reflect a diluted (or scaled back) amount of knowledge about the parameters
regularizing	A type of weakly informative prior that is meant to constrain the parameter space to help with estimation of the posterior distribution. Examples include \(N(0, \sigma^2 = 2)\) priors on logistic regression coefficients, and shrinkage priors when the number of predictors is greater than the sample size (i.e. \(p > n\) problems)
informative	A prior that is carefully designed to reflect the current knowledge (and uncertainty) of the parameter.

Bayesian workflow

This figure from Gelman et al. (2020) is a little overwhelming. It is more targeted toward people who are developing new Bayesian models from scratch rather than using a platform like brms but can still be useful.

References

Banner, Katharine M., Kathryn M. Irvine, and Thomas J. Rodhouse. 2020. “The Use of Bayesian Priors in Ecology: The Good, the Bad and the Not Great.” Methods in Ecology and Evolution 11 (8): 882–89. https://doi.org/10.1111/2041-210X.13407.

Carpenter, Bob. 2017. “Computational and Statistical Issues with Uniform Interval Priors.” Statistical Modeling, Causal Inference, and Social Science. http://andrewgelman.com/2017/11/28/computational-statistical-issues-uniform-interval-priors/.

Chung, Yeojin, Sophia Rabe-Hesketh, Vincent Dorie, Andrew Gelman, and Jingchen Liu. 2013. “A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models.” Psychometrika 78 (4): 685–709. https://doi.org/10.1007/s11336-013-9328-2.

Crome, F. H. J., M. R. Thomas, and L. A. Moore. 1996. “A Novel Bayesian Approach to Assessing Impacts of Rain Forest Logging.” Ecological Applications 6: 1104–23.

Edwards, Don. 1996. “Comment: The First Data Analysis Should Be Journalistic.” Ecological Applications 6 (4): 1090–94.

Fink, Daniel. 1997. “A Compendium of Conjugate Priors.” https://web.archive.org/web/20090529203101/http://www.people.cornell.edu/pages/df36/CONJINTRnew%20TEX.pdf.

Gelman, Andrew. 2006. “Prior Distributions for Variance Parameters in Hierarchical Models (Comment on Article by Browne and Draper).” Bayesian Analysis 1 (3): 515–34. https://doi.org/10.1214/06-BA117A.

Gelman, Andrew, and Jennifer Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge, England: Cambridge University Press.

Gelman, Andrew, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. 2020. “Bayesian Workflow.” arXiv:2011.01808 [Stat], November. http://arxiv.org/abs/2011.01808.

Ibrahim, Joseph G., Ming-Hui Chen, Yeongjin Gwon, and Fang Chen. 2015. “The Power Prior: Theory and Applications.” Statistics in Medicine 34 (28): 3724–49. https://doi.org/10.1002/sim.6728.

Inchausti, Pablo. 2023. Statistical Modeling With R: A Dual Frequentist and Bayesian Approach for Life Scientists. Oxford University Press.

Kallioinen, Noa, Topi Paananen, Paul-Christian Bürkner, and Aki Vehtari. 2022. “Detecting and Diagnosing Prior and Likelihood Sensitivity with Power-Scaling.” arXiv. https://doi.org/10.48550/arXiv.2107.14054.

Lewandowski, Daniel, Dorota Kurowicka, and Harry Joe. 2009. “Generating Random Correlation Matrices Based on Vines and Extended Onion Method.” Journal of Multivariate Analysis 100 (9): 1989–2001. https://doi.org/10.1016/j.jmva.2009.04.008.

Lindley, Dennis V. 1980. “The Bayesian Approach to Statistics.” ORC 80-9. Operations Research Center, University of California, Berkeley. https://apps.dtic.mil/sti/pdfs/ADA087836.pdf.

Nicenboim, Bruno, Daniel Schad, and Shravan Vasishth. n.d. An Introduction to Bayesian Data Analysis for Cognitive Science. https://vasishth.github.io/bayescogsci/book/.

Sarma, Abhraneel, and Matthew Kay. 2020. “Prior Setting in Practice: Strategies and Rationales Used in Choosing Prior Distributions for Bayesian Analysis.” In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. CHI ’20. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3313831.3376377.

Schielzeth, Holger. 2010. “Simple Means to Improve the Interpretability of Regression Coefficients: Interpretation of Regression Coefficients.” Methods in Ecology and Evolution 1 (2): 103–13. https://doi.org/10.1111/j.2041-210X.2010.00012.x.

Singmann, Henrik, David Kellen, Gregory E. Cox, Suyog H. Chandramouli, Clintin P. Davis-Stober, John C. Dunn, Quentin F. Gronau, et al. 2023. “Statistics in the Service of Science: Don’t Let the Tail Wag the Dog.” Computational Brain & Behavior 6 (1): 64–83. https://doi.org/10.1007/s42113-022-00129-2.

Wan, Xiang, Wenqian Wang, Jiming Liu, and Tiejun Tong. 2014. “Estimating the Sample Mean and Standard Deviation from the Sample Size, Median, Range and/or Interquartile Range.” BMC Medical Research Methodology 14 (1): 135. https://doi.org/10.1186/1471-2288-14-135.

Xie, Yang, and Bradley P. Carlin. 2006. “Measures of Bayesian Learning and Identifiability in Hierarchical Models.” Journal of Statistical Planning and Inference 136 (10): 3458–77. https://doi.org/10.1016/j.jspi.2005.04.003.

Last updated: 14 March 2024 19:33