(Non)linearity

Licensed under the Creative Commons attribution-noncommercial license. Please share & remix noncommercially, mentioning its origin.

What is a “nonlinear” model?

• a model whose objective function (usu. likelihood) cannot be expressed as a linear function of predictors (\(\beta_0 + \beta_1 x_1 + \beta_2 x_2\) …)

What’s so good about linear models?

• solution by linear algebra, rather than iteration
• generalized linear models - iterative linear algebra (IRLS: @kane_glms_2016, @arnold_computational_2019)
• guarantees:
  • performance
  • convergence
  • unimodal, log-quadratic likelihoods
• no hyperparameter tuning
• no tests for convergence
• no need to pick starting values
• robust to parameter scaling and correlation
• interpretability
• convenient framework for covariates/interactions/etc.

what does “linear” include?

• additive models (splines for general smooth continuous functions)
• linearization tricks
  • e.g. generalized Ricker: \(y = a^b x e^{cx} \to \log(y) = \log(a) + b \log(x) + c x\)
  • Michaelis-Menten: \(y = ax/(b+x) \to 1/y = (b/a) (1/x) + (1/a)\)
  • power-law (allometric) fitting: \(y=ax^b \to \log(y) = a + b \log(x)\)
• borderline:
  • generalized least squares
  • generalized linear models (link function)
  • (G)L mixed models

Nonlinear models

• everything else

• “nonlinear” \(\leftrightarrow\) most zoology = the study of non-elephant animals (Ulam)
• many methods use locally linear approaches (gradient descent; Newton)

• today’s class focuses on less-canned/one-off solutions

Simple examples

• power-law curves (if additive: \(y=ax^b\) can be linearized)
• logistic curves, unknown asymptote (\(K\), or prob <1), and generalizations (e.g. Richards equation)
• simple movement models (log-Normal step length/von Mises turning angle)
• saturating curves (functional responses: Holling/Rogers equation)

Harder examples

• trajectory-matching ODEs
• seed shadow models
• mixture models

Special-case toolboxes

Many classes of problems have special algorithms that linearize or otherwise solve most of the problem, then solve a low-dimensional nonlinear optimization problem

• Kalman filtering
• hidden Markov models (Viterbi algorithm)
• linear mixed models
• generalized linear (mixed) models (IRLS/penalized IRLS+Laplace/quadrature)
• generalized least-squares
• mixture models via expectation-maximization
• capture-mark-recapture data
• machine learning (neural networks, random forests, …)

General advice

• simplify problem expression
• use most efficient algorithms
• good starting values (more later)