• Why compare models?
• Residuals, explained variation, and \(R^2\)
• Why plain \(R^2\) can reward overfitting
• Nested models and anova() comparisons
• Coefficients before overall tests
• Interactions and multi-level predictors
• Using emmeans to compare levels
• Choosing and reporting a selected model

Models are alternative explanations of the data.

Model comparison helps us ask whether a more complex model is justified.

For example:

• does adding sex improve the age model?
• does adding smoker status improve the model further?
• are we explaining meaningful variation, or just adding complexity?

The aim is not to find the biggest model.

The aim is to choose one model that is useful, justified, and interpretable.

A single coefficient can be easy to interpret. For example, an age slope tells us how much reaction time changes for one year of age.

But coefficient interpretation becomes harder with:

• categorical predictors with more than two levels
• interactions
• more complex model terms

Model comparison asks whether the whole term improves the model:

• does smoker status improve the model overall?
• does the age-by-smoker interaction improve the model overall?

We interpret relevant coefficients, predictions, or emmeans comparisons.

For anova() comparisons of linear models:

• models must use the same outcome
• models must use the same rows of data
• models must be nested
• added predictors should be theoretically plausible
• the usual linear-model assumptions still matter

We focus on sums of squares: total, explained, and unexplained variation.

Likelihood / deviance become more important for binomial and count models.

A residual is the difference between an observed value and the value predicted by the model.

model_age <- lm(rt ~ age, data = blomkvist)
residuals(model_age)

     1      2      3      4      5      6 
 -97.9  -57.6  -33.9  -97.4 -128.8 -148.2 

Residuals close to zero mean the model predictions are close to the observed values.

Residuals far from zero mean the prediction errors are larger.

To measure unexplained variation, we square the residuals and add them up.

The official name is the residual sum of squares, or RSS.

\[ \text{unexplained variation} = \sum \text{residuals}^2 \]

sum(residuals(model_age)^2)

[1] 5880557

This is the amount of variation the model has not explained.

We can split variation in the outcome into three ideas:

• total variation, or TSS: how much observations vary around the mean
• unexplained variation, or RSS: what remains in the residuals
• explained variation, or ESS: what the model accounts for

\[ \text{TSS} = \text{ESS} + \text{RSS} \]

A useful model should reduce unexplained variation and increase explained variation.

\(R^2\) describes the proportion of total variation explained by the model.

\[ R^2 = \frac{\text{explained variation}}{\text{total variation}} \]

In R, deviance() gives us the residual sum of squares for an lm() model. So here, deviance is the unexplained variation, or RSS.

\(R^2\) is useful because it tells us how much variation the model explains.

Two models are nested when the simpler model is contained inside the more complex model.

model_0 <- lm(rt ~ 1, data = blomkvist)
model_1 <- lm(rt ~ age, data = blomkvist)
model_2 <- lm(rt ~ age + smoker, data = blomkvist)
model_3 <- lm(rt ~ age * smoker, data = blomkvist)

Each model adds something to the model before it.

Nested:

• rt ~ age compared with rt ~ age + sex
• rt ~ age + smoker compared with rt ~ age * smoker
• rt ~ 1 compared with rt ~ age

Not nested:

• rt ~ age compared with rt ~ sex
• rt ~ age + sex compared with rt ~ age + smoker

For anova(), the simpler model must be contained in the more complex model.

Nested model comparison asks whether the more complex model reduces unexplained variation enough to justify the added term.

extra_explained is the extra variation explained by moving to the more complex model.

Each row compares a more complex model with the model just above it.

In R’s table:

• RSS is unexplained variation
• Sum of Sq is extra explained variation from the added predictor
• Sum of Sq = previous RSS - new RSS

Focus on:

• which predictor was added
• how much unexplained variation decreased
• the F statistic for that added predictor
• the p-value for that comparison
• whether the more complex model is worth reporting

A common mistake is to present full results from every model that was fitted.

For most assessment reports, use this order:

1. Fit models that match plausible hypotheses.
2. Compare those models.
3. Select the best-supported final model.
4. Interpret coefficients and predictions from the selected model.

The comparison explains the choice. If an added predictor improved fit, report the F-test result briefly, for example: adding smoker status improved fit compared with the age-only model, \(F\)(2, 261) = 4.05, \(p\) = .019. The detailed interpretation belongs to the selected model.

A useful paragraph says:

• which models were compared
• what predictors were added
• what the comparison showed
• the F-test result when a model improved fit
• which model was selected
• why that model was selected

Model comparison helps choose between plausible models.

It does not prove that the selected model is appropriate.

After selecting a normal linear model, we still need residual checks for:

• linearity
• equality of variance
• approximate normality of residuals
• unusual observations

We will practise these checks in the next session.

Choose one transfer file:

• part-1-transfer-own-data-model-comparison.Rmd if you have your own data
• part-1-transfer-chinese-model-comparison.Rmd if you do not yet have your own data

Use this time to:

• practise one nested model comparison
• check that the comparison is valid
• draft model comparison text
• note which residual checks the selected model will need