• From model choice to model checks
• Residuals: observed minus predicted
• Residual distribution
• Residuals against predicted values
• Residuals against predictors
• Predictor overlap and VIF
• Writing up model checks
• Formative report support

In Part 1, model comparison helped us choose between nested models.

Today we ask a different question:

Model checks do not prove that a model is correct.

They help us spot problems that would make the model hard to trust.

For a normal linear model, useful checks include:

• are residuals roughly centred around zero?
• are residuals very skewed?
• do residuals show patterns across predicted values?
• do residuals show patterns across predictors?
• do predictors overlap too strongly with each other?

The first four checks use residuals.

The last one uses VIF.

A residual is the difference between an observed value and the value predicted by the model.

\[ \text{residual} = \text{observed value} - \text{predicted value} \]

model_rt <- lm(rt ~ age + sex, data = blomkvist)

model_check_data <- mutate(blomkvist,
                           predicted = predict(model_rt),
                           residual = rt - predicted)

head(model_check_data)

# A tibble: 6 × 7
     id   age sex    smoker    rt predicted residual
  <dbl> <dbl> <chr>  <chr>  <dbl>     <dbl>    <dbl>
1     1    84 male   former  702.      762.    -60.8
2     2    37 female no      471.      552.    -81.3
3     3    62 female yes     639.      695.    -56.8
4     4    85 female former  708       827.   -119. 
5     5    73 male   former  607.      699.    -92.0
6     6    65 male   no      542.      653.   -112. 

A model makes predictions.

Residuals show where those predictions went wrong.

If the residuals look random and relatively well behaved, that supports the model.

If residuals show strong patterns, the model may be missing something important.

A histogram shows whether residuals are roughly centred on zero and whether the distribution is very asymmetric.

ggplot(model_check_data, aes(x = residual)) +
  geom_histogram(bins = 30) +
  geom_vline(xintercept = 0, linetype = "dashed") +
  labs(x = "Residual", y = "Count")

Large skew can be a warning sign.

Skew describes asymmetry.

Values close to zero suggest little skew. Larger positive or negative values suggest stronger asymmetry.

residual_skew

[1] 1.85

A skew value is not a pass/fail test.

It is one summary to use alongside the plot.

This plot checks whether residuals show a pattern across predicted values.

ggplot(model_check_data, aes(x = predicted, y = residual)) +
  geom_point(alpha = .35) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(x = "Predicted value", y = "Residual")

Look for a cloud of points around zero, without a strong curve or funnel shape.

This plot checks whether residuals show a pattern across age.

ggplot(model_check_data, aes(x = age, y = residual)) +
  geom_point(alpha = .35) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(x = "Age", y = "Residual")

A strong pattern would suggest that the age effect has not been captured well.

When a model has more than one predictor, predictors can overlap.

For example:

• age may relate to reaction time
• sex may relate to reaction time
• age and sex might also be related to each other in the sample

If predictors overlap strongly, the model has a harder time estimating their separate effects.

VIF stands for variance inflation factor.

VIF checks how much a predictor is explained by the other predictors.

car::vif(model_rt)

age sex 
  1   1 

Rough guide:

• close to 1: little concern
• around 2 to 5: worth noticing
• above 5 or 10: potential concern, depending on context

VIF does not tell us whether a predictor matters.

It tells us whether predictors are too entangled for the model to estimate their separate effects cleanly.

A short model-checking paragraph can be enough.

Example structure:

I checked the selected model by inspecting the residuals. The histogram suggested that … . The residuals-versus-predicted plot suggested that … . The residuals-versus-age plot suggested that … . The VIF values suggested that … . Overall, these checks suggest that …

Keep the write-up descriptive and honest.

From the NOW learning room, open part-2-model-checks.Rmd.

You will:

1. fit one linear model
2. calculate predicted values and residuals
3. inspect the residual distribution
4. calculate residual skew
5. plot residuals against predicted values and age
6. calculate and interpret VIF
7. write a short model-checking paragraph

• Were the residuals roughly centred around zero?
• Was the residual distribution strongly skewed?
• Did residuals show patterns across predicted values?
• Did residuals show patterns across age?
• Were the VIF values concerning?
• What would you write in a short model-checking paragraph?

After the model-checking exercise, open part-2-formative-report-guide.Rmd.

Use it to check that your formative report has:

• a research question
• a short description of the data and variables
• descriptives and a descriptive plot
• a fitted model
• model results from tidy()
• predictions and a prediction plot
• model checks
• a short conclusion

Use this time to work on your own formative report.

Priority tasks:

• make sure the report knits
• check that your data file is in the right folder
• write your model formula
• create one useful plot
• draft your model-checking paragraph
• note any questions for next week

Before formative submission:

• RMarkdown knits from a fresh R session
• all data files are included
• report has a research question
• descriptive plot is included
• model is interpreted in context
• predictions are shown if relevant
• model comparison is reported briefly if used
• model checks are described honestly
• limitations are briefly acknowledged