MT5762 Lecture 16

We've seen low the linear model encompasses a number of our previous methods

These model the signal with a design matrix \( \mathbf{X} \) and associated parameters \( {\beta} \) which are estimated assuming the noise is iid (independent and identically distributed) Normal

We look at extending this model for signal to interactions and more

My favourite equation

Many models are effectively regressions - they are broadly of the form:

\[ y = f(x_1,..., x_p) + noise \]

So we need to estimate the signal \( f \), which is usually contingent on an idea of the distribution of the error (because we have to tease the signal and noise apart)

As we'll see, our estimates for the signal are optimised in light of our model for noise

The term linear can be used differently in different contexts

• It might refer to some shape of relationship e.g. a straight line relating some variable \( x \) to \( y \)
• Here we mean the linear form. In short we model the signal as \( y = f(x1,...x_p) \) where it is

\[ y = \mathbf{X}\boldsymbol{\beta} \]

• Recall this matrix equation means things like:

\[ y_i = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p \]

we've included categorical covariates in our regression equation by cunning use of the design matrix (via dummy variables).

• This requires one level of the factor be set as baseline
• The baseline level is picked by software (e.g. alphabetical), but can be changed
• The baseline parameter is the mean of \( y \) in this level, accounting for other covariates
• The other factor level parameters are (usually) the mean \( y \) differences from this baseline

Returning to the mortgage default data:

• Fitting a numeric (property value) and factor (job type) variable to predict debt-to-income ratio
• We interpret…

  factModel <- lm(DEBTINC ~ JOB + VALUE, data = loanData)

  summary(factModel)

To test between factors that are not baseline levels requires:

• Changing the baseline, or
• Requesting contrasts be outputted e.g. like Tukey's

Adding categorical covariates to this mix gives different intercepts by factor level - so conceptually multiple lines, planes or hyperplanes

The interpretation of these terms as presented is straightforwards (but not for interactions later):

• Each slope parameter is the (estimated) change in the mean response, for each unit increase in \( x \), even if there are many
• The factor variables indicate the differences from baseline - so:
  • for lines, the vertical shift from the baseline line for each factor level
  • for planes, the vertical shift from the baseline plane for each factor level
  • for hyperplanes, the vertical shift from the baseline hyperplane for each factor level

Covariates don't necessarily affect \( y \) in isolation

The form we've seen so far has our covariates entering into the equation in a simple additive fashion i.e.

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 \]

This means the affect of \( x_1 \) on \( y \) can be isolated/separated:

• If numeric \( x \): “for every unit increase of \( x_1 \), \( y \) increases by \( \beta_1 \)”
• If categorical \( x \) then we're giving the average difference from one factor level to the baseline level

The world is rarely this simple.

• The effect of \( x_1 \) say on \( y \) might be different given the values of other \( x \)
• In which case, we have an interaction
• The effect of \( x_1 \) on \( y \) cannot be represented without reference to other \( x \):

“for every unit increase of \( x_s \), \( y \) increases by \( A \) units given \( x_p \) is \( B \)”

Thus - the marginal \( x \) terms cannot describe the relationship with \( y \) alone

Signal they permit

• When mixing numeric and categorical/factor covariates previously, we effectively allowed different intercepts for the lines/planes/hyperplanes, depending on the factor-level
• An interaction here alters the slopes/orientation within the factor levels

Consider factor \( x_1 \) with two levels \( a \) and \( b \), and a numeric \( x_2 \)

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \times x_2) \]

• Through the magic of dummy variables, \( \beta_0 \) is the mean of \( y \) when \( x_1 \) = baseline (\( a \) say) and \( x_2 \) = 0.
• \( \beta_2 \) is the increase in mean \( y \) with a unit increase in \( x_2 \) if \( x_1 \) = baseline (\( a \) level)
• \( \beta_3 \) modifies the slope parameter \( \beta_2 \) if \( x_1 \) = \( b \) level

In short, our model has two lines with different intercepts and slopes

• Intercept for factor \( x_1 = a \) is \( \beta_0 \)
• Intercept for factor \( x_1 = b \) is \( \beta_0 + \beta_1 \)
• Slope for factor \( x_1 = a \) is \( \beta_2 \)
• Slope for factor \( x_1 = b \) is \( \beta_2 +\beta_3 \)

Let's look at the design matrix

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} 1 & 0 & x_{2,1} & 0\\ 1 & 0 & x_{2,2} & 0\\ 1 & 0 & x_{2,3} & 0\\ 1 & 1 & x_{2,4} & x_{2,4}\\ 1 & 1 & x_{2,5} & x_{2,5}\\ 1 & 1 & x_{2,6} & x_{2,6}\\ \end{bmatrix} \begin{bmatrix} \hat{\beta_0}\\ \hat{\beta_1}\\ \hat{\beta_2}\\ \hat{\beta_3}\\ \end{bmatrix} = \begin{bmatrix} \hat{\beta_0} (+ 0\beta_1) + \hat{\beta_2}x_{2,1} (+ 0\hat{\beta_3})\\ \hat{\beta_0} (+ 0\beta_1) + \hat{\beta_2}x_{2,2} (+ 0\hat{\beta_3})\\ \hat{\beta_0} (+ 0\beta_1) + \hat{\beta_2}x_{2,3} (+ 0\hat{\beta_3})\\ \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2}x_{2,4} + \hat{\beta_3} x_{2,4}\\ \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2}x_{2,5} + \hat{\beta_3} x_{2,5}\\ \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2}x_{2,6} + \hat{\beta_3} x_{2,6}\\ \end{bmatrix} \]

Let's look at the last line, to see the slope modifier

\[ y_6 = \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2}x_{2,6} + \beta_3 x_{2,6} = (\hat{\beta_0} + \hat{\beta_1}) + (\hat{\beta_2} + \hat{\beta}_3)x_{2,6} \]

• the intercept if you are factor \( x_1=b \) is \( \hat{\beta_0} + \hat{\beta_1} \)
• the slope if you are factor \( x_1=b \) is \( \hat{\beta_2} + \hat{\beta}_3 \).
• \( \beta_1 \) alters the intercept and \( \beta_3 \) alters the slope, when you move from \( x_1=a \) to \( x_1=b \)

This extends to more factor levels and more covariates generally. The magic of design matrices and matrix algebra.

• In linear model output, we get tests on all parameters
  • A \( p \)-value implies a test and therefore an \( H_0 \)
  • If not specified, usually this is \( H_0: \theta = 0 \) i.e. is the parameter plausibly zero?
• The interpretation differs depending on the parameter type

• For a slope parameter, is there linear relationship between mean \( y \) and \( x \)?
• For a non-baseline factor parameter, is there a significant difference between this level and baseline, in terms of average \( y \)?
• For an interaction, is the slope significantly different between factor levels? i.e. is the slope modifier plausibly zero?

Model terms involving one covariate are terms main effects (or marginal terms). They may also appear in interactions with other covariates.

As a principle:

• We don't interpret main effects parameters if they are involved in an interaction (a higher order term)
• This makes sense, as the relationship can't be isolated - it depends on another covariate
• Similarly we ignore tests on marginal parameters if involved in an interaction

Signal they permit

• Similar to the previous example, an interaction between two numeric covariates means a slope parameter becomes conditional on another covariate's value
• Whereas before the slope altered by factor level, here it is a continuous alteration
• We'll only consider the simplest case - but many complex (hyper-)surfaces are interactions

Consider numeric \( x_1 \) and \( x_2 \), the equation is the same as the previous

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \times x_2) \]

• We have an intercept and two marginal/main effects \( \beta_1 x_1 \) and \( \beta_2 x_2 \)
• \( \beta_3 \) is again an interaction that adds to these

In short, we fit a (hyper) plane, plus an interaction surface

Let's look at the design matrix

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} 1 & x_{1,1} & x_{2,1} & x_{1,1} \times x_{2,1}\\ 1 & x_{1,2} & x_{2,2} & x_{1,2} \times x_{2,2}\\ 1 & x_{1,3} & x_{2,3} & x_{1,3} \times x_{2,3}\\ 1 & x_{1,4} & x_{2,4} & x_{1,4} \times x_{2,4}\\ 1 & x_{1,5} & x_{2,5} & x_{1,5} \times x_{2,5}\\ 1 & x_{1,6} & x_{2,6} & x_{1,6} \times x_{2,6}\\ \end{bmatrix} \begin{bmatrix} \hat{\beta_0}\\ \hat{\beta_1}\\ \hat{\beta_2}\\ \hat{\beta_3}\\ \end{bmatrix} = \]

Let's look at the design matrix

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} \hat{\beta_0} + \beta_1 x_{1,1} + \beta_2 x_{2,1} + \beta_3 (x_{1,1} \times x_{2,1})\\ \hat{\beta_0} + \beta_1 x_{1,2} + \beta_2 x_{2,2} + \beta_3 (x_{1,2} \times x_{2,2})\\ \hat{\beta_0} + \beta_1 x_{1,3} + \beta_2 x_{2,3} + \beta_3 (x_{1,3} \times x_{2,3})\\ \hat{\beta_0} + \beta_1 x_{1,4} + \beta_2 x_{2,4} + \beta_3 (x_{1,4} \times x_{2,4})\\ \hat{\beta_0} + \beta_1 x_{1,5} + \beta_2 x_{2,5} + \beta_3 (x_{1,5} \times x_{2,5})\\ \hat{\beta_0} + \beta_1 x_{1,6} + \beta_2 x_{2,6} + \beta_3 (x_{1,6} \times x_{2,6})\\ \end{bmatrix} \]

Let's look how this modifies the slope WRT \( x_1 \):

\[ \hat{\beta_0} + \beta_1 x_{1} + \beta_3 (x_{1} \times x_{2}) \]

with some rearrangement

\[ \hat{\beta_0} + (\beta_1 + \beta_3 (x_{2}))x_{1} \]

So, the slope for \( x_1 \) changes by \( \beta_3 \) for a unit increase in \( x_2 \)

• We test as per previous - the test on the interaction parameter indicating whether there is a significant interaction between the covariates

Signal they permit

• Similar to previous variants, an interaction here means predictions based on one factor covariate are conditional on the levels of another
• In effect, we are now estimating for all the combinations of factor levels for the interacting covariates

Consider factor \( x_1 \) with two levels \( A \) and \( B \), and \( x_2 \) with two levels \( C \) and \( D \)

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 = B, x_2 = D) \]

Through the magic of dummy variables:

• \( \beta_0 \) is the mean of \( y \) at baseline e.g. \( x_1=A \) and \( x_2=C \).
• \( \beta_1 \) is the change in mean \( y \) shifting from \( x_1=A \) to \( x_1=B \)

Consider factor \( x_1 \) with two levels \( A \) and \( B \), and \( x_2 \) with two levels \( C \) and \( D \)

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 = B, x_2 = D) \]

• \( \beta_2 \) is the change in mean \( y \) moving from \( x_2=C \) to \( x_2=D \)
• \( \beta_3 \) represents the interaction - “additional” changes in mean \( y \) from changing both \( x_1 \) and \( x_2 \) due to an interaction effect

In short, these estimate different means for all combinations of factor levels:

• Mean of \( y \) at \( AC \) is \( \beta_0 \)
• Mean of \( y \) at \( BC \) is \( \beta_0 + \beta_1 \)
• Mean of \( y \) at \( AD \) is \( \beta_0 + \beta_2 \)
• Mean of \( y \) at \( BD \) is \( \beta_0 + \beta_1 + \beta_2 + \beta_3 \)

If \( x_1 \) and \( x_2 \) were simply additive, then \( \beta_3 \) is not needed.

Let's look at the design matrix

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 1 & 0 & 1 & 0 \\ 1 & 0 & 1 & 0 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ \end{bmatrix} \begin{bmatrix} \hat{\beta_0}\\ \hat{\beta_1}\\ \hat{\beta_2}\\ \hat{\beta_3}\\ \end{bmatrix} = \begin{bmatrix} \hat{\beta_0} (+ 0\hat{\beta_1} + 0\hat{\beta_2} + 0\hat{\beta_3})\\ \hat{\beta_0} (+ 0\hat{\beta_1} + 0\hat{\beta_2} + 0\hat{\beta_3})\\ \hat{\beta_0} + \hat{\beta_1} + (0\hat{\beta_2} + 0\hat{\beta_3})\\ \hat{\beta_0} + \hat{\beta_1} + (0\hat{\beta_2} + 0\hat{\beta_3})\\ \hat{\beta_0} (+ 0\hat{\beta_1}) + \hat{\beta_2} + (0\hat{\beta_3})\\ \hat{\beta_0} (+ 0\hat{\beta_1}) + \hat{\beta_2} + (0\hat{\beta_3})\\ \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2} + \hat{\beta_3}\\ \hat{\beta_0} + \hat{\beta_1} + \hat{\beta_2} + \hat{\beta_3}\\ \end{bmatrix} \]

• If the factor covariates add simply, then knowing \( \beta_1 \) and \( \beta_2 \) would be enough to predict the BD combination
• In that case there is no interaction and the underlying \( \beta_3 \) is zero
• So a test on \( \beta_3 \) tests whether some interaction is in play

• Testing progresses as previous - noting we can't interpret main effects in the presence of interactions