MT5762 Lecture 13

The simple linear regression model has the form:

\[ \begin{align*} \textrm{response variable} =& \textrm{intercept} + \textrm{slope} \times \textrm{explanatory variable} (+ \textrm{noise})\\ y=&\beta_0+\beta_1 x\\ \end{align*} \]

So there is a simple model for the signal/underlying non-random component. In practice our observations will deviate from this.

Hence there are two numbers (parameters) that describe a linear model:

• The slope (\( \beta_1 \)) This describes how steep the relationship is (aka slope/gradient). The \( y \)-value changes by \( \beta_1 \) for every one unit increase in \( x \).
• The intercept parameter (\( \beta_0 \)) The \( y \)-value when \( x=0 \). Where the line cuts the \( y \)-axis.

Where should the line lie?

We want to establish a criterion for “best” fit to the data

For example - where should the line go?

[examples and drawing ensues]

We may not be able to immediately agree, but we can agree on what is a badly fitting line.

So, we've settled on a criterion - Ordinary Least Squares (OLS). We look to minimise discrepencies between what we observed and what the model predicts i.e. aim for small \( (y_i-\hat{y_i}) \):

\[ \min \left(\sum_i (y_i-\hat{y}_i)^2 \right) \]

This idea is used all over the place. We are minimising the sum-of-squares (SS)

How do we change SS?

• \( y_i \) are clearly fixed, so we change \( \hat{y} \)
• How do we change \( \hat{y} \)?

\( \hat{y} \) is generally a function of our bunch of \( x \) and some parameters \( \boldsymbol{\theta} \):

\[ y = f(x_1, ..., \boldsymbol{\theta}) \]

• \( x \) are also fixed - we change \( \hat{y} \) (and hence SS) by choosing \( f \) and altering its \( \boldsymbol{\theta} \)

How do we change SS?

• So for a simple \( y = \beta_0 + \beta_1 x \), we move \( \beta_0 \) and \( \beta_1 \)
• Some changes make SS go up (therefore bad), some make SS go down (therefore good)
• These are general ideas:
  • We have a parameter space (here defined by \( \beta_0 \) and \( \beta_1 \))
  • We have an objective function, which defines values of parameters to be better or worse than others
  • Here our “objective” is to minimise SS (the objective function - sometimes called a cost function)

I can't overstate the importance of this general process: linear models, generalised linear models, non-linear models, neural networks, in fact any model fitted to data

The intercept and slope estimates (\( \hat{\beta}_0 \) and \( \hat{\beta}_1 \) respectively) can be found, by hand, using:

\[ \hat{\beta}_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} \]

\[ \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x} \]

of course, you'd use a computer

You'll see we get \( p \)-values in the output

These imply tests, although none were explicit in our model specification

We get them for “free” and are often are relevant

• Each fitted line is an estimate of the true/population regression line
• The null hypothesis is that there is no linear relationship between \( x \) and \( y \) i.e. true slope is zero: \( H_0:\beta_1=0 \).
• The alternative hypothesis is that there is a linear relationship between \( x \) and \( y \) i.e. true slope is not zero: \( H_1:\beta_1 \neq 0 \).

If \( H_0 \) were true, we'd expect \( t \)-test statistics between -2 and 2 a lot of the time. Here we have 9.1, the \( p \)-value is correspondingly very small.

We conclude a significant linear relationship between \( x \) and \( y \)

We can get confidence intevals for the parameters quite easily too:

  confint(loanLM)

                   2.5 %       97.5 %
(Intercept) 3.124005e+01 3.225352e+01
VALUE       1.566567e-05 2.426589e-05

So, for every 100,000 increase in value of the property, average debt-to-income ratio increases by about 1.57 to 2.43 (I'm playing with the units for interpretability).

Note the intercept is pretty meaningingless (why?)

This is a measure of linear association between two numeric variables

It takes on value:

• 1 when they have a perfect positive association
• -1 when they have a perfect negative association
• 0 when they have no linear association
• Varying values between

We can test for the signifcance of these things - the outputs are quite self-explanatory given what we have seen to date.

  cor.test(y_0.8neg[,1], y_0.8neg[,2])

    Pearson's product-moment correlation

data:  y_0.8neg[, 1] and y_0.8neg[, 2]
t = -15.033, df = 98, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.8862127 -0.7640981
sample estimates:
       cor 
-0.8351714 

• Can be expressed in many different ways giving different interpretations
• Two are given here:

\[ R^2=1-\frac{(n-1)^{-1}\sum_{i=1}^n(y_i-\hat{y}_i)^2}{(n-1)^{-1}\sum_{i=1}^n(y_i-\bar{y})^2} = 1-\frac{{\mathrm SSE}/(n-1)}{{\mathrm SST}/(n-1)} \]

• SSE is the sum of squared errors for your model, the SST is the total sum of squares for your data.
• This formulation shows the interpretation as “the proportion of variance explained by the model”

• An alternative formulation (which justifies the name \( R^2 \)) is
  \[ R^2=[Corr(\hat{\mathbf{y}}, \mathbf{y})]^2 \]
• where the \( \bar{\hat{y}} \) is the mean of the fitted values, \( \mathbf{y} \) and \( \hat{\mathbf{y}} \) are the vectors of observed and predicted response values.
• So it is the square of the correlation between observed reponse values and those predicted under the model

• Bounded by 0 and 1,
• \( R^2=1 \) is a perfect agreement between the model predictions and the observed data
• The value 1 would indicate that all of the variability in the response is accounted for by the model
• A value of zero indicates the worst possible fit, and there is no relationship between the observed data and the model predictions.

When we fit a linear model we assume the following:

• Linearity the relationship is linear
• Normality the errors are Normally distributed with mean zero
• Constant spread the errors have the same standard deviation for all values of \( x \)
• Independent observations the errors are independent

We are only checking two things: the model for signal and the model for noise

Reposed as the two things:

• Signal was the straight line a good model for the signal?
• Noise does the stuff left over look like independent draws from a single Normal distribution?

The residuals (\( y_i-\hat{y_i} \)) give us clues about the nature of the noise - so we look at these mainly

The residuals from the fitted model are the estimates of the unobserved errors and make a useful tool for checking many of our assumptions

• Linearity Scatter plot of \( y \) versus \( x \). We want to see a linear trend with constant scatter about the trend. Check for outliers.

Outliers are “weird” observations:

• a very large residual
• inconsistent with the model (signal or noise)

They may be genuine - our model may be wrong. They may be mistakes.

As for previous tests/models:

• Normal shape use Normal quantile-quantile (normal-QQ) plot of the residuals and/or a test for Normality:
  • \( H_0: \) The data come from a Normal distribution
  • \( H_1: \) The data don't come from a Normal distribution
  • small \( p \)-values provide evidence against the null hypothesis - conclude the data isn't Normally distributed when the \( p \)-value is small

Many types of data are obviously a priori non-Normal. See the GLMs in MT5761 for more tools.

NB You may encounter skewness and kurtosis measures.

These two parameters can be used to describe aspects of shape relative to Normality:

• skewness \( \gamma_1 \): a measure of asymmetry, \( \gamma_1=0 \) for the Normal distribution.
• Kurtosis \( \gamma_2 \): a measure of peaky-ness of the density of the distribution, \( \gamma_2=3 \) for the Normal distribution.

You should be conversant with terms left/right skew and lepto/platy-kurtosis (too pointy or too flat for a Normal)

Similar to the simpler tests/models, we can plot the residuals in various ways to see if independent draws from a single Normal distribution seems plausible

• Constant spread Scatter plot of residuals versus \( x \) or residuals versus the fitted values (\( \hat{y} \)). We want to see a patternless horizontal band (analogous to looking for homoscedacisity in ANOVA or \( t \)-tests)
• Independent observations Scatter plot of residuals in serial/some logical order (e.g. order/time of collection?)

We're often working with models of the form:

\[ y_i = \beta_0 + \beta_1 \times x_i + ... \]

This is long-winded. Matrices help.

This compact expression is really useful:

\[ y = \mathbf{X}\boldsymbol{\beta} \]

Which for our fitted regression would be:

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} 1 & x_1\\ 1 & x_2\\ \vdots & \vdots\\ 1 & x_n\\ \end{bmatrix} \begin{bmatrix} \hat{\beta_0}\\ \hat{\beta_1}\\ \end{bmatrix} \]

Which is simply \( y_i = \beta_0 + \beta_1 x_i \), but could cover much more for the same space.

[does stuff on the board - attending lectures is useful]