Linear regression finds the straight line that best fits a set of data points. Once we have that line, we can use it to make predictions.

• Simple linear regression uses one variable \(x\) to predict another variable \(y\).
• We use the built-in R dataset mtcars:
  • \(x\) = wt, the weight of a car (in 1000 lb)
  • \(y\) = mpg, its fuel economy (miles per gallon)

Our question: if a car gets 1000 lb heavier, how many miles per gallon does it lose?

Each dot is one car. Heavier cars get fewer miles per gallon, and the dots roughly follow a straight line, so linear regression is a reasonable model here.

The model for our data is

\[ \texttt{mpg} = \beta_0 + \beta_1\cdot \texttt{wt} + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0; \,\sigma^2) \]

• \(\beta_0\) is the intercept and \(\beta_1\) is the slope. \(\beta_1\) answers our question.
• \(\varepsilon\) is random error: real cars don’t sit exactly on the line.

We don’t know \(\beta_0\) and \(\beta_1\), so we estimate them from the data. The fitted line is

\[ \widehat{\texttt{mpg}} = \hat{\beta}_0 + \hat{\beta}_1\cdot \texttt{wt}, \qquad \hat{\beta}_0 = b_0 \text{ (estimate of } \beta_0), \quad \hat{\beta}_1 = b_1 \text{ (estimate of } \beta_1) \]

The error (residual) for car \(i\) is the actual value minus the fitted value: \(e_i = y_i - \hat{y}_i\).

Three lines through the same 4 points. The error for a point is its vertical distance to the line.

\(L_1: \sum |e_i| = 2 + 0 + 0 + 2 = 4 \qquad L_2: \sum |e_i| = 1 + 1 + 1 + 1 = 4 \qquad L_3: \sum |e_i| = 0 + 2 + 2 + 0 = 4\)

\(L_1: \sum e_i^2 = 2^2 + 0^2 + 0^2 + 2^2 = 8 \qquad L_2: \sum e_i^2 = 1^2 + 1^2 + 1^2 + 1^2 = 4 \qquad L_3: \sum e_i^2 = 8\)

Adding up the errors gives a tie. Adding up the squared errors picks \(L_2\) as the best line.

Step 1. The best line is the one with the smallest sum of squared errors:

\[ SSE = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 \]

Step 2. We don’t have to try every line. The slope and intercept of the best line are

\[ SS_x = \sum_{i=1}^n (x_i - \bar{x})^2, \qquad b_1 = {\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) \over SS_x}, \qquad b_0 = \bar{y} - b_1 \bar{x} \]

where \(\bar{x}\) is the average weight and \(\bar{y}\) is the average mpg.

For mtcars this gives \(b_1 = -5.34\) and \(b_0 = 37.29\).

R’s lm() function finds \(b_0\) and \(b_1\) for us, and summary() shows the results.

mod <- lm(mpg ~ wt, data=mtcars)
summary(mod)

## 
## Call:
## lm(formula = mpg ~ wt, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528, Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

A confidence interval gives a range of likely values for \(\beta_0\) and \(\beta_1\). First we estimate \(\sigma^2\) with the mean squared error:

\[ s^2 = MSE = {SSE \over n-2} = {1\over n-2} \sum_{i=1}^n (y_i - \hat{y}_i)^2 \]

The standard errors of \(b_0\) and \(b_1\) are

\[ SE(b_0) = \sqrt{s^2 \left({1 \over n} + {\bar{x}^2 \over SS_x}\right)}, \qquad SE(b_1) = \sqrt{s^2 \over SS_x} \]

and the 90% confidence intervals are

\[ b_0 \pm t_{0.05,\, n-2} \cdot SE(b_0), \qquad b_1 \pm t_{0.05,\, n-2} \cdot SE(b_1) \]

y = mtcars$mpg; x = mtcars$wt; n = length(y)
SSx = sum((x - mean(x))^2)
b1 = sum((x-mean(x)) * (y-mean(y)))/SSx;   b0 = mean(y) - b1 * mean(x)
e = y - (b0 + b1 * x)
s_sq = sum(e^2) / (n-2);  s = sqrt(s_sq)
SEb0 = sqrt(s_sq * (1 / n + mean(x) ^ 2 / SSx))
SEb1 = sqrt(s_sq / SSx)
tcrit = qt(p=0.05, df=n-2, lower.tail=FALSE)
Lb0 = b0 - SEb0 * tcrit;  Rb0 = b0 + SEb0 * tcrit
Lb1 = b1 - SEb1 * tcrit;  Rb1 = b1 + SEb1 * tcrit

## data frame with 90% confidence intervals for b0 and b1
CI = data.frame(matrix(c(Lb0, Rb0, Lb1, Rb1), nrow=2, byrow=T),row.names=c("b0","b1"))
colnames(CI) = c("lwr","upr")   ## naming the columns of data frame CI
CI   ## printing data frame CI

##          lwr       upr
## b0 34.098303 40.471950
## b1 -6.293412 -4.395531

Hover over a point to see its weight and mpg.

• Fitted line: \(\widehat{\texttt{mpg}} = 37.29 -5.34 \cdot \texttt{wt}\)
• Answer: each extra 1000 lb costs about 5.34 mpg.
• We are 90% confident that \(\beta_1\) is between -6.29 and -4.4. The interval doesn’t include 0, so weight really does affect mpg.
• The grey band is the 90% confidence band from geom_smooth(method="lm", level=0.90).