Two variables, one question

A heavier car burns more fuel. Everyone knows that. The useful questions are sharper:

  • How much more, per extra thousand pounds?
  • How confident can we be in that number?

Simple linear regression answers both. We model the response \(Y\) as a straight line in the predictor \(x\), plus noise:

\[Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad i = 1, \dots, n\]

\[\varepsilon_i \ \stackrel{iid}{\sim}\ N(0, \sigma^2)\]

The slope \(\beta_1\) is the whole point: it is the expected change in \(Y\) for a one-unit increase in \(x\).

The data: 32 cars

mtcars ships with R — road-test figures for 32 models from a 1974 issue of Motor Trend. We take fuel economy (mpg) as the response and weight (wt, in thousands of pounds) as the predictor.

First five cars in the data set
Weight (1000 lb) MPG Horsepower Cylinders
Mazda RX4 2.62 21.0 110 6
Mazda RX4 Wag 2.88 21.0 110 6
Datsun 710 2.32 22.8 93 4
Hornet 4 Drive 3.21 21.4 110 6
Hornet Sportabout 3.44 18.7 175 8

A small, clean, complete data set: no missing values in either variable.

Look before you fit

The relationship is clearly negative and close enough to straight to justify a line. Always plot first — a line can be fitted to anything, including data that has no business being fitted with one.

The least-squares criterion

Infinitely many lines pass near those points. Least squares picks the one minimizing the total squared vertical distance to the data:

\[S(b_0, b_1) \;=\; \sum_{i=1}^{n} \left( y_i - b_0 - b_1 x_i \right)^2\]

Setting \(\partial S/\partial b_0 = \partial S/\partial b_1 = 0\) gives a unique solution in closed form:

\[\hat\beta_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}, \qquad \hat\beta_0 = \bar{y} - \hat\beta_1 \bar{x}\]

No iteration, no search. A later slide shows what that minimum looks like as a surface.

Fitting it in R

One function call does the whole computation:

fit <- lm(mpg ~ wt, data = mtcars)
signif(coef(summary(fit)), 4)
##             Estimate Std. Error t value  Pr(>|t|)
## (Intercept)   37.290     1.8780  19.860 8.242e-19
## wt            -5.344     0.5591  -9.559 1.294e-10

The Estimate column holds \(\hat\beta_0\) and \(\hat\beta_1\). The fitted line is

\[\hat{y} = 37.29 - 5.34\,x\]

Each additional 1,000 lbs is associated with about 5.34 fewer miles per gallon.

The fitted line

The shaded band is a confidence interval for the mean response at each weight, not a range for individual cars. It is narrowest at \(\bar{x}\) and flares at the extremes.

Checking the assumptions

Residuals should scatter without pattern around zero. These show a mild curvature, which hints that the true relationship bends slightly.

Least squares, seen as a surface

Every point on this surface is one candidate line. The red dot is lm()’s answer, sitting at the bottom of the bowl. Drag to rotate.

Is the slope real?

Under the model, the least-squares estimator is \(t\)-distributed, which turns the slope into a testable claim. For \(H_0: \beta_1 = 0\) against \(H_a: \beta_1 \neq 0\):

\[t = \frac{\hat\beta_1 - 0}{SE(\hat\beta_1)}, \qquad SE(\hat\beta_1) = \frac{s}{\sqrt{S_{xx}}}, \qquad s^2 = \frac{\sum_{i=1}^{n} e_i^2}{n - 2}\]

Here \(t = -9.56\) on \(30\) degrees of freedom — far into the tail. Separately, the proportion of variation the line accounts for is

\[r^2 = 1 - \frac{SSE}{SST} = 0.753\]

Predicting a new car

new_cars <- data.frame(wt = c(2.0, 3.0, 4.0))
round(predict(fit, newdata = new_cars, interval = "prediction"), 2)
##     fit   lwr   upr
## 1 26.60 20.13 33.06
## 2 21.25 14.93 27.57
## 3 15.91  9.53 22.29

fit is the point prediction; lwr and upr bound a 95% prediction interval — a range for one new car, not for the average of many. It is much wider than the band shown earlier, because it carries both the uncertainty in the line and the scatter of individual cars around it.

What we can and cannot conclude

  • The fit is strong: weight accounts for about 75% of the variation in fuel economy.
  • The slope is precisely estimated, so the direction and rough size of the effect are not in doubt.
  • The residuals bend, so the straight line is an approximation — a quadratic term or a log transform would likely do better.
  • None of this is causal. Heavy cars also tend to have bigger engines, and this model cannot separate the two.