We often have two measurements on the same units and want to know:

• Is there a relationship between \(x\) and \(y\)?
• If so, how strong is it, and in which direction?
• Given a new \(x\), what \(y\) should we predict, and how uncertain is that prediction?

Simple linear regression is the smallest useful version of this problem!

Running example: the mtcars data (32 cars, Motor Trend 1974). Does a car’s weight explain its fuel economy?

For observations \(i = 1, \dots, n\):

\[ Y_i \;=\; \beta_0 \;+\; \beta_1 x_i \;+\; \varepsilon_i, \qquad \varepsilon_i \;\stackrel{\text{iid}}{\sim}\; N(0, \sigma^2) \]

Equivalently, the line describes the conditional mean of the response:

\[ \mathbb{E}[Y \mid X = x] = \beta_0 + \beta_1 x, \qquad \operatorname{Var}(Y \mid X = x) = \sigma^2 \]

The cloud slopes down and looks reasonably straight: a line is a defensible summary.

fit <- lm(mpg ~ wt, data = mtcars)
coef(fit)

## (Intercept)          wt 
##   37.285126   -5.344472

# the closed form, by hand, as a check
x <- mtcars$wt; y <- mtcars$mpg
beta1 <- sum((x - mean(x)) * (y - mean(y))) / sum((x - mean(x))^2)
beta0 <- mean(y) - beta1 * mean(x)
c(beta0 = beta0, beta1 = beta1)

##     beta0     beta1 
## 37.285126 -5.344472

lm() does not actually invert \((X^\top X)\); it uses a QR decomposition of the design matrix.

\[ \widehat{\text{mpg}} \;=\; 37.29 \; -5.34 \times \text{wt} \]

• Slope \(\hat{\beta}_1 = -5.344\): one extra 1,000 lbs of weight is associated with about 5.34 fewer mpg, on average, across these cars.
• Intercept \(\hat{\beta}_0 = 37.29\): the predicted mileage of a weightless car. The data run from 1.513 to 5.424 thousand lbs, so \(x = 0\) is far outside the observed range — the intercept here anchors the line, it does not describe a real vehicle.

The bowl is a long, tilted valley, not a round basin: \(\hat{\beta}_0\) and \(\hat{\beta}_1\) are strongly negatively correlated. Centering \(x\) at \(\bar{x}\) rotates the valley upright and decorrelates them.

The total variation in \(y\) splits exactly in two:

\[ \underbrace{\sum_i (y_i - \bar{y})^2}_{\text{SST}} \;=\; \underbrace{\sum_i (\hat{y}_i - \bar{y})^2}_{\text{SSR (explained)}} \;+\; \underbrace{\sum_i (y_i - \hat{y}_i)^2}_{\text{SSE (left over)}} \]

\[ R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}}, \qquad \hat{\sigma} = \sqrt{\frac{\text{SSE}}{n-2}} \]

Here \(R^2 = 0.7528\) — weight accounts for about 75% of the variation in mileage — and \(\hat{\sigma} = 3.046\) mpg is the typical size of a residual.

In simple linear regression only, \(R^2 = r_{xy}^2\).

\[ \operatorname{SE}(\hat{\beta}_1) = \frac{\hat{\sigma}}{\sqrt{S_{xx}}}, \qquad t = \frac{\hat{\beta}_1 - 0}{\operatorname{SE}(\hat{\beta}_1)} \;\sim\; t_{\,n-2} \quad \text{under } H_0\!:\beta_1 = 0 \]

confint(fit, "wt", level = 0.95)

##        2.5 %    97.5 %
## wt -6.486308 -4.202635

\(t = -9.56\) on 30 degrees of freedom. A slope of zero is not plausible here.

Curvature here means the mean is wrong; a funnel shape means the variance is. This plot bends slightly upward at both ends, hinting that mpg vs weight is mildly convex.

Points track the line reasonably well.

• Simple linear regression models the conditional mean of \(y\) as a straight line in \(x\), with normal, constant-variance, independent errors.
• Least squares has a closed form; the loss surface is a convex bowl with one minimum, and \(\hat{\beta}_1 = r_{xy}\,s_y/s_x\).
• For mtcars, \(\widehat{\text{mpg}} = 37.29 -5.34\,\text{wt}\), with \(R^2 = 0.753\) and a slope that is highly significant.
• Predicting a new observation is a harder problem than estimating the mean response, and the intervals reflect that.