What Is Simple Linear Regression?

  • A statistical method that models the relationship between one numeric response \(Y\) and one numeric predictor \(x\).
  • Used across engineering, biology, economics, and the physical sciences to quantify how one quantity changes with another, and to make predictions.
  • Running example: can we predict a car’s fuel efficiency (miles per gallon) from its weight? We’ll use the built-in mtcars dataset (32 cars, road-tested by Motor Trend magazine).
  • Intuition: heavier cars need more energy to move, so we expect fuel efficiency to decrease as weight increases.

The Statistical Model

For observations \(i = 1, \dots, n\), the simple linear regression model is

\[ Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad \varepsilon_i \overset{iid}{\sim} N(0, \sigma^2) \]

  • \(\beta_0\) : the intercept, the mean of \(Y\) when \(x = 0\)
  • \(\beta_1\) : the slope, the mean change in \(Y\) per unit increase in \(x\)
  • \(\varepsilon_i\) : random error, capturing everything not explained by \(x\)

Key assumptions (L.I.N.E.):

\[ \textbf{L}\text{inearity} \quad \textbf{I}\text{ndependence} \quad \textbf{N}\text{ormality of } \varepsilon_i \quad \textbf{E}\text{qual variance } (\sigma^2 \text{ constant}) \]

Least-Squares Estimation

We estimate \(\beta_0, \beta_1\) by minimizing the sum of squared errors:

\[ \text{SSE}(\beta_0, \beta_1) \;=\; \sum_{i=1}^{n} \left( y_i - \beta_0 - \beta_1 x_i \right)^2 \]

Setting \(\partial \text{SSE}/\partial \beta_0 = 0\) and \(\partial \text{SSE}/\partial \beta_1 = 0\) and solving gives the least-squares estimates:

\[ \hat\beta_1 = \frac{\displaystyle\sum_{i=1}^{n} (x_i - \bar x)(y_i - \bar y)} {\displaystyle\sum_{i=1}^{n} (x_i - \bar x)^2} \;, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x \]

The fitted line \(\hat y = \hat\beta_0 + \hat\beta_1 x\) is the unique line minimizing SSE, visualized in 3D a few slides from now.

The Data

R Code: Fitting the Model

The scatterplot and fitted line on the previous slide came from:

ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point(color = "#1F6F8B", size = 2.6, alpha = 0.85) +
  geom_smooth(method = "lm", se = TRUE, color = "#C0392B",
              fill = "#C0392B", alpha = 0.15)

The underlying model is fit with lm():

fit = lm(mpg ~ wt, data = mtcars)
coef(fit)
## (Intercept)          wt 
##   37.285126   -5.344472
summary(fit)$r.squared
## [1] 0.7528328

Visualizing the Least-Squares Criterion

The red marker sits at the bottom of the bowl: the \((\hat\beta_0, \hat\beta_1)\) pair that minimizes SSE. Drag to rotate.

Checking Model Assumptions: Residuals

Points scatter fairly evenly above and below zero (no fanning), so equal variance looks reasonable; the slight U-shape hints at mild nonlinearity, and a quadratic term in weight might improve the fit further.

Interpreting the Results

Fitted model:

\[ \widehat{\text{mpg}} = 37.29 \; -5.34 \times \text{wt} \]

  • Slope \(\hat\beta_1 = -5.34\): each additional 1000 lbs of weight is associated with about 5.34 fewer miles per gallon, on average.
  • Intercept \(\hat\beta_0 = 37.29\): the (extrapolated) mean mpg for a car of zero weight, not practically meaningful on its own.
  • \(R^2 = 0.753\): about 75.3% of the variability in mpg is explained by weight.

A prediction at a new value \(x_0\) comes with a prediction interval:

\[ \hat y_0 \;\pm\; t_{n-2,\,\alpha/2}\; s \sqrt{1 + \frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}} \]

Example: Predicting for a New Car

For a car weighing 3.5k lbs: predicted mpg \(= 18.58\), 95% prediction interval \((12.25,\; 24.9)\).

Summary

  • Simple linear regression models a numeric response as a linear function of one predictor plus random error.
  • \(\hat\beta_0, \hat\beta_1\) are chosen to minimize SSE, visualized directly as the lowest point of the 3D error surface.
  • \(R^2\) measures how much variability the line explains; residual plots check whether the model’s assumptions are reasonable.
  • Prediction intervals quantify the uncertainty in a forecast for a new observation, not just the average trend.
  • The same machinery extends far beyond cars: dose–response curves in biology, stress–strain relationships in engineering, calibration curves in chemistry, and more.