What Is Simple Linear Regression?

Simple linear regression relates a numeric response \(Y\) to one predictor \(X\). \[ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, \qquad \varepsilon_i \sim \mathcal{N}(0, \sigma^2). \]

It is useful when a straight line can describe how two variables move together.

Data Example

We use the built-in mtcars dataset and focus on weight (wt) and miles per gallon (mpg).

df <- mtcars %>%
  select(mpg, wt, hp, cyl) %>%
  mutate(cyl = factor(cyl))
kable(head(df), caption = "Preview of the data")
Preview of the data
mpg wt hp cyl
Mazda RX4 21.0 2.620 110 6
Mazda RX4 Wag 21.0 2.875 110 6
Datsun 710 22.8 2.320 93 4
Hornet 4 Drive 21.4 3.215 110 6
Hornet Sportabout 18.7 3.440 175 8
Valiant 18.1 3.460 105 6

Fit the Model

MPG as a function of weight:

slr <- lm(mpg ~ wt, data = df)
summary(slr)
## 
## Call:
## lm(formula = mpg ~ wt, data = df)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528, Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

MPG vs Weight (ggplot)

ggplot(df, aes(x = wt, y = mpg)) +
  geom_point(alpha = 0.9) +
  geom_smooth(method = "lm", se = TRUE) +
  labs(title = "Heavier Cars Tend to Have Lower MPG",
       x = "Weight (1000 lbs)", y = "MPG")

Residuals vs Fitted (ggplot)

aug <- augment(slr)
ggplot(aug, aes(.fitted, .resid)) +
  geom_hline(yintercept = 0, linetype = 2) +
  geom_point(alpha = 0.9) +
  labs(title = "Residuals vs Fitted", x = "Fitted MPG", y = "Residuals")

OLS Formulas

\[ \hat{\beta}_1 = \frac{\sum_i (X_i-\bar{X})(Y_i-\bar{Y})}{\sum_i (X_i-\bar{X})^2}, \qquad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\,\bar{X}. \]

Unbiased error variance: \[ \hat{\sigma}^2 = \frac{1}{n-2}\sum_i (Y_i - \hat{Y}_i)^2. \]

Testing the Slope

\[ H_0: \beta_1 = 0, \quad H_a: \beta_1 \neq 0, \qquad t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)} \sim t_{n-2}. \]

A small p-value suggests weight is a meaningful predictor of MPG.

3D Scatter (Plotly)

Confidence & Prediction Intervals

new <- tibble(wt = c(2.2, 3.0, 3.8))
ci <- predict(slr, newdata = new, interval = "confidence")
pi <- predict(slr, newdata = new, interval = "prediction")
cbind(new, mean = ci[, "fit"], ci_lwr = ci[, "lwr"], ci_upr = ci[, "upr"],
      pred_lwr = pi[, "lwr"], pred_upr = pi[, "upr"])
##    wt     mean   ci_lwr   ci_upr pred_lwr pred_upr
## 1 2.2 25.52729 23.92780 27.12678 19.10442 31.95016
## 2 3.0 21.25171 20.12444 22.37899 14.92987 27.57355
## 3 3.8 16.97613 15.69084 18.26143 10.62422 23.32805

Summary

Simple linear regression is a clear way to describe how two variables relate.
Plot the data, fit the line, check residuals, and report uncertainty (SEs, CIs, p-values).