What Is Simple Linear Regression?

Simple Linear Regression models the linear relationship between:

  • A response variable \(Y\) (dependent variable)
  • A single predictor variable \(X\) (independent variable)

Core idea: We seek a straight-line function \(\hat{Y} = f(X)\) that minimizes prediction error.

Common Applications:

Field Predictor (\(X\)) Response (\(Y\))
Economics Advertising spend Sales revenue
Medicine Drug dosage Blood pressure
Engineering Temperature Material strength
Ecology Rainfall Crop yield

The Statistical Model

The population regression model is:

\[Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, \quad i = 1, 2, \ldots, n\]

where:

  • \(Y_i\) — the \(i\)-th observed response
  • \(X_i\) — the \(i\)-th predictor value
  • \(\beta_0\) — the intercept (unknown parameter)
  • \(\beta_1\) — the slope (unknown parameter)
  • \(\varepsilon_i\) — the random error term

Distributional assumption on errors:

\[\varepsilon_i \overset{iid}{\sim} \mathcal{N}(0,\, \sigma^2)\]

Errors are independent, identically distributed Normal with mean zero and constant variance \(\sigma^2\).

Ordinary Least Squares (OLS) Estimation

We estimate \(\beta_0\) and \(\beta_1\) by minimizing the Sum of Squared Residuals:

\[\text{SSR} = \sum_{i=1}^{n} \left(Y_i - b_0 - b_1 X_i\right)^2\]

Setting partial derivatives to zero gives the OLS estimators:

\[\hat{\beta}_1 = \frac{\sum_{i=1}^{n}(X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^{n}(X_i - \bar{X})^2} = \frac{S_{XY}}{S_{XX}}\]

\[\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\,\bar{X}\]

These are BLUE — Best Linear Unbiased Estimators — by the Gauss-Markov theorem.

The fitted value for observation \(i\):

\[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i\]

Example Dataset: Cars

R’s built-in cars dataset (50 observations of stopping distances).

  • Predictor \(X\): speed (mph)
  • Response \(Y\): stopping distance (ft)

Goodness of Fit: R-squared

Total variability in \(Y\) decomposes as:

\[\underbrace{\sum_{i=1}^{n}(Y_i - \bar{Y})^2}_{SST} = \underbrace{\sum_{i=1}^{n}(\hat{Y}_i - \bar{Y})^2}_{SSReg} + \underbrace{\sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2}_{SSE}\]

The coefficient of determination:

\[R^2 = \frac{SSReg}{SST} = 1 - \frac{SSE}{SST}, \qquad 0 \leq R^2 \leq 1\]

##   Estimated Intercept (b0): -17.5791
##   Estimated Slope     (b1):  3.9324
##   R-squared               :  0.6511
##   Residual Std Error      : 15.3796

An \(R^2 \approx 0.65\) means about 65% of the variation in stopping distance is explained by speed alone.

Residual Diagnostics

Checking model assumptions via residual plots is critical.

Left: residuals should scatter randomly around 0. Right: points should follow the diagonal for normality.

3D Interactive Plot: Regression Plane

Hypothesis Testing for the Slope

We test whether \(X\) has any significant linear effect on \(Y\):

\[H_0: \beta_1 = 0 \quad \text{vs.} \quad H_1: \beta_1 \neq 0\]

Test statistic under \(H_0\) (follows a \(t\)-distribution with \(n-2\) degrees of freedom):

\[t^* = \frac{\hat{\beta}_1}{\text{SE}(\hat{\beta}_1)} = \frac{\hat{\beta}_1}{\hat{\sigma}/\sqrt{S_{XX}}} \sim t(n-2)\]

Reject \(H_0\) if \(|t^*| > t_{\alpha/2,\, n-2}\).

## Coefficient Table (cars dataset):
##                 Estimate  Std.Error  t-value   p-value
##   (Intercept)   -17.5791     6.7584   -2.601   0.0123
##   speed           3.9324     0.4155    9.464   < 0.001

With \(p < 0.001\), we strongly reject \(H_0\): speed is a highly significant predictor.

R Code: Building the Model

library(ggplot2)
library(broom)

# Load the built-in cars dataset
data(cars)

# Fit the simple linear regression model
model <- lm(dist ~ speed, data = cars)

# Print model summary
summary(model)

# Tidy coefficient table
tidy(model)

# Augment data with fitted values and residuals
aug <- augment(model)

# Scatter plot with regression line
ggplot(cars, aes(x = speed, y = dist)) +
  geom_point(color = "#e94560", size = 3) +
  geom_smooth(method = "lm", color = "#0f3460", se = TRUE) +
  labs(title = "Stopping Distance vs. Speed",
       x = "Speed (mph)", y = "Stopping Distance (ft)") +
  theme_minimal()

Key functions: lm() fits the model, summary() reports estimates and tests, augment() appends diagnostics to the original data.

Confidence and Prediction Intervals

For a new value \(X = x^*\), two types of intervals:

95% Confidence Interval for the mean response \(E[Y \mid X = x^*]\):

\[\hat{Y}^* \pm t_{\alpha/2,\,n-2} \cdot \hat{\sigma} \sqrt{\frac{1}{n} + \frac{(x^* - \bar{X})^2}{S_{XX}}}\]

95% Prediction Interval for a single future observation \(Y^*\):

\[\hat{Y}^* \pm t_{\alpha/2,\,n-2} \cdot \hat{\sigma} \sqrt{1 + \frac{1}{n} + \frac{(x^* - \bar{X})^2}{S_{XX}}}\]

Summary and Key Takeaways

Simple Linear Regression is one of the most foundational tools in statistics.

Concept Formula
Model \(Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i\)
OLS Slope \(\hat\beta_1 = S_{XY}/S_{XX}\)
OLS Intercept \(\hat\beta_0 = \bar Y - \hat\beta_1 \bar X\)
R-squared \(R^2 = 1 - SSE/SST\)
t-test statistic \(t^* = \hat\beta_1 / \text{SE}(\hat\beta_1)\)

Findings from the Cars example:

  • \(\hat\beta_1 \approx 3.93\): each additional mph adds ~3.93 ft of stopping distance
  • \(R^2 \approx 0.65\): speed explains 65% of variance in stopping distance
  • \(p < 0.001\): the relationship is highly statistically significant

Next steps: multiple regression, model selection (AIC/BIC), cross-validation.