Simple linear regression models the relationship between a single predictor variable \(x\) and a response variable \(y\) by fitting a straight line.

The goal is to find the line that best summarizes how \(y\) changes as \(x\) changes.

• Predictor (\(x\)): the explanatory variable (e.g. weight of a car)
• Response (\(y\)): the outcome we want to explain (e.g. fuel efficiency)

We use the built-in mtcars data set throughout: does a car’s weight predict its miles per gallon?

We assume the response is a linear function of the predictor, plus random error:

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad i = 1, \dots, n \]

where

• \(\beta_0\) is the intercept (value of \(y\) when \(x = 0\)),
• \(\beta_1\) is the slope (change in \(y\) per unit change in \(x\)),
• \(\varepsilon_i \sim N(0, \sigma^2)\) are independent random errors.

The fitted line used for prediction is

\[ \hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x. \]

We choose \(\hat{\beta}_0\) and \(\hat{\beta}_1\) to minimize the residual sum of squares:

\[ RSS = \sum_{i=1}^{n} \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right)^2 . \]

Solving gives the closed-form least-squares estimates:

\[ \hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})} {\sum_{i=1}^{n} (x_i - \bar{x})^2}, \qquad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}. \]

Intuitively, the slope is the covariance of \(x\) and \(y\) divided by the variance of \(x\).

There is a clear negative trend: heavier cars tend to get fewer miles per gallon.

The blue line is the least-squares fit; the shaded band is the 95% confidence region for the mean response.

model <- lm(mpg ~ wt, data = mtcars)
coef(model)

## (Intercept)          wt 
##   37.285126   -5.344472

The estimated model is

\[ \widehat{\text{mpg}} = 37.29 -5.34 \cdot \text{wt}. \]

So each additional 1000 lbs of weight is associated with a drop of about 5.34 miles per gallon.

The coefficient of determination measures the proportion of variance in \(y\) explained by the model:

\[ R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}. \]

For our model, \(R^2 = 0.753\), meaning weight alone explains about 75.3% of the variation in fuel efficiency.

Adding horsepower as a second predictor gives a regression plane. Rotate the plot below to explore it.

• Simple linear regression fits \(\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x\) by minimizing the residual sum of squares.
• The slope tells us the expected change in \(y\) per unit of \(x\).
• \(R^2\) summarizes how much variation the model explains.
• The idea extends naturally to multiple predictors — a line becomes a plane.

Key takeaway: regression turns a cloud of points into an interpretable, predictive relationship.