What Is Simple Linear Regression?

  • Simple linear regression is one of the first tools we learn in statistics for looking at how two variables are related.
  • We have a predictor \(X\) (something we already know) and a response \(Y\) (something we want to explain or predict), and we try to fit a straight line through the data.
  • Once we have that line, we can use it to guess \(Y\) for a new value of \(X\).
  • To keep things simple, I’ll use one example the whole way through: predicting a student’s exam score from how many hours they studied.

The Model

Here’s how we write the model in math notation:

\[ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, \qquad i = 1, \dots, n \]

  • \(\beta_0\) is the intercept, so it’s what we’d predict for \(Y\) if \(X\) were \(0\).
  • \(\beta_1\) is the slope, telling us how much \(Y\) changes on average when \(X\) goes up by one unit.
  • \(\varepsilon_i\) is just the “leftover” error we can’t explain, and we assume it’s random noise: \(\varepsilon_i \sim N(0, \sigma^2)\).

For this model to actually make sense, a few assumptions need to roughly hold: the relationship should be linear, the observations independent, the spread of the errors constant, and the errors approximately normal.

Estimating the Coefficients

So how do we actually find \(\beta_0\) and \(\beta_1\)? The idea behind least squares is to pick the line that makes the total squared distance between the data points and the line as small as possible:

\[ SSE(\beta_0, \beta_1) = \sum_{i=1}^{n} \left(y_i - \beta_0 - \beta_1 x_i\right)^2 \]

Working through the calculus (minimizing \(SSE\)) gives us these formulas we can just plug our data into:

\[ \hat\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^n (x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x \]

R Code: Making Up Some Data & Fitting the Model

set.seed(301)
n <- 50
study_hours <- runif(n, min = 1, max = 10)
exam_score  <- 50 + 5 * study_hours + rnorm(n, sd = 8)
df  <- data.frame(study_hours, exam_score)
fit <- lm(exam_score ~ study_hours, data = df)

p1 <- ggplot(df, aes(x = study_hours, y = exam_score)) +
  geom_point(color = "#8C1D40", size = 2) +
  geom_smooth(method = "lm", formula = y ~ x, color = "black", se = TRUE) +
  labs(x = "Study Hours", y = "Exam Score", title = "Exam Score vs. Study Hours") +
  theme_minimal()

I made up a realistic dataset since I didn’t have a real one handy: study_hours between 1 and 10, exam_score going up with study hours plus some noise. lm() fits the model, and this same chunk builds the plot on the next slide.

ggplot Plot: Fitted Regression Line

ggplot Plot: Residual Diagnostics

This is one way we check if the model is doing a good job: if the residuals just look like random scatter around zero with no obvious shape, that’s a good sign the linearity and constant-variance assumptions are reasonable here.

Visualizing Least Squares in 3D

This lets us actually see what least squares is doing: the surface is \(SSE(\beta_0,\beta_1)\) for many intercept/slope combos, and the black dot marks the lowest point on it — which lines up exactly with \((\hat\beta_0, \hat\beta_1)\) from the earlier slide.

Interpreting the Results

  • Putting the estimated numbers into the model gives: \(\widehat{\text{Score}} = 52.25 + 4.54 \times \text{Hours}\)
  • The slope tells us that for every extra hour studied, the exam score goes up by about 4.54 points, on average.
  • The intercept says a student who studied 0 hours would be predicted to score around 52.25 (this is mostly just a mathematical starting point, not something we’d expect to see in real life).
  • \(R^2 = 0.666\), meaning study hours alone explain about 66.6% of the differences we see in exam scores. The rest is probably other stuff we didn’t measure.

Summary

  • Simple linear regression is really just about fitting the “best” straight line through a scatter of points.
  • “Best” here means least squares: the line that minimizes the total squared error between the data and the fitted line.
  • We can check whether the model is reasonable by looking at the scatter plot and the residual plot.
  • \(R^2\) gives us a quick sense of how much of the variation in the response our predictor is actually explaining.
  • Once this makes sense, it’s a pretty short jump to multiple regression, where you just add more predictors to the same idea.