• Simple linear regression is one of the first tools we learn in statistics for looking at how two variables are related.
• We have a predictor \(X\) (something we already know) and a response \(Y\) (something we want to explain or predict), and we try to fit a straight line through the data.
• Once we have that line, we can use it to guess \(Y\) for a new value of \(X\).
• To keep things simple, I’ll use one example the whole way through: predicting a student’s exam score from how many hours they studied.

Here’s how we write the model in math notation:

\[ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, \qquad i = 1, \dots, n \]

• \(\beta_0\) is the intercept, so it’s what we’d predict for \(Y\) if \(X\) were \(0\).
• \(\beta_1\) is the slope, telling us how much \(Y\) changes on average when \(X\) goes up by one unit.
• \(\varepsilon_i\) is just the “leftover” error we can’t explain, and we assume it’s random noise: \(\varepsilon_i \sim N(0, \sigma^2)\).

For this model to actually make sense, a few assumptions need to roughly hold: the relationship should be linear, the observations independent, the spread of the errors constant, and the errors approximately normal.

So how do we actually find \(\beta_0\) and \(\beta_1\)? The idea behind least squares is to pick the line that makes the total squared distance between the data points and the line as small as possible:

\[ SSE(\beta_0, \beta_1) = \sum_{i=1}^{n} \left(y_i - \beta_0 - \beta_1 x_i\right)^2 \]

Working through the calculus (minimizing \(SSE\)) gives us these formulas we can just plug our data into:

\[ \hat\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^n (x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x \]

set.seed(301)
n <- 50
study_hours <- runif(n, min = 1, max = 10)
exam_score  <- 50 + 5 * study_hours + rnorm(n, sd = 8)
df <- data.frame(study_hours, exam_score)

fit <- lm(exam_score ~ study_hours, data = df)

p1 <- ggplot(df, aes(x = study_hours, y = exam_score)) +
  geom_point(color = "#8C1D40", size = 2) +
  geom_smooth(method = "lm", formula = y ~ x, color = "black", se = TRUE) +
  labs(x = "Study Hours", y = "Exam Score",
       title = "Exam Score vs. Study Hours") +
  theme_minimal()

Since I don’t have a real dataset handy, I generated a fake but realistic one: study_hours between 1 and 10, and exam_score that goes up with study hours plus some random noise. Then lm() fits the model for us, and this same chunk builds the plot on the next slide.

This is one way we check if the model is doing a good job: if the residuals just look like random scatter around zero with no obvious shape, that’s a good sign the linearity and constant-variance assumptions are reasonable here.

I thought this was a nice way to actually see what least squares is doing instead of just trusting the formula: this surface is \(SSE(\beta_0,\beta_1)\) for a bunch of different intercept/slope combos, and the black dot marks the lowest point on the whole surface — which lines up exactly with the \((\hat\beta_0, \hat\beta_1)\) from the formulas on the earlier slide.

• Putting the estimated numbers into the model gives: \(\widehat{\text{Score}} = 52.25 + 4.54 \times \text{Hours}\)
• The slope tells us that for every extra hour studied, the exam score goes up by about 4.54 points, on average.
• The intercept says a student who studied 0 hours would be predicted to score around 52.25 (this is mostly just a mathematical starting point, not something we’d expect to see in real life).
• \(R^2 = 0.666\), meaning study hours alone explain about 66.6% of the differences we see in exam scores. The rest is probably other stuff we didn’t measure.

• Simple linear regression is really just about fitting the “best” straight line through a scatter of points.
• “Best” here means least squares: the line that minimizes the total squared error between the data and the fitted line.
• We can check whether the model is reasonable by looking at the scatter plot and the residual plot.
• \(R^2\) gives us a quick sense of how much of the variation in the response our predictor is actually explaining.
• Once this makes sense, it’s a pretty short jump to multiple regression, where you just add more predictors to the same idea.