1. Stating the Gauss-Markov Assumptions

Under the following conditions, OLS gives us the Best Linear Unbiased Estimator (BLUE):

  1. The model is written so that the outcome is a straight-line combination of the coefficients (linear in the parameters).
  2. The data are a random draw from the population we care about.
  3. The explanatory variable actually varies, and no input is an exact copy or exact combination of another input (no perfect collinearity).
  4. Whatever we left out of the model averages to zero at every value of the inputs, and is unrelated to those inputs (the zero conditional mean, or exogeneity condition).
  5. The size of the errors is the same across all values of the inputs (constant variance, or homoskedasticity).
  6. The errors do not carry information about one another (no correlation between errors), which for a clean random cross-section usually follows from assumption 2.

One note worth making up front: normality of the errors is not one of the Gauss-Markov assumptions; it is a separate condition we lean on for exact small-sample inference, which is exactly why the CLT background in the prompt matters once the sample gets large.

2. Plain-English Explanation (non-technical audience)

Think of a regression as a recipe that turns some ingredient into a prediction. The first assumption just says the recipe adds things up in a simple, straight-line way, because a straight-line recipe is the one OLS actually knows how to build; if the true relationship bends, our straight ruler will keep missing. The second says the people (or houses, or firms) in our sample were picked fairly, not cherry-picked, so that what we learn from the sample carries over to everyone else. The third says the ingredient we are studying has to actually change from case to case, and it cannot secretly be the same thing as another ingredient wearing a different label, otherwise the model cannot tell their effects apart. The fourth is the big one: everything we did not put in the model, all the stuff dumped into the leftover “surprise” term, has to be unrelated to the ingredient we are measuring; if the surprises quietly move with our ingredient, we will blame the ingredient for effects that belong to something else. The fifth says the surprises should be about the same size no matter how large the prediction is, because if the model is very precise for cheap houses but wildly off for expensive ones, our sense of how trustworthy each estimate is gets distorted. The sixth says one surprise should not tip us off about the next one; if knowing you overshot on house A tells you that you probably overshot on house B, then the cases are not really giving us independent pieces of information, and we end up thinking we know more than we do.

3. Technical Explanation (matrix-algebra audience)

Writing the model as \(y = X\beta + u\), the first assumption is that \(y\) is linear in \(\beta\); the columns of \(X\) can be transformed however we like (logs, squares), but the parameters enter additively. Random sampling gives us independent draws so the rows of \(X\) and \(u\) are i.i.d. The no-perfect-collinearity condition requires \(X\) to have full column rank, so that \(X'X\) is invertible and the estimator \(\hat{\beta} = (X'X)^{-1}X'y\) is even defined and unique. Exogeneity is the statement \(E(u \mid X) = 0\), and it is what delivers unbiasedness, since \(E(\hat{\beta} \mid X) = \beta + (X'X)^{-1}X'E(u \mid X) = \beta\) only when that conditional mean vanishes. Homoskedasticity and no serial correlation together say \(\mathrm{Var}(u \mid X) = \sigma^2 I\), a scalar times the identity matrix, which is precisely the structure the Gauss-Markov theorem needs to show that \(\hat{\beta}_{OLS}\) has the smallest variance in the class of linear unbiased estimators. When that variance matrix instead looks like \(\sigma^2 \Omega\) with a non-scalar \(\Omega\), OLS stays unbiased but is no longer efficient, and the usual \(\hat{\sigma}^2 (X'X)^{-1}\) standard errors are wrong. So the first four assumptions buy us an unbiased estimator, and the last two upgrade “unbiased” to “best (minimum variance) linear unbiased,” which is the actual content of Gauss-Markov.

4. The Data and the Regression

I used the Boston housing dataset from the MASS package, a cross-sectional dataset of 506 census tracts in the Boston area, which comfortably clears the 120-observation bar for the CLT to be doing real work.

data(Boston)
dim(Boston)          # 506 rows, well above 120
## [1] 506  14
summary(Boston[, c("medv", "lstat")])
##       medv           lstat      
##  Min.   : 5.00   Min.   : 1.73  
##  1st Qu.:17.02   1st Qu.: 6.95  
##  Median :21.20   Median :11.36  
##  Mean   :22.53   Mean   :12.65  
##  3rd Qu.:25.00   3rd Qu.:16.95  
##  Max.   :50.00   Max.   :37.97

I regressed median home value on the percentage of lower-socioeconomic-status population in the tract. The estimating equation is:

\[ medv_i = \beta_0 + \beta_1 \, lstat_i + u_i \]

and the fitted line comes out to:

\[ \widehat{medv}_i = 34.554 - 0.950 \, lstat_i \]

Units. The dependent variable medv is median home value in thousands of dollars. The independent variable lstat is measured in percentage points (the share of the tract’s population classified as lower status).

my_reg <- lm(medv ~ lstat, data = Boston)
summary(my_reg)
## 
## Call:
## lm(formula = medv ~ lstat, data = Boston)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -15.168  -3.990  -1.318   2.034  24.500 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 34.55384    0.56263   61.41   <2e-16 ***
## lstat       -0.95005    0.03873  -24.53   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 6.216 on 504 degrees of freedom
## Multiple R-squared:  0.5441, Adjusted R-squared:  0.5432 
## F-statistic: 601.6 on 1 and 504 DF,  p-value: < 2.2e-16
confint(my_reg)
##                 2.5 %     97.5 %
## (Intercept) 33.448457 35.6592247
## lstat       -1.026148 -0.8739505

Interpreting the coefficients. The slope of \(-0.950\) says that a one percentage-point increase in lstat is associated with a drop of about 0.95 thousand dollars, roughly $950, in predicted median home value. The intercept of \(34.554\) is the predicted median value (about $34,554) for a tract with lstat = 0, but since the minimum lstat in the data is 1.73, that intercept is really just anchoring the line rather than describing any real neighborhood.

Significance. Both coefficients are significant at any conventional level; the slope has a t-statistic near \(-24.5\) and a p-value below \(2\times10^{-16}\), so it clears \(\alpha = 0.001\) without any drama, and the 95% confidence interval for the slope, roughly \([-1.03, -0.87]\), comfortably excludes zero.

Economic magnitude. This is not a trivially small effect. Moving lstat up by about 10 points (roughly the spread between the 25th and 75th percentiles) predicts almost a $9,500 fall in median value, which is large against a sample mean home value of about $22,500. So the relationship is both statistically and economically meaningful.

5. The Four Diagnostic Plots

par(mfrow = c(2, 2))
plot(my_reg)

par(mfrow = c(1, 1))

What each chart measures and the rules of thumb

The Residuals vs Fitted plot puts the raw residuals against the fitted values, and its job is to check whether the straight-line form is appropriate and whether the average error stays near zero across the whole range of predictions; you want a shapeless cloud sitting around the horizontal zero line with a flat red loess curve, and a visible bend or U-shape is the tell-tale sign that the relationship is curved rather than straight. The Normal Q-Q plot orders the standardized residuals and compares them to what a normal bell curve would produce, so points hugging the diagonal mean the errors look roughly normal, while systematic drift off the line at either end flags skew or heavy tails, which is the thing that makes small-sample p-values and confidence intervals untrustworthy. The Scale-Location plot uses the square root of the absolute standardized residuals against the fitted values, which strips away the sign of the errors and isolates the single question of whether their spread is constant; a flat red line with even scatter supports constant variance, whereas an upward slope means the errors fan out as predictions grow. The Residuals vs Leverage plot puts standardized residuals against leverage and overlays Cook’s distance contours, hunting for individual points that are both unusual in their x-value and actively pulling the fitted line toward themselves; the rule of thumb is that points creeping past the dashed Cook’s distance curves (around 0.5, and definitely past 1) are influential enough to warrant investigation. Taken together, the healthy pattern is no visible structure in the first plot, points sitting on the line in the second, a flat band in the third, and no single dominating point in the fourth. When those hold, the constant-variance and correct-form pieces of Gauss-Markov are plausibly satisfied and the standard errors can be believed. When they fail, the coefficient estimates may still be unbiased, but the reported precision and any inference built on it start to wobble.

What the plots suggest for this model (are Gauss-Markov assumptions seriously violated?)

The Residuals vs Fitted plot shows a clear U-shaped curve rather than a flat band, so the straight-line form is missing real curvature in the medv-lstat relationship (a quadratic term, when I checked, is significant at p far below 0.001). The Scale-Location line trends upward, which points to non-constant error variance, so homoskedasticity looks strained too. The Q-Q plot bends away from the line at the upper tail, meaning the residuals are right-skewed rather than normal, which is not a Gauss-Markov assumption itself but does undercut the usual inference. So yes, the functional-form (linearity) and constant-variance conditions both look seriously violated here, while no single observation is wildly influential (the largest Cook’s distance sits well under 0.5).

Trying a transformation

Swapping lstat for log(lstat) lifts the R-squared from 0.544 to 0.665 and visibly straightens out the curve in the Residuals vs Fitted plot, so the fit clearly improves.

log_reg <- lm(medv ~ log(lstat), data = Boston)
summary(log_reg)$r.squared        # 0.665 vs 0.544 for the level model
## [1] 0.6649462

The cost is interpretability: the slope now describes how medv responds to percent changes in lstat rather than one-point changes, so a 1% rise in lstat maps to roughly a $125 fall in value, which is a less intuitive thing to say out loud. Even so, the log model is the better-behaved one here, and the trade of a cleaner residual plot for a slightly clumsier sentence feels worth it.

One thing I am still chewing on: I am not sure whether the log transform is genuinely fixing the form of the relationship or just papering over an omitted-variable problem. A tract’s lstat is surely tangled up with things I left out, like average number of rooms or crime rate, and if those are what is really driving the curvature and the fanning variance, then the log is treating a symptom rather than the cause. I would be curious whether adding one or two of those regressors flattens the residual plots on its own, without any transformation at all.