POL 682: Linear Regression Analysis
University of Arizona
School of Government and Public Policy
2026-02-02
This lecture covers the Gauss-Markov assumptions that underlie the Ordinary Least Squares (OLS) estimator.
Population Regression Function (PRF)
\[Y_i = \alpha + \beta X_i + \epsilon_i\]
Sample Regression Function (SRF)
\[Y_i = a + b X_i + e_i\]
We rarely observe the PRF (the exception being a census). Instead, we observe the SRF and use \(a\) and \(b\) as point estimates of \(\alpha\) and \(\beta\).
Because our estimates are subject to sampling error, point estimates should always accompany an indicator of uncertainty.
If we make some assumptions about the PRF, the OLS estimator has several desirable properties.
The PRF is linear in parameters.
\[Y_i = \alpha + \beta X_i + \epsilon_i\]
This assumption is necessary for estimation and we’ll used to demonstrate unbiasedness.
The Xs are exogenous and fixed.
\[cov(X_i, \epsilon_i) = 0\]
This assumption is necessary for estimation and is used to demonstrate unbiasedness.
Regardless of \(X_i\), the error has a zero mean: \(E(\epsilon_i) = 0\)
\[ \begin{aligned} E(\epsilon_i | X_i) &= 0\\ E(\epsilon_i) &= 0 \end{aligned} \]
There is no systematic effect of the error on our predicted values \(\hat{Y_i}\).
This assumption is necessary for inference.
Regardless of \(X_i\), the error has equal variance: \(var(\epsilon_i) = \sigma^2\)
\[ \begin{aligned} var(\epsilon_i | X_i) &= \sigma^2, \quad \forall i\\ var(\epsilon_i) &= \sigma^2, \quad \forall i \end{aligned} \]
The variance around \(\hat{Y_i}\) is the same across all levels of \(X_i\).
This assumption is necessary for inference.
\[\epsilon_i \sim N(0, \sigma)\]
Then, \(Y_i \sim N(\alpha + \beta X_i, \sigma)\)
This is an extension of assumptions 3 and 4. If we assume normality, the OLS estimator has minimum variance among all unbiased estimators.
This assumption is necessary for inference.
\[cov(\epsilon_i, \epsilon_j) = 0, \quad \forall i \neq j\]
The ith error term is uncorrelated with the jth error term, for all \(i \neq j\).
This is the no autocorrelation assumption. With the exception of the relationship between \(Y\)s determined by \(X\), there is no residual relationship between \(Y\) variables.
This assumption is necessary for inference.
Assumption 7: Variation in \(X\)
Assumption 8: Correct Specification
Assumption 9: No Perfect Multicollinearity
| Assumption | Description | Required For |
|---|---|---|
| 1. Linearity | PRF linear in parameters | Estimation |
| 2. Exogeneity | \(cov(X_i, \epsilon_i) = 0\) | Estimation |
| 3. Zero Mean | \(E(\epsilon_i) = 0\) | Inference |
| 4. Homoskedasticity | \(var(\epsilon_i) = \sigma^2\) | Inference |
| 5. Normality | \(\epsilon_i \sim N(0, \sigma)\) | Inference |
| 6. Independence | \(cov(\epsilon_i, \epsilon_j) = 0\) | Inference |
| 7. Variation in X | \(var(X) > 0\) | Estimation |
| 8. Correct Specification | Functional form, IVs | Estimation |
| 9. No Multicollinearity | \(r_{x_1,x_2} \neq 1\) | Estimation |
The Gauss-Markov Theorem states:
The Ordinary Least Squares (OLS) estimator is the Best Linear Unbiased Estimator (OLS is BLUE) with minimum variance of all unbiased linear estimators.
Important: “Best” means something specific as a statistical characteristic, but does not mean ideal or advisable in all circumstances, even when the GM assumptions hold.
When the GM assumptions hold, we can demonstrate:
This is important for efficiency—the estimators have minimum sampling variance and smallest error.
\[\begin{bmatrix} y_{1}= & b_0+ b_1 x_{1}+ e_1\\ y_{2}= & b_0+ b_1 x_{2}+ e_2\\ y_{3}= & b_0+ b_1 x_{3}+ e_3\\ \vdots\\ y_{n}= & b_0+ b_1 x_{n} + e_n \end{bmatrix}\]
For each \(y_i\), it is a composite of:
\[ \begin{aligned} b &= \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2}\\ &= \frac{\sum (X_i - \bar{X})Y_i - \bar{Y}\sum(X_i - \bar{X})}{\sum (X_i - \bar{X})^2}\\ &= \frac{\sum (X_i - \bar{X})Y_i}{\sum (X_i - \bar{X})^2} \end{aligned} \]
Notice: \(\bar{Y}\) disappears because deviations from the mean always sum to zero.
\[ b = \sum k_i Y_i, \text{ where } k_i = \frac{(X_i - \bar{X})}{\sum (X_i - \bar{X})^2} \]
The OLS estimator \(b\) is a linear function of \(Y_i\) with weights \(k_i\).
The weights \(k_i\) have four important properties:
| Property | Statement | Why it matters |
|---|---|---|
| 1. | \(\sum k_i = 0\) | Eliminates \(\alpha\) when proving \(E(b) = \beta\) |
| 2. | \(\sum k_i X_i = 1\) | Ensures the coefficient on \(\beta\) equals 1 |
| 3. | \(\sum k_i^2 = \frac{1}{\sum(X_i - \bar{X})^2}\) | Used to derive \(var(b)\) |
| 4. | \(k_i\) are constants | Because \(X\) is fixed (A2), we can factor \(k_i\) out of expectations |
\[ \sum k_i = \sum \frac{(X_i - \bar{X})}{\sum (X_j - \bar{X})^2} = \frac{\sum(X_i - \bar{X})}{\sum (X_j - \bar{X})^2} = \frac{0}{\sum (X_j - \bar{X})^2} = 0 \]
The numerator equals zero because deviations from the mean always sum to zero:
\[\sum(X_i - \bar{X}) = \sum X_i - n\bar{X} = n\bar{X} - n\bar{X} = 0\]
\[ \sum k_i X_i = \frac{\sum(X_i - \bar{X})X_i}{\sum (X_j - \bar{X})^2} \]
Decompose \(X_i = (X_i - \bar{X}) + \bar{X}\):
\[ \begin{aligned} \sum(X_i - \bar{X})X_i &= \sum(X_i - \bar{X})^2 + \bar{X}\underbrace{\sum(X_i - \bar{X})}_{=0}\\ &= \sum(X_i - \bar{X})^2 \end{aligned} \]
Therefore: \(\sum k_i X_i = \frac{\sum(X_i - \bar{X})^2}{\sum (X_j - \bar{X})^2} = 1\)
\[ \sum k_i^2 = \sum \left[\frac{(X_i - \bar{X})}{\sum(X_j - \bar{X})^2}\right]^2 = \frac{\sum(X_i - \bar{X})^2}{\left[\sum(X_j - \bar{X})^2\right]^2} = \frac{1}{\sum(X_i - \bar{X})^2} \]
This property is crucial for deriving the variance of \(b\).
[,1] [,2]
[1,] 5.123798 14.825352
[2,] 2.703320 9.064260
[3,] 6.960291 10.375307
[4,] 3.169354 13.111677
[5,] 6.525794 10.725355
[6,] 4.700879 9.762086
Call:
lm(formula = Y ~ X)
Coefficients:
(Intercept) X
5.1900 0.9368 Property 1: sum(k_i) = -1.292369e-16 Property 2: sum(k_i * X_i) = 1 Property 3: sum(k_i^2) = 0.002765756 1/sum(x_i^2) = 0.002765756
Slope from sum(k_i * Y_i): 0.9367911 Slope from lm(): 0.9367911 The simulation confirms:
lm()Example Usage
Recall that \(a = \bar{Y} - b\bar{X}\). Substituting \(b = \sum k_i Y_i\):
\[ \begin{aligned} a &= \frac{1}{n}\sum Y_i - \bar{X} \sum k_i Y_i\\ a &= \sum \left(\frac{1}{n} - \bar{X} k_i\right) Y_i\\ a &= \sum c_i Y_i \end{aligned} \]
where \(c_i = \frac{1}{n} - \bar{X} \cdot \frac{(X_i - \bar{X})}{\sum (X_i - \bar{X})^2}\)
The OLS estimator \(a\) is also a linear function of \(Y_i\) with weights \(c_i\).
These are important results because they make subsequent operations more tractable:
\(b\) is an unbiased estimator of \(\beta\).
In words: The expected value of \(b\) equals the population parameter \(\beta\).
More intuitive: The average value of \(b\) across repeated samples equals the population parameter \(\beta\).
\[ \begin{aligned} b &= \sum k_i Y_i\\ b &= \sum k_i (\alpha+\beta X_i+\epsilon_i)\\ b &= \alpha \sum k_i + \beta \sum k_i X_i + \sum k_i \epsilon_i\\ b &= \alpha \cdot 0 + \beta \cdot 1 + \sum k_i \epsilon_i\\ b &= \beta+\sum k_i \epsilon_i\\ E(b) &= \beta \end{aligned} \]
Constants (fixed, known values):
Random Variables (vary across samples):
Why this matters: Constants can be factored out of expectations: \(E(cX) = cE(X)\)
The simulation confirms unbiasedness:
Used in this derivation:
But what about the variance? We’ve shown unbiasedness, but how much does \(b\) vary from sample to sample?
\[ var(b) = E\left[(b - E(b))^2\right] \]
The variance measures how spread out the sampling distribution of \(b\) is around its expected value.
We established that \(b = \beta + \sum k_i \epsilon_i\). Since \(E(b) = \beta\):
\[ b - E(b) = b - \beta = \sum k_i \epsilon_i \]
Therefore:
\[ \begin{aligned} var(b) &= E\left[(b - \beta)^2\right]\\ &= E\left[(\sum_i k_i \epsilon_i)^2\right] \end{aligned} \]
\[ \begin{aligned} var(b) &= E\left[(\sum_i k_i \epsilon_i)(\sum_j k_j \epsilon_j)\right]\\ &= E\left[\sum_i \sum_j k_i k_j \epsilon_i \epsilon_j\right] \end{aligned} \]
The inside of the expectation is a double summation over all \(i\) and \(j\). It forms a matrix of terms, each weighted by \(k_i k_j\) and involving the product \(\epsilon_i \epsilon_j\).
We now use two assumptions about the error terms:
Covariance measures how two random variables move together:
\[cov(X, Y) = E[(X - E(X))(Y - E(Y))] = E(XY) - E(X)E(Y)\]
Key rearrangement: \(E(XY) = cov(X, Y) + E(X)E(Y)\)
Variance is the covariance of a variable with itself:
\[var(X) = cov(X, X) = E(X^2) - [E(X)]^2\]
Important: If \(E(X) = 0\), then \(E(X^2) = var(X)\)
The expectation of the double sum consists of:
Off-diagonal (\(i \neq j\)): \[E(\epsilon_i \epsilon_j) = cov(\epsilon_i, \epsilon_j) + E(\epsilon_i)E(\epsilon_j) = 0 + 0 = 0\] (by A6 and A3)
Diagonal (\(i = j\)): \[E(\epsilon_i^2) = var(\epsilon_i) + [E(\epsilon_i)]^2 = \sigma^2 + 0 = \sigma^2\] (by A4 and A3)
So only the diagonal terms (\(i = j\)) remain:
\[ \begin{aligned} var(b) &= E\left[\sum_i k_i^2 \epsilon_i^2\right]\\ &= \sum_i k_i^2 E(\epsilon_i^2)\\ &= \sum_i k_i^2 \cdot \sigma^2\\ &= \sigma^2 \sum k_i^2 \end{aligned} \]
\[var(b) = \frac{\sigma^2}{\sum(X_i - \bar{X})^2}\]
This formula reveals what affects the precision of slope estimates:
We learn more about \(E(Y|X) = \beta\) when \(X_i\) varies.
Consider what we have:
\[var(b) = \frac{\sigma^2}{\sum x_i^2}\]
But there are two problems:
It can be shown that \(\hat{\sigma^2}\) is an unbiased estimator of \(\sigma^2\) (i.e., \(E(\hat{\sigma}^2) = \sigma^2\)).
\[ \begin{aligned} \hat{\sigma^2} &= RSS/(n-k-1)\\ &= \sum e_i^2/(n-k-1) \end{aligned} \]
And,
\[ \begin{aligned} var(b) &= \frac{\hat{\sigma^2}}{\sum x_i^2}\\ var(a) &= \frac{\hat{\sigma^2} \sum X_i^2}{n\sum x_i^2} \end{aligned} \]
To show minimum variance, let’s generate an alternate estimate of \(b\), call it \(b^*\).
\[ \begin{aligned} b &= \sum k_i Y_i\\ b^* &= \sum w_i Y_i \end{aligned} \]
If \(b^*\) is unbiased, then \(\sum w_i = 0\), and \(\sum w_i X_i = 1\).
Even without knowing these weights, we can show:
\[ var(b^*) = \sigma^2 \left(\sum w_i - \frac{x_i}{\sum x_i}\right)^2 + \frac{\sigma^2}{\sum x_i^2} \]
Consider what this means:
Under the Gauss-Markov assumptions:
OLS is the Best Linear Unbiased Estimator (BLUE)
POL 682 | The Gauss-Markov Assumptions