Chapter 1 - Omitted Variables and R-squared
- Remember MLR/Gauss-Markov assumptions:
- Linear Parameters (MLR1)
- Random Samples (MLR2)
- No perfect co-linearity (MLR3)
- No correlation for all unobserved factors (MLR4)
- \(E(u|X_n) = 0\)
- MLR 4 will fail many times
- Homoskedasticity
- \(\textit{Var}(u|X_n) =
\sigma^2\)
- MLR5 will also fail many times
- Generally make the assumption \(E(\beta_j)
= \beta_j\)
- The expected value of \(\beta_j\)
is the acutal value, i.e., our error term u is 0.
- How can we determine omitted variable bias?
True Model
- Population Model that we assume MLR 1-5 hold true.
\[Y = \beta_0 + \beta_1X_1 +
\beta_2X_2+\beta_3X_3 +u\]
- We can run a restricted model that omits the variable we
think is biased. For example, assume \(X_2\) is our omitted variable:
True Model: \[Y = \beta_0 + \beta_1X_1 +
\beta_2X_2+\beta_3X_3 +u\] Restricted Model: \[Y = \gamma_0 + \gamma_1X_1 +
\textit{omitted}+\gamma_3X_3 +\theta\]
The coefficient and error term symbol changes since our beta
estimates change when we omit \(X_2\).
- The bias is found in the difference between \(\beta_1\) and \(\gamma_1\) or \(\beta_3\) and \(\gamma_3\)
- \(\gamma_1\) or \(\gamma_3\) is the biased
estimator.
Bias Formula
\[E(\hat{Y}) = \beta_1
+\beta_2\times\frac{\textit{covariance}(X_1,X_2)}{\textit{variance}(X_1)}\]
Where \(\beta_2
\times\frac{covariance(X_1,X_2)}{variance(X_1)}\) is the
bias term.
- The signs of \(\beta_2\) and the
covariance ‘sign’ the bias:
- What’s the correlation between the omitted variable and the
dependent variable?
- What’s the correlation between the omitted variable and the other
independent variable of interest?
- Positive signage \(\rightarrow\)
upward bias
- Negative signage \(\rightarrow\)
downward bias
- 0 \(\rightarrow\) no
bias
- We can also use an auxiliary model to sign the bias:
True Model:
\[Y = \beta_0 + \beta_1X_1 +
\beta_2X_2+\beta_3X_3 +u\]
Restricted Model: \[Y = \gamma_0 +
\gamma_1X_1 + \textit{omitted}+\gamma_3X_3 +\theta\]
Auxiliary Model:
\[X_2 = \delta_0 + \delta_1X_1 +
\delta_3X_3 + \tau\]
- When we ‘plug’ the Auxiliary model into the ‘true’ model, we derive
the formula for signing the bias:
\[Y = \beta_0 + \beta_1X_1 +
\beta_2(\delta_0 + \delta_1X_1 + \delta_3X_3 + \tau)+\beta_3X_3
+u\]
\[Y=\beta_0 +\beta_1X_1 + \beta_2\delta_0
+ \beta_2\delta_1X_1 + \beta_2\delta_3X_3 + \beta_2\tau + \beta_3X_3
+u\]
- Grouping terms based on the position of the \(\delta\) and \(\gamma\) coefficient:
\[Y = (\beta_0 + \beta_2\delta_0) +
(\beta_1X_1+\beta_2\delta_1X_1) +
(\beta_2X_3\delta_3+\beta_3X_3)+(\beta_2\tau + u)\]
Using the auxiliary model, we assume:
- \((\beta_0 + \beta_2\delta_0) =
\gamma_0\)
- \((\beta_1X_1+\beta_2\delta_1X_1) =
\gamma_1X_1\)
- \((\beta_2X_3\delta_3+\beta_3X_3) =
\gamma_3X_3\)
- \((\beta_2\tau + u) = \theta\)
R-squared
\(R^2\) explains variation in data
\(\rightarrow\) Our independent
variables can explain \(100\times R^2\)
percent of the variation in the dependent variable.
\(R^2\) Formulae:
\[R^2
=\frac{\sum{(\hat{y}-\bar{y})^2}}{\sum{({y}-\bar{y})^2}} =
\frac{\textit{sum of square errors }}{\textit{sum of square totals }} =
\frac{SSE}{SST}\]
\[R^2 = 1 - \frac{SSR}{SST} =1-
\frac{\textit{sum of square residuals}}{\textit{sum of square totals }}
= 1-\frac{\sum({y-\hat{y})^2}}{\sum{({y}-\bar{y})^2}}\]
With Multi Linear
Relationships (MLR), we cannot assume
\(R^2 = r^2\) due to the relationship
between \(X_1\) and \(X_2\)
\(R^2\) increases as we
add more independent variables
- Cannot trust \(R^2\) as sole
indicator of a ‘good’ model
- Use adjusted \(R^2\) (\(\bar{R^2}\)) as better indicator of ‘good’
model
- \(\bar{R^2} = 1 - \frac{SSR}{n-k-1}
\times\frac{n-1}{SST}\)
- n is sample size, k is number of variables
Four things to realize about \(R^2\):
- A high \(R^2\) does not
satisfy conclusions
- A high \(R^2\) does not
imply ‘correct’ functional form
- A high \(R^2\) does not
imply a casual relationship
- A high \(R^2\) does not
mean you have the correct variables
Chapter 3 - Hypothesis Testing
- Hypothesis tests are tests of likelihoods
- We can use different distributions to get a probability of
likelihood reflected in a test statistic
- Common distributions: z distribution (normal distribution),
t distribution, \(\chi^2\),
F distribution
- Note: when n > ~120; the t distribution is
equivalent to the z distribution.
- When we know population standard deviation, use z-score ( and thus
normal distribution)
- When we do not know population standard deviation, use
t-score (slightly normal distribution)
- Degrees of Freedom (df) is \(n-1\)
- Use t-table
\[z= \frac{\hat{\beta_n} -
\beta_n}{\textit{standard deviation}(\beta_n)} =
\frac{\textit{predicted} - \textit{actual}}{\textit{standard
deviation}(\beta_n)}\]
\[t= \frac{\hat{\beta_n} -
\beta_n}{\textit{standard error}(\beta_n)} = \frac{\textit{predicted} -
\textit{actual}}{\textit{standard error}(\beta_n)}\]
IMPORTANT: Need MLR6 (error terms are normally
distributed) to conduct inference test on \(\beta_n\) (estimates), but not
means.
- Different hypotheses for:
- \(H_0\) - null hypothesis
- What we assume
- can be single parameter (t-test), or multi-parameter (F-test)
- Ex: \(H_0: \beta_1 = \beta_2\)
- \(H_{alt}\) - alternative
hypothesis
- What we are looking for
- The negation of \(H_0\)
- Ex: \(H_{alt}: \beta_1 \ne
\beta_2\)
- There are different hypothesis test statistics for different
statistical purposes:
- z-test/t-test
- Single parameter test for means or some value of x
- Can be one-tailed or two-tailed
- One-tailed indicates a direction; rarely used since we generally
don’t assume we know the direction
- \(H_0: \beta_1 > x; H_{alt}: \beta_1
< x\)
- Two-tailed does not have direction; only looks for distance from
critical test statistic value (z-crit,t-crit,
F-crit, etc.)
- \(H_0: \beta_1 = x; H_{alt}: \beta_1 \ne
x\)
- Common Critical Values(assuming \(n \rightarrow \infty\))
- z and t Tests (two-tailed)
- 90 percent confidence level: 1.645
- 95 percent confidence level: 1.96
- 99 percent confidence level: 2.58
- F Test (\(df_n = 2\))
- 90 percent confidence level: 2.3
- 95 percent confidence level: 3
- 99 percent confidence level: 4.61
- \(\chi^2\) Test (\(df = 3\))
- 90 percent confidence level: 6.25
- 95 percent confidence level: 7.81
- 99 percent confidence level: 11.34
- Generally want to calculate a test statistic that is greater than
the above values to reject \(H_0\)
- This is not always what we want though…
- F test
- Joint test of significance
- Tests whether the variables of interest affect the variation of the
dependent value
- Can be multi-parameter (ex: \(H_0: \beta_1
= 0 \cap \beta_2 = 10\))
- Note: when an F-test only as a
single parameter, the F-stat calculated is the square
of the equivalent t-stat (\(t_{stat} =
\sqrt{F_{stat}}\))
- Null hypothesis is always and; i.e., all
restrictions in \(H_0\) must be
true .
- The alternative hypothesis is XOR, at least one restriction in \(H_0\) must fail in order to reject \(H_0\)
- \(H_{alt}:\beta_1 \ne 0 \cup \beta_2 \ne
10\)
- When number of restrictions is equal to number of variables (\(q=k\)) and the
restrictions all test for any effect (\(\beta_n =0\)), this is an overall
F-test
- Cannot indicate direction in F-test,
- Can only test linear relationships (constant slope)
- The more restrictions, the worse our restricted model
gets
- F-tests are always negative
- Conducting an F-test
- Sum of Square Totals for unrestricted and restricted model must be
equal to conduct an F-test
- \[F = \frac{\frac{SSR_{restricted} -
SSR_{unrestricted}}{q}}{\frac{SSR_{unrestricted}}{n-k-1}}\]
- Where q is the number of restrictions (count the number of
equal signs in \(H_0\)), n is
the number of observations, and k is the number of variables in
the unrestricted model.
- You can also used the \(R^2\)
formula to calculate an F-stat:
- \(F = \frac{\frac{R^2_{unrestricted}
-R^2_{restricted}}{q}}{\frac{1-R^2_{unrestricted}}{n-k-1}}\)
- Note: You cannot use \(\bar{R^2}\) to calculate an
F-stat
- Chow Test is a version of the F-test
- tests whether two groups should be regressed together or separately
- \(H_0: \beta_0 = \gamma_0 \cap \beta_1
=\gamma_1 \cap\beta_n = \gamma_n \rightarrow\) the slopes and
intercepts of the two groups are the same
- \(H_{alt}: \beta_0 \ne \gamma_0 \cup
\beta_1 \ne\gamma_1 \cup\beta_n \ne \gamma_n \rightarrow\) the
slopes and intercepts of the two groups are not the same
- Uses different F-stat formula:
- \(F = \frac{SSR_{pooled}- (SSR_1 +
SSR_2)}{(SSR_1 + SSR_2)} \times\frac{n-2(k+1)}{k+1}\)
- Need at least one indicator variable to conduct a Chow test
- Pooled model, model for group 1, and model for group 2 do
not include the indicator variable
- Can also use a fully-interacted model
- Model where all independent variables are multiplied by the
indicator variable and includes the indicator variable of
interest in the model
- \(Y = \beta_0 + \beta_1X_1 + \beta_2X_2
+\beta_3X_3+\beta_4[indicator]+\beta_5X_1\times[indicator]+...+\beta_nX_n\times[indicator]\)
- \(\chi^2\) tests
- Can test association and independence of two variable groups
- Not really covered in Quant II, but the \(\chi^2\) distribution is necessary for our
logit and probit model later on,tests for heteroskedasticity, and test
like ANOVA (Analysis of
Variance).