Today we covered the different applications of t-tests and F-tests in multiple linear regression. We also covered partial F-tests that is useful in testing a subset of predictors. The t-tests will give information on whether an individual predictor is significant or not, while the F-test tests for the existence of a linear relationship between the response and at least one predictor.

Below is the code for the water runoff example. The response is the amount of stream runoff near Bishop, CA (given in acre-feet), and the predictors are snowfall amounts (in inches) at various measurement sites.

library(alr3)
## Warning: package 'alr3' was built under R version 3.4.3
## Loading required package: car
## Warning: package 'car' was built under R version 3.4.3
data(water)
attach(water)
watermod_full <- lm(BSAAM ~ APMAM + APSAB + APSLAKE + OPBPC + OPRC + OPSLAKE)
summary(watermod_full)
## 
## Call:
## lm(formula = BSAAM ~ APMAM + APSAB + APSLAKE + OPBPC + OPRC + 
##     OPSLAKE)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -12690  -4936  -1424   4173  18542 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 15944.67    4099.80   3.889 0.000416 ***
## APMAM         -12.77     708.89  -0.018 0.985725    
## APSAB        -664.41    1522.89  -0.436 0.665237    
## APSLAKE      2270.68    1341.29   1.693 0.099112 .  
## OPBPC          69.70     461.69   0.151 0.880839    
## OPRC         1916.45     641.36   2.988 0.005031 ** 
## OPSLAKE      2211.58     752.69   2.938 0.005729 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7557 on 36 degrees of freedom
## Multiple R-squared:  0.9248, Adjusted R-squared:  0.9123 
## F-statistic: 73.82 on 6 and 36 DF,  p-value: < 2.2e-16

If we were to run a t-test on the “OPBPC” predictor, we would fail to reject \(H_0\). There is not sufficient evidence to suggest that after accounting for all other predictors, there is a linear relationship between runoff and the snowfall at this measurement site. Thus, we could drop this predictor from our model.

The t-tests for multiple linear regression are similar to the ones for simple linear regression, with a minor difference in interpretation.

The F-test would be testing all of the predictors with the following hypotheses:

\(H_0\): all \(\beta_i\) for \(i = 1, ... k\) are equal to zero.

\(H_a\): at least one \(\beta_i \neq 0\)

The F-test statistic is 73.82, and follows an F-distribution with 6 and 36 degrees of freedom. The P-value is virutally zero, so we reject \(H_0\) in favor of \(H_a\). There is strong evidence to suggest that the snowfall at at least one of the sites helps predict runoff.

If an F-test fails to reject \(H_0\), then we can drop all of the predictors. I feel like I have a good understanding of the difference between t-tests and F-tests. It did take some careful reading of what the questions in the in-class examples were asking to figure out which test to use.

Testing Multiple predictors

The t-test can only test one predictor at a time, while the F-tests can only test all of the predictors. In order to test a group, or subset, of predictors, we need to run a partial F-test. First, we must create a reduced model.

The multiple linear regression model with all of the predictors is known as the complete model. The reduced model has only some of the predictors from the complete model. When we create this model, we will exclude the predictors we are considering dropping. Then we run an F-test on this reduced model.

Doing the partial F-test using the runoff example was useful in helping me understand how to set up the reduced model. We are testing to see if we can drop all measurement sites starting with the letter “O”, so we create a reduced model that excludes the predictors we are testing. Once we create both models, we use the anova function to run the F-test.

watermod_full <- lm(BSAAM ~ APMAM + APSAB + APSLAKE + OPBPC + OPRC + OPSLAKE)
watermod_reduced <- lm(BSAAM ~ APMAM + APSAB + APSLAKE)
anova(watermod_full , watermod_reduced)
## Analysis of Variance Table
## 
## Model 1: BSAAM ~ APMAM + APSAB + APSLAKE + OPBPC + OPRC + OPSLAKE
## Model 2: BSAAM ~ APMAM + APSAB + APSLAKE
##   Res.Df        RSS Df  Sum of Sq     F    Pr(>F)    
## 1     36 2.0558e+09                                  
## 2     39 2.5116e+10 -3 -2.306e+10 134.6 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

If we drop good predictors when creating our reduced model, our predictions will be farther from the observed values, and the RSS of our reduced model will be larger than the RSS of the complete model. This gives a larger F-test statistic, resulting in a smaller p-value and thus we are more inclined to reject the null hypothesis.

If, on the other hand, we drop useless or poor predictors, our predictions will not be very different from observed values, which gives a smaller difference between the RSS’s of the reduced and complete models. This gives a smaller F-test statistic and in turn a larger p-value. We would be less inclined to reject the null hypothesis.

The test stat is 0 when our reduced model is the same as our complete model. If our reduced model only drops one predictor, then we are essentially just doing the t-test.

From the output, we see the p-value is virtually 0. We reject the null hypothesis (the coefficients for these predictors are all 0) in favor of the alternate (at least one of the coefficients are not 0). In plain words, at least one of “O” predictors has a significant linear relationship with runoff. We should keep them in our model.

The partial F-test requires a few additional steps, but overall I found it fairly easy to understand after doing few examples.