4.1 Multiple Regression

Run the regression: Test Score explained by Student-Teacher Ratio (STR) and percentage of English learners. Test the hypothesis that a change in STR has no effect on test scores once we control for percentage of English learners.

library("AER")
data("CASchools")
attach(CASchools)
head(CASchools)

teacher_stu_ratio <- CASchools$students/CASchools$teachers
model1<-lm(CASchools$math~teacher_stu_ratio+CASchools$english)
df_eng<-data.frame(CASchools[,c(12,14)],teacher_stu_ratio)  
plot(df_eng, pch=10, col="blue", main="All Correlations Presented")

summary(model1)

Call:
lm(formula = CASchools$math ~ teacher_stu_ratio + CASchools$english)

Residuals:
    Min      1Q  Median      3Q     Max 
-48.230 -11.579   0.040   9.794  48.290 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)       680.1894     7.8751  86.373   <2e-16 ***
teacher_stu_ratio  -0.9129     0.4041  -2.259   0.0244 *  
CASchools$english  -0.5655     0.0418 -13.528   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 15.37 on 417 degrees of freedom
Multiple R-squared:  0.3316,    Adjusted R-squared:  0.3284 
F-statistic: 103.4 on 2 and 417 DF,  p-value: < 2.2e-16

We see the STR is significant with .024 P value and -2.25 T test.The estimated coefficient is ~ -0.9. The statistic can be computed:

(-0.9-0) /  0.4

[1] -2.25

Construct a 90% confidence interval for the coefficient on STR in the above model.

-0.9+1.64*0.4

[1] -0.244

-0.9-1.64*0.4

[1] -1.556

Add the independent variable expenditures per pupil to the above model.

model2<-lm(CASchools$math~teacher_stu_ratio+CASchools$english+CASchools$expenditure)

Note the effect of STR on test scores before and after adding the expenditures in the district.

summary(model2)

Call:
lm(formula = CASchools$math ~ teacher_stu_ratio + CASchools$english + 
    CASchools$expenditure)

Residuals:
    Min      1Q  Median      3Q     Max 
-50.051 -11.339  -0.419  10.324  48.086 

Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
(Intercept)           653.590525  16.234014  40.261   <2e-16 ***
teacher_stu_ratio      -0.318301   0.513019  -0.620   0.5353    
CASchools$english      -0.570080   0.041750 -13.654   <2e-16 ***
CASchools$expenditure   0.002822   0.001508   1.872   0.0619 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 15.32 on 416 degrees of freedom
Multiple R-squared:  0.3372,    Adjusted R-squared:  0.3324 
F-statistic: 70.54 on 3 and 416 DF,  p-value: < 2.2e-16

It’s no longer significant!

What is the effect on standard errors when expenditure was added?

Standard erorr was incraesed from .4 to .51.

Test the joint hypothesis that neither STR nor expenditure per pupil have an effect on test scores.

model3<-lm(CASchools$math~teacher_stu_ratio+CASchools$expenditure)
summary(model3)

Call:
lm(formula = CASchools$math ~ teacher_stu_ratio + CASchools$expenditure)

Residuals:
    Min      1Q  Median      3Q     Max 
-45.549 -13.332  -0.546  12.872  52.614 

Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
(Intercept)           676.183721  19.411041  34.835  < 2e-16 ***
teacher_stu_ratio      -1.601647   0.606192  -2.642  0.00855 ** 
CASchools$expenditure   0.001622   0.001809   0.897  0.37048    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 18.42 on 417 degrees of freedom
Multiple R-squared:  0.04009,   Adjusted R-squared:  0.03549 
F-statistic: 8.708 on 2 and 417 DF,  p-value: 0.0001972

Since we expect both coefficients to be 0 but STR is not. #### Why can’t you just test the individual coefficients one at a time to answer this question? One at a time test is not like the joint hypothesis test since it requires multiply tests that actually change the .5 threshold to accept false results.

What happens if the regressors are correlated? How can you use the adjusted R-squared to specify a good model in case of such correlation?

We use adjusted R squared?

“Because the variance of the sampling distribution of the regression coefficient would be larger (by a factor of the VIF) if it were correlated with other variables in the model, the p-values would be higher (i.e., less significant) than they otherwise would. The variances of the regression coefficients would be larger, as already discussed. In general, this is hard to know without solving for the model. Typically, if only one of two is significant, it will be the one that had the stronger bivariate correlation with YY. How the predicted values and their variance would change is quite complicated. It depends on how strongly correlated the variables are and the manner in which they appear to be associated with your response variable in your data. Regarding this issue, it may help you to read my answer here: Is there a difference between ‘controlling for’ and ‘ignoring’ other variables in multiple regression?”

Use the F-statistic to test the joint hypothesis. Calculate the rule of thumb F statistic. Why is it different from the F statistic?

model3<-lm(CASchools$math~teacher_stu_ratio+CASchools$expenditure)
model4<-lm(CASchools$math~teacher_stu_ratio*0+CASchools$expenditure*0)
par(mfrow = c(2, 2))
plot(model3)

par(mfrow = c(2, 2))
plot(model4)

ssr_dif<- (sum((residuals(model4))^2)-sum((residuals(model3))^2)) / 2
ssr_dif

[1] 205829.1

ssr_norm<-(sum((residuals(model3))^2))
denom <- (420-2-1)
ssr_norm / denom

[1] 339.2384

ssr_dif/ssr_norm

[1] 606.7388

Check:

qf(.95, df1 = 417,2)

[1] 19.49333

Differences probably because of homoskadasticity issues.