Assignment 2: Linear Regression
Rahab Lawshea
2/10/2022
2. Carefully explain the difference between the KNN classifier and KNN regression methods
The KNN regression method uses a given value for K and a prediction point to identify the training observations that are closest to the prediction point. The regression then estimates f(x_{0}) using the average of all the training responses.
The KNN classifier differs in that it predicts the classification by looking at the k nearest neighbors around the observation. Then outputting the most frequent classification.
The main difference is the KNN classifier method assumes the outcome based on the frequency of the observations. Whereas in the regression approach, the outcome is based on the average value of the nearest neighbors.
9. This question involves the use of multiple linear regression on the Auto data set.
(a) Produce a scatterplot matrix which includes all of the variables in the data set.
#load libraries
library(MASS)
library(ISLR)## Warning: package 'ISLR' was built under R version 4.0.5?Auto## starting httpd help server ... done#scatterplot matrix
pairs(Auto)
(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative.
cor(Auto[,1:8])## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000re_auto = Auto[,1:8](c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output.
lm.fit=lm(mpg ~., data=re_auto)
lm.fit##
## Call:
## lm(formula = mpg ~ ., data = re_auto)
##
## Coefficients:
## (Intercept) cylinders displacement horsepower weight
## -17.218435 -0.493376 0.019896 -0.016951 -0.006474
## acceleration year origin
## 0.080576 0.750773 1.426140summary(lm.fit)##
## Call:
## lm(formula = mpg ~ ., data = re_auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the response?
Based on the R-squared value of 0.8215 we can say that 82.15% of the variation can be explained by the predictors in the model. We can also note that the F-statistic’s p-value is less than 2.2e-16 which is statistically significant. This shows that at least one predictor in the the model has a significant relationship to the response variable.
ii. Which predictors appear to have a statistically significant relationship to the response?
Based on the p-values produced, we can see that displacement, weight, year, and origin are significant to the model.
iii. What does the coefficient for the year variable suggest?
The coefficient for the year variable suggests that for every one year increase we will see a 0.750 increase in the mpg.
(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow=c(2,2))
plot(lm.fit) 
When we look at the Residuals vs.Fitted plot we’ll notice that the data isn’t linear. We can also note that there is non-constant variance or heteroscedasticity because of the funnel shape near the end of Residuals vs Fitted plot.
From the Normal Q-Q plot we can note that the data is mostly normally distributed but it is slightly skewed.
plot( predict(lm.fit),rstudent(lm.fit))
plot(hatvalues (lm.fit))
Using the rstudent function we can confirm that there are outliers since there are studentized residuals greater than 3.
Taking a closer look at the leverage plot using the hatvalues function we can see that there are some high leverage points in the data.
(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
lm.fit2 <- lm(mpg ~ cylinders * displacement + horsepower * weight + acceleration * year+displacement*weight, data = re_auto)
summary(lm.fit2)##
## Call:
## lm(formula = mpg ~ cylinders * displacement + horsepower * weight +
## acceleration * year + displacement * weight, data = re_auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.4358 -1.5485 0.0925 1.3506 11.6691
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.175e+02 1.849e+01 6.356 5.93e-10 ***
## cylinders 5.192e-01 6.162e-01 0.843 0.40000
## displacement -4.673e-02 1.743e-02 -2.681 0.00765 **
## horsepower -1.542e-01 4.788e-02 -3.220 0.00139 **
## weight -1.100e-02 1.007e-03 -10.919 < 2e-16 ***
## acceleration -7.381e+00 1.167e+00 -6.324 7.12e-10 ***
## year -7.360e-01 2.436e-01 -3.021 0.00269 **
## cylinders:displacement -5.065e-04 2.765e-03 -0.183 0.85476
## horsepower:weight 3.236e-05 1.265e-05 2.559 0.01087 *
## acceleration:year 9.578e-02 1.520e-02 6.303 8.08e-10 ***
## displacement:weight 1.200e-05 6.709e-06 1.789 0.07446 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.811 on 381 degrees of freedom
## Multiple R-squared: 0.8736, Adjusted R-squared: 0.8703
## F-statistic: 263.4 on 10 and 381 DF, p-value: < 2.2e-16Here we see that the interaction between cylinders and displacement is not significant. The interactions horsepower * weight and acceleration * year are significant. The interaction between displacement*weight was significant at the 0.1 level.
(f) Try a few different transformations of the variables, such as log(X), X, X^2. Comment on your findings.
par(mfrow = c(2, 2))
plot(log(Auto$horsepower), Auto$mpg)
plot(sqrt(Auto$horsepower), Auto$mpg)
plot((Auto$horsepower)^2, Auto$mpg)
par(mfrow = c(2, 2))
plot(log(Auto$acceleration), Auto$mpg)
plot(sqrt(Auto$acceleration), Auto$mpg)
plot((Auto$acceleration)^2, Auto$mpg)
Both acceleration and horsepower were insignificant in the original linear model.The plots above show that the log transformation of the horsepower variable will provide a better fit. However, transforming acceleration did not seem help in the fit of the model.
10. This question should be answered using the Carseats data set.
(a) Fit a multiple regression model to predict Sales using Price, Urban, and US.
?Carseats
attach(Carseats)
lm.fit3 <- lm(Sales~Price+Urban+US)
summary(lm.fit3)##
## Call:
## lm(formula = Sales ~ Price + Urban + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
Based on the coefficient of the Price variable we can interpret that for every price increase of 1 we will see a sales decrease by 54.46 units. Having a store in an urban area did not effect sales. The coefficient of US tells us that having a store in the US increases sales by 1200.57 units more than having a store outside the US.
(c) Write out the model in equation form, being careful to handle the qualitative variables properly.
Sales = 13.043469-0.054459* Price-0.021916 * Urban_{Yes}+1.200573*US_{Yes}, where Urban_{Yes} = 1 for urban and 0 for not urban. US_{Yes} = 1 for US store and 0 for store outside the US.
(d) For which of the predictors can you reject the null hypothesis H0:βj=0?
Based on the information above you can reject the null hypothesis for Price and US.
(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
lm.fit4<-lm(Sales~Price+US)
summary(lm.fit4)##
## Call:
## lm(formula = Sales ~ Price + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16(f) How well do the models in (a) and (e) fit the data?
Even though both models have significant F-statistics, the R-squared for both models shows that only 23.93% of the variability can be explained by the models. The models do not appear to fit the data.
(g) Using the model from (e), obtain 95% confidence intervals for the coefficient(s)
confint(lm.fit4)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632(h) Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow = c(2, 2))
plot(lm.fit4)
plot( predict(lm.fit4),rstudent(lm.fit4))
Looking at the plot for the studentized residuals, as per the cutoff in the book, there doesn’t seem to be any observations below -3 or above 3.
plot(hatvalues (lm.fit))
Taking a closer look at the leverage values we can note that there is a high leverage point that exceeds 2(p/n) or 0.01.
12. This problem involves simple linear regression without an intercept.
(a) Recall that the coefficient estimate βˆ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?
The coefficients are the same if the sum of x^2 = the sum of y^2.
(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
set.seed(1)
x <- 1:100
sum(x^2)## [1] 338350y <- 2 * x + rnorm(100, sd = 0.1)
sum(y^2)## [1] 1353606fit.X<-lm(y~x + 0)
fit.Y<-lm(x~y + 0)summary(fit.Y)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.115418 -0.029231 -0.002186 0.031322 0.111795
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 5.00e-01 3.87e-05 12920 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.04502 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.669e+08 on 1 and 99 DF, p-value: < 2.2e-16summary(fit.X)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.223590 -0.062560 0.004426 0.058507 0.230926
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 2.0001514 0.0001548 12920 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.09005 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.669e+08 on 1 and 99 DF, p-value: < 2.2e-16(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
x<-1:100
sum(x^2)## [1] 338350y<-100:1
sum(y^2)## [1] 338350fit.Y <- lm(y ~ x + 0)
fit.X <- lm(x ~ y + 0)summary(fit.Y)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08summary(fit.X)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08