Assignment #2, Chapter 3
April Jacob
2022-08-30
Questions 2, 9, 10, 12
2. Carefully explain the differences between the KNN classifier and KNN regression methods.
#KNN Classifier estimates qualitative results based on the majority (or most common group found) and will be the classifier of the the data point in question. (e.g., if the value of K is equal to one, then you use the nearest-neighbor to determine the class of the data point. If the value of K is equal to nine, then you use the nine nearest neighbors, etc.)
#KNN Regression estimates the association between the independent variables and the continuous outcome based averaging the result of the KNN.9. This questions involes the use of multiple linear regression on the Auto data set.
library(ISLR2)- Produce a scatterplot matrix which includes all of the variables in the data set.
plot(Auto)
- Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, cor() which is qualitative.
auto <- Auto
auto$name=NULL #excludes name variable
cor(auto)## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000- Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:
- Is there a relationship between the predictors and the response?
- Which predictors appear to have a statistically significant relationship to the response?
- What does the coefficient for the year variable suggest?
auto.mlr <- lm(mpg~ .-name, data=Auto)
summary(auto.mlr)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16#i) Yes, there seem to be relationships between predictors and the response.
#ii) The predictors that appear to have a statistically significant relationship are displacement, weight, year, and origin.
#iii) The coefficient for the year variable is 0.750773 which suggests that, on average, the newer car of every year increases its MPG by 0.75.- Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow=c(2,2))
plot(auto.mlr)
par(mfrow=c(1,1))
#i) The residuals v. fitted plot shows some outliers in top right corner; the slight u-shape pattern indicate non-linearity; the slight funnel shape and spread of the residuals starts small but increases which shows an example of heteroscedasticity.
#ii) The residuals v. leverage plot shows no observations within Cook's d (dashed red lines), which means there are no influential points.- Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
interact.fit <- lm(mpg~.-name + horsepower*displacement, data=Auto)
origin.hp <- lm(mpg~.-name + horsepower*origin, data=Auto)
summary(origin.hp)##
## Call:
## lm(formula = mpg ~ . - name + horsepower * origin, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.277 -1.875 -0.225 1.570 12.080
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -2.196e+01 4.396e+00 -4.996 8.94e-07 ***
## cylinders -5.275e-01 3.028e-01 -1.742 0.0823 .
## displacement -1.486e-03 7.607e-03 -0.195 0.8452
## horsepower 8.173e-02 1.856e-02 4.404 1.38e-05 ***
## weight -4.710e-03 6.555e-04 -7.186 3.52e-12 ***
## acceleration -1.124e-01 9.617e-02 -1.168 0.2434
## year 7.327e-01 4.780e-02 15.328 < 2e-16 ***
## origin 7.695e+00 8.858e-01 8.687 < 2e-16 ***
## horsepower:origin -7.955e-02 1.074e-02 -7.405 8.44e-13 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.116 on 383 degrees of freedom
## Multiple R-squared: 0.8438, Adjusted R-squared: 0.8406
## F-statistic: 258.7 on 8 and 383 DF, p-value: < 2.2e-16#displacement and horsepower; horsepower and origin- Try a few different transformations of the variables, such as log(x), √X, X2. Comment on your findings.
summary(lm(mpg ~ . -name + log(acceleration), data=Auto)) #very significant but less so than acceleration##
## Call:
## lm(formula = mpg ~ . - name + log(acceleration), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.7931 -2.0052 -0.1279 1.9299 13.1085
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.552e+01 1.479e+01 3.077 0.00224 **
## cylinders -2.796e-01 3.193e-01 -0.876 0.38172
## displacement 8.042e-03 7.805e-03 1.030 0.30344
## horsepower -3.434e-02 1.401e-02 -2.450 0.01473 *
## weight -5.343e-03 6.854e-04 -7.795 6.15e-14 ***
## acceleration 2.167e+00 4.782e-01 4.532 7.82e-06 ***
## year 7.560e-01 4.978e-02 15.186 < 2e-16 ***
## origin 1.329e+00 2.724e-01 4.877 1.58e-06 ***
## log(acceleration) -3.513e+01 7.886e+00 -4.455 1.10e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.249 on 383 degrees of freedom
## Multiple R-squared: 0.8303, Adjusted R-squared: 0.8267
## F-statistic: 234.2 on 8 and 383 DF, p-value: < 2.2e-16summary(lm(mpg ~ . -name + I(horsepower^2), data=Auto)) #doesn't change the significance##
## Call:
## lm(formula = mpg ~ . - name + I(horsepower^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.5497 -1.7311 -0.2236 1.5877 11.9955
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.3236564 4.6247696 0.286 0.774872
## cylinders 0.3489063 0.3048310 1.145 0.253094
## displacement -0.0075649 0.0073733 -1.026 0.305550
## horsepower -0.3194633 0.0343447 -9.302 < 2e-16 ***
## weight -0.0032712 0.0006787 -4.820 2.07e-06 ***
## acceleration -0.3305981 0.0991849 -3.333 0.000942 ***
## year 0.7353414 0.0459918 15.989 < 2e-16 ***
## origin 1.0144130 0.2545545 3.985 8.08e-05 ***
## I(horsepower^2) 0.0010060 0.0001065 9.449 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.001 on 383 degrees of freedom
## Multiple R-squared: 0.8552, Adjusted R-squared: 0.8522
## F-statistic: 282.8 on 8 and 383 DF, p-value: < 2.2e-1610. This question should be answered using the Carseats data set.
- Fit a multiple regression model to predict Sales using Price, Urban, and US.
head(Carseats)## Sales CompPrice Income Advertising Population Price ShelveLoc Age Education
## 1 9.50 138 73 11 276 120 Bad 42 17
## 2 11.22 111 48 16 260 83 Good 65 10
## 3 10.06 113 35 10 269 80 Medium 59 12
## 4 7.40 117 100 4 466 97 Medium 55 14
## 5 4.15 141 64 3 340 128 Bad 38 13
## 6 10.81 124 113 13 501 72 Bad 78 16
## Urban US
## 1 Yes Yes
## 2 Yes Yes
## 3 Yes Yes
## 4 Yes Yes
## 5 Yes No
## 6 No Yesstr(Carseats)## 'data.frame': 400 obs. of 11 variables:
## $ Sales : num 9.5 11.22 10.06 7.4 4.15 ...
## $ CompPrice : num 138 111 113 117 141 124 115 136 132 132 ...
## $ Income : num 73 48 35 100 64 113 105 81 110 113 ...
## $ Advertising: num 11 16 10 4 3 13 0 15 0 0 ...
## $ Population : num 276 260 269 466 340 501 45 425 108 131 ...
## $ Price : num 120 83 80 97 128 72 108 120 124 124 ...
## $ ShelveLoc : Factor w/ 3 levels "Bad","Good","Medium": 1 2 3 3 1 1 3 2 3 3 ...
## $ Age : num 42 65 59 55 38 78 71 67 76 76 ...
## $ Education : num 17 10 12 14 13 16 15 10 10 17 ...
## $ Urban : Factor w/ 2 levels "No","Yes": 2 2 2 2 2 1 2 2 1 1 ...
## $ US : Factor w/ 2 levels "No","Yes": 2 2 2 2 1 2 1 2 1 2 ...summary(Carseats)## Sales CompPrice Income Advertising
## Min. : 0.000 Min. : 77 Min. : 21.00 Min. : 0.000
## 1st Qu.: 5.390 1st Qu.:115 1st Qu.: 42.75 1st Qu.: 0.000
## Median : 7.490 Median :125 Median : 69.00 Median : 5.000
## Mean : 7.496 Mean :125 Mean : 68.66 Mean : 6.635
## 3rd Qu.: 9.320 3rd Qu.:135 3rd Qu.: 91.00 3rd Qu.:12.000
## Max. :16.270 Max. :175 Max. :120.00 Max. :29.000
## Population Price ShelveLoc Age Education
## Min. : 10.0 Min. : 24.0 Bad : 96 Min. :25.00 Min. :10.0
## 1st Qu.:139.0 1st Qu.:100.0 Good : 85 1st Qu.:39.75 1st Qu.:12.0
## Median :272.0 Median :117.0 Medium:219 Median :54.50 Median :14.0
## Mean :264.8 Mean :115.8 Mean :53.32 Mean :13.9
## 3rd Qu.:398.5 3rd Qu.:131.0 3rd Qu.:66.00 3rd Qu.:16.0
## Max. :509.0 Max. :191.0 Max. :80.00 Max. :18.0
## Urban US
## No :118 No :142
## Yes:282 Yes:258
##
##
##
## mrm <- lm(Sales ~ Price + Urban + US, data = Carseats)
summary(mrm)##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16- Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
#PRICE: As price increase, sales decrease per the low p-value.
#URBANYES: Sales are not affected if the store is in an urban area per the high p-value.
#USYES: US stores sell more car seats than a store outside of the US.- Write out the model in equation form, being careful to handle the qualitative variables properly.
#Salesi=β0+β1⋅Pricei+β2⋅di+β3⋅ei+ϵi
#Sales = 13.04 + -0.05 Price + -0.02 UrbanYes + 1.20 USYes
#Where di=1 if store i is in an urban location and 0 otherwise and ei=1 if store i in the US 0 otherwise.- For which of the predictors can you reject the null hypothesis H 0 : β j = 0?
#Price & US.- On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
mrm2 <- lm(Sales ~ Price + US, data= Carseats)
summary(mrm2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16- How well do the models in (a) and (e) fit the data?
#Based on their respective R-square values(in summary tables), these two models are mediocre (only 24% change in response explained).- Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
confint(mrm2)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632- Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow=c(2,2))
plot(mrm2)
par(mfrow=c(1,1))12. This problem involves simple linear regression without an intercept.
- Recall that the coefficient estimate for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?
#The coeffecient remains the same.- Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
set.seed(1)
x=rnorm(100)
y=x+rnorm(100)
head(data.frame(y,x))## y x
## 1 -1.2468205 -0.6264538
## 2 0.2257592 0.1836433
## 3 -1.7465503 -0.8356286
## 4 1.7533096 1.5952808
## 5 -0.3250769 0.3295078
## 6 0.9468189 -0.8204684regy <- lm(y~x+0)
summary(regy)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.9154 -0.6472 -0.1771 0.5056 2.3109
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.9939 0.1065 9.334 3.1e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9586 on 99 degrees of freedom
## Multiple R-squared: 0.4681, Adjusted R-squared: 0.4627
## F-statistic: 87.13 on 1 and 99 DF, p-value: 3.1e-15regx <- lm(x~y+0)
summary(regx)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.35410 -0.37468 0.09974 0.48799 1.55406
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.47099 0.05046 9.334 3.1e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.6599 on 99 degrees of freedom
## Multiple R-squared: 0.4681, Adjusted R-squared: 0.4627
## F-statistic: 87.13 on 1 and 99 DF, p-value: 3.1e-15- Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
set.seed(2)
x=rnorm(100)
y=abs(x)
head(data.frame(y,x))## y x
## 1 0.89691455 -0.89691455
## 2 0.18484918 0.18484918
## 3 1.58784533 1.58784533
## 4 1.13037567 -1.13037567
## 5 0.08025176 -0.08025176
## 6 0.13242028 0.13242028regy2 <- lm(y~x+0)
summary(regy2)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## 0.00488 0.39563 0.86041 1.45172 2.47908
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.01116 0.10050 0.111 0.912
##
## Residual standard error: 1.161 on 99 degrees of freedom
## Multiple R-squared: 0.0001246, Adjusted R-squared: -0.009975
## F-statistic: 0.01234 on 1 and 99 DF, p-value: 0.9118regx2 <- lm(x~y+0)
summary(regx2)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.4791 -0.8654 -0.1418 0.8047 2.0675
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.01116 0.10050 0.111 0.912
##
## Residual standard error: 1.161 on 99 degrees of freedom
## Multiple R-squared: 0.0001246, Adjusted R-squared: -0.009975
## F-statistic: 0.01234 on 1 and 99 DF, p-value: 0.9118