Assignment 2, Chapter 3
Adam Ramsey
2023-02-14
Problem 2
Carefully explain the differences between the KNN classifier
and KNN regression methods.
The KNN classifier attempts to determine which classification a value
would have based on nearest neighbors, while the KNN regression attempts
to predict the output value itself based on nearest neighbors.
Problem 9
This question involves the use of multiple linear regression
on the Auto data set.
library(ISLR2)
attach(Auto)(a) Produce a scatterplot matrix which includes all of the variables in the data set.
pairs(Auto[,1:9])
(b) Compute the matrix of correlations between the variables
using the function cor() . You will need to exclude the
name variable, which is qualitative.
cor(Auto[ ,-9])## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000(c) Use the lm() function to perform a multiple
linear regression with mpg as the response and all other
variables except name as the predictors. Use the
summary() function to print the results. Comment on the
output. For instance:
miles<-lm(mpg~.-name,Auto)
summary(miles)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the
response?
Some of the predictors have an effect on the response.
ii. Which predictors appear to have a statistically
significant relationship to the response?
Cylinders, Weight, Year, and Origin appear to be statistically
significant.
iii. What does the coefficient for the year
variable suggest?
The coefficient for year is .7508 suggesting that for every year, there
is a .7508 increase in mileage, ignoring all other variables.
(d) Use the plot() function to produce
diagnostic plots of the linear regression fit. Comment on any problems
you see with the fit. Do the residual plots suggest any unusually large
outliers? Does the leverage plot identify any observations with
unusually high leverage?
par(mfrow=c(2,2))
plot(miles)
The Q-Q plot is only linear up to a point. There do not appear to be any large outliers. There is one observation that has abnormally high leverage.
(e) Use the * and : symbols to fit
linear regression models with interaction effects. Do any interactions
appear to be statistically significant?
inter<-lm(mpg~.-name+displacement*weight+year:cylinders, Auto)
summary(inter)##
## Call:
## lm(formula = mpg ~ . - name + displacement * weight + year:cylinders,
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.5164 -1.5950 -0.1015 1.3021 12.5609
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -6.632e+01 1.166e+01 -5.686 2.59e-08 ***
## cylinders 1.209e+01 2.161e+00 5.594 4.24e-08 ***
## displacement -6.739e-02 1.063e-02 -6.340 6.50e-10 ***
## horsepower -3.994e-02 1.199e-02 -3.332 0.000948 ***
## weight -1.011e-02 6.936e-04 -14.574 < 2e-16 ***
## acceleration 1.006e-01 8.498e-02 1.184 0.237154
## year 1.561e+00 1.455e-01 10.724 < 2e-16 ***
## origin 5.936e-01 2.525e-01 2.351 0.019249 *
## displacement:weight 2.078e-05 2.200e-06 9.448 < 2e-16 ***
## cylinders:year -1.543e-01 2.762e-02 -5.588 4.38e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.853 on 382 degrees of freedom
## Multiple R-squared: 0.8694, Adjusted R-squared: 0.8664
## F-statistic: 282.6 on 9 and 382 DF, p-value: < 2.2e-16The displacement by weight interaction appears to be significant.
(f) Try a few different transformations of the variables, such as \(log(X), \sqrt{X}, X^2\) . Comment on your findings.
inter<-lm(mpg~.-name+displacement*sqrt(weight)+log(acceleration), Auto)
summary(inter)##
## Call:
## lm(formula = mpg ~ . - name + displacement * sqrt(weight) + log(acceleration),
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.0734 -1.6727 -0.0208 1.6210 12.4185
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 8.894e+01 2.704e+01 3.289 0.001100 **
## cylinders 3.109e-02 3.167e-01 0.098 0.921844
## displacement -9.144e-02 5.059e-02 -1.807 0.071511 .
## horsepower -4.456e-02 1.269e-02 -3.513 0.000497 ***
## weight 6.149e-03 9.576e-03 0.642 0.521139
## acceleration 1.985e+00 4.431e-01 4.479 9.92e-06 ***
## year 7.961e-01 4.492e-02 17.724 < 2e-16 ***
## origin 5.199e-01 2.588e-01 2.009 0.045286 *
## sqrt(weight) -1.532e+00 8.979e-01 -1.706 0.088793 .
## log(acceleration) -3.248e+01 7.331e+00 -4.431 1.23e-05 ***
## displacement:sqrt(weight) 1.571e-03 8.493e-04 1.850 0.065062 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.899 on 381 degrees of freedom
## Multiple R-squared: 0.8655, Adjusted R-squared: 0.862
## F-statistic: 245.3 on 10 and 381 DF, p-value: < 2.2e-16detach(Auto)log of acceleration is also significant, but not as strong as acceleration itself.
Problem 10
This question should be answered using the
Carseats data set.
attach(Carseats)(a) Fit a multiple regression model to predict
Sales using Price , Urban, and
US .
fit<-lm(Sales~Price+Urban+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + Urban + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16(b) Provide an interpretation of each coefficient in the
model. Be careful—some of the variables in the model are
qualitative!
For each $1 increase of price, there is a $54 decrease in sales. Urban
has no effect on sales. Sales in US are $1200 higher than outside the
US.
(c) Write out the model in equation form, being careful to
handle the qualitative variables properly.
\(Sales =
13.043469-0.054459Price-0.021916Urban_{Yes}+1.200573US_{Yes}\)
(d) For which of the predictors can you reject the null
hypothesis \(H_0 : β_j =
0\)?
Price and US
(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
fit<-lm(Sales~Price+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16(f) How well do the models in (a) and (e) fit the
data?
Not good, the models only explain 23% of the variance in sales.
(g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
confint(fit)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632(h) Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow=c(2,2))
plot(fit)
detach(Carseats)There do not appear to be any heavily influential observations in the model.
Problem 12
This problem involves simple linear regression without an intercept.
(a) Recall that the coefficient estimate \(\hat{β}\) for the linear regression of
\(Y\) onto \(X\) without an intercept is given by
(3.38). Under what circumstance is the coefficient estimate for the
regression of \(X\) onto \(Y\) the same as the coefficient estimate
for the regression of \(Y\) onto \(X\)?
Coefficient estimates will be the same whenever they have the same
denominator.
(b) Generate an example in R with \(n = 100\) observations in which the
coefficient estimate for the regression of \(X\) onto \(Y\) is different from the
coefficient estimate for the regression of \(Y\) onto \(X\).
set.seed(1)
x1<-rnorm(100)
y1<-x1*x1
xy<-lm(x1~y1+0)
yx<-lm(y1~x1+0)
summary(xy)$coefficients[1,1]## [1] 0.1078456summary(yx)$coefficients[1,1]## [1] 0.2598674(c) Generate an example in R with \(n = 100\) observations in which the
coefficient estimate for the regression of \(X\) onto \(Y\) is the same as the coefficient
estimate for the regression of \(Y\)
onto \(X\).
set.seed(1)
x1<-rnorm(100)
y1<-sample(x1)
xy<-lm(x1~y1+0)
yx<-lm(y1~x1+0)
summary(xy)$coefficients[1,1]## [1] -0.07767695summary(yx)$coefficients[1,1]## [1] -0.07767695