Assignment #2
Jared Chappelle
2022-09-10
Problem 2.
Carefully explain the differences between the KNN classifier
and KNN regression methods
The KNN regression method is used to solve regression problems, whereas
the KNN classifier method is used to solves classification problems. The
KNN classification method is where you identify the neighborhood of x
and then estimate the probability of Y being in a particular class
within that neighborhood. For KNN regression, you also identify the
neighborhood of x, and you estimate Y as the average of all the
different training responses in the neighborhood.
Problem 9.
This question involves the use of multiple linear regression on the Auto data set.
(a) Produce a scatterplot matrix which includes all of the variables in the data set.
library(ISLR)
pairs(Auto)
(b) Compute the matrix of correlations between the variables
using the function cor(). You will need to exclude the name
variable, cor() which is qualitative.
cor(subset(Auto,select=-name))## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:
lm.fit<-lm(mpg~.-name,data=Auto)
summary(lm.fit)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the
response?
Yes there is a relationship between the predictors and the response; the
p-value is extremely small at 2.2e-16 meaning that the results are
significant.
ii. Which predictors appear to have a statistically
significant relationship to the response?
It seems that the predictors displacement, weight, year, and origin are
all statistically significant.
iii. What does the coefficient for the year variable
suggest?
The coefficient for year suggests that with all other variables held
equal, with each year the mpg increases by that amount. The coefficient
is about .75, so each year cars because more fuel efficient by about .75
mpg.
(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow=c(2,2))
plot(lm.fit)
According to the Residuals vs Fitted plot there seems to be some non-linearity in the data. Perhaps our model is not the best fit for this data. Also there are some outliers that can be seen in the Residuals vs Leverage plot with several above 2 and a couple below -2. There is also one really large leverage point which is point 14.
(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
lm.fit1<-lm(mpg~displacement*horsepower+weight*year,data=Auto)
summary(lm.fit1)##
## Call:
## lm(formula = mpg ~ displacement * horsepower + weight * year,
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -7.6471 -1.7624 -0.0861 1.3492 12.2700
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -6.454e+01 1.355e+01 -4.763 2.70e-06 ***
## displacement -5.119e-02 7.614e-03 -6.723 6.45e-11 ***
## horsepower -1.461e-01 1.650e-02 -8.851 < 2e-16 ***
## weight 1.751e-02 4.656e-03 3.761 0.000195 ***
## year 1.533e+00 1.747e-01 8.776 < 2e-16 ***
## displacement:horsepower 3.979e-04 4.624e-05 8.604 < 2e-16 ***
## weight:year -2.883e-04 6.194e-05 -4.654 4.48e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.901 on 385 degrees of freedom
## Multiple R-squared: 0.8639, Adjusted R-squared: 0.8618
## F-statistic: 407.5 on 6 and 385 DF, p-value: < 2.2e-16It seems that both the interactions between displacement
and horsepower, and weight and
year are both statistically significant.
lm.fit2<-lm(mpg~origin*displacement+origin*horsepower,data=Auto)
summary(lm.fit2)##
## Call:
## lm(formula = mpg ~ origin * displacement + origin * horsepower,
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.8956 -2.5367 -0.5417 1.9103 18.4531
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 25.11629 1.61757 15.527 < 2e-16 ***
## origin 8.21071 1.12027 7.329 1.37e-12 ***
## displacement -0.09920 0.01560 -6.361 5.69e-10 ***
## horsepower 0.14034 0.02862 4.904 1.39e-06 ***
## origin:displacement 0.05009 0.01310 3.823 0.000154 ***
## origin:horsepower -0.15106 0.01847 -8.177 4.25e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.059 on 386 degrees of freedom
## Multiple R-squared: 0.7329, Adjusted R-squared: 0.7295
## F-statistic: 211.9 on 5 and 386 DF, p-value: < 2.2e-16It seems that both the interactions between origin and
displacement, and origin and
horsepower are also statistically significant.
(f) Try a few different transformations of the variables, such as log(X), √ X, X2. Comment on your findings.
lm.fit3<-lm(mpg~.-name + log(weight),data=Auto)
summary(lm.fit3)##
## Call:
## lm(formula = mpg ~ . - name + log(weight), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.6516 -1.6398 -0.1671 1.5973 12.7247
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 269.474171 31.136919 8.654 < 2e-16 ***
## cylinders -0.498204 0.292415 -1.704 0.08924 .
## displacement 0.013527 0.006832 1.980 0.04843 *
## horsepower -0.022137 0.012483 -1.773 0.07696 .
## weight 0.007657 0.001631 4.694 3.73e-06 ***
## acceleration 0.045763 0.089486 0.511 0.60936
## year 0.797808 0.046383 17.200 < 2e-16 ***
## origin 0.719552 0.262819 2.738 0.00647 **
## log(weight) -41.320927 4.446725 -9.292 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.01 on 383 degrees of freedom
## Multiple R-squared: 0.8543, Adjusted R-squared: 0.8513
## F-statistic: 280.8 on 8 and 383 DF, p-value: < 2.2e-16The log transformation of weight is significant, and
perhaps even a better predictor than weight by itself.
lm.fit4<-lm(mpg~.-name + I(weight^2),data=Auto)
summary(lm.fit4)##
## Call:
## lm(formula = mpg ~ . - name + I(weight^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.4706 -1.6701 -0.1488 1.6383 12.5429
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.479e+00 4.614e+00 0.321 0.74867
## cylinders -2.840e-01 2.917e-01 -0.974 0.33083
## displacement 1.371e-02 6.793e-03 2.019 0.04418 *
## horsepower -2.435e-02 1.243e-02 -1.959 0.05083 .
## weight -2.049e-02 1.580e-03 -12.970 < 2e-16 ***
## acceleration 6.571e-02 8.895e-02 0.739 0.46055
## year 7.999e-01 4.615e-02 17.331 < 2e-16 ***
## origin 7.418e-01 2.603e-01 2.850 0.00461 **
## I(weight^2) 2.237e-06 2.341e-07 9.556 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.994 on 383 degrees of freedom
## Multiple R-squared: 0.8558, Adjusted R-squared: 0.8528
## F-statistic: 284.2 on 8 and 383 DF, p-value: < 2.2e-16The squared transformation of weight is also significant
but does not seem to be a better predictor than weight
itself. The t value in this model for weight is different
than the t value for \(lm.fit3\).
Problem 10.
This question should be answered using the Carseats data set.
library(ISLR)
attach(Carseats)(a) Fit a multiple regression model to predict Sales using
Price , Urban , and US
.
fit<-lm(Sales~Price+Urban+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + Urban + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16(b) Provide an interpretation of each coefficient in the
model. Be careful—some of the variables in the model are
qualitative!
From the table above, Price and US are
statistically signifcant predictors of Sales, for every $1
increase in price, sales decrease by $54. Sales inside the US are $1,200
higher than sales outside of the US.The Urban variable has no effect on
sales.
(c) Write out the model in equation form, being careful to
handle the qualitative variables properly.
\(Sales=13.043469 - 0.054459Price -
0.021916UrbanYes + 1.200573USYes\)
(d) For which of the predictors can you reject the null
hypothesis \(H_0 : β_j =
0\)?
Price and US
(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
fit<-lm(Sales~Price+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16(f) How well do the models in (a) and (e) fit the
data?
I would say that neither of the models fit the data particularly well,
the R squared for both are quite low. Only about 23.93% of the
variability is explained by both models, and the second model didn’t
change much after taking out Urban.
(g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
confint(fit)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632(h) Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow=c(2,2))
plot(fit)
We can see in the Residuals vs Leverage plot there are several outliers that are over 2 and under -2. There are also a few high leverage points, with one being past the .04 and a few others past .02.
Problem 12
This problem involves simple linear regression without an intercept.
(a) Recall that the coefficient estimate βˆ for the linear
regression of Y onto X without an intercept is given by (3.38). Under
what circumstance is the coefficient estimate for the regression of X
onto Y the same as the coefficient estimate for the regression of Y onto
X?
The coefficients will be the same as long as the summation of \(x_j^2=y_j^2\)
(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
set.seed(10)
x<-1:100
sum(x^2)## [1] 338350y<-x+rnorm(100,sd=.5)
sum(y^2)## [1] 338175.7fit.Y<-lm(y~x+0)
fit.X<-lm(x~y+0)
summary(fit.Y)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.08625 -0.39810 -0.08762 0.32124 1.13583
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.9997094 0.0008171 1223 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4753 on 99 degrees of freedom
## Multiple R-squared: 0.9999, Adjusted R-squared: 0.9999
## F-statistic: 1.497e+06 on 1 and 99 DF, p-value: < 2.2e-16summary(fit.X)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.13027 -0.31572 0.09218 0.40202 1.08795
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 1.0002245 0.0008175 1223 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4754 on 99 degrees of freedom
## Multiple R-squared: 0.9999, Adjusted R-squared: 0.9999
## F-statistic: 1.497e+06 on 1 and 99 DF, p-value: < 2.2e-16(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
x<-1:100
sum(x^2)## [1] 338350y<-100:1
sum(y^2)## [1] 338350fit.Y<-lm(y~x+0)
fit.X<-lm(x~y+0)
summary(fit.Y)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08summary(fit.X)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08