Assignment 2
David Rodriguez
6/14/2020
Question 2
While KNN classifier and KNN regression are related, there are some differences. The classifier uses a probability to determine which class a point belongs to. The observation will be placed with the class in which the highest probability was calculated. The regression version uses an average, rather than a probability. The KNN regression calculates the average value of K points nearest to the point in question.
Question 9
This question involves the use of multiple linear regression on the Auto data set
library(ISLR)## Warning: package 'ISLR' was built under R version 3.6.3attach(Auto)a) Scatterplot matrix of all variables
pairs(Auto)
Removing the name variable and missing data rows and convert horsepower variable to an integer
Auto2 = Auto[,-9]
attach(Auto2)## The following objects are masked from Auto:
##
## acceleration, cylinders, displacement, horsepower, mpg,
## origin, weight, yearAuto2$horsepower = as.integer(Auto2$horsepower)b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative (removed in previous code chunk).
cor(Auto2, Auto2)## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results.
auto.fit = lm(mpg~ ., data = Auto2)
summary(auto.fit)##
## Call:
## lm(formula = mpg ~ ., data = Auto2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16c) i)Is there a relationship between the predictors and the response?
Yes, the p value is very small (2.2e-16)
ii) Which predictors appear to have a statistically significant relationship to the response
The predictors that appear to be statistically significant are displacement, weight, acceleration, year and origin.
iii) What does the coefficient for the year variable suggest?
With a coefficient of .7508, this suggests that mpg efficiency has improved by about .75 miles per gallon. So cars can travel .75 miles further on one gallon of gas then models of the previous year.
d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
The residuals vs. fitted appears to have a curve. There is also some outliers on the tail end of the QQ plot.On the leverage plot, there are some observations with unusually high leverage. R has some built in tools on these plots that show the observation numbers of potential outliers. We coule omit these points and re-run our model to see how much they affect model performance.
par(mfrow = c(2,2))
plot(auto.fit)
(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant? Significant interaction effects appear to be displacement and weight, displacement and year and acceleration and year.**
auto.fit2 = lm(mpg~ displacement*weight + displacement*year + acceleration:year, data = Auto2)
summary(auto.fit2)##
## Call:
## lm(formula = mpg ~ displacement * weight + displacement * year +
## acceleration:year, data = Auto2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.8117 -1.7142 0.0188 1.4206 13.2777
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -3.483e+01 7.150e+00 -4.872 1.62e-06 ***
## displacement 9.818e-02 3.672e-02 2.674 0.007822 **
## weight -1.055e-02 6.629e-04 -15.916 < 2e-16 ***
## year 1.124e+00 8.840e-02 12.715 < 2e-16 ***
## displacement:weight 1.998e-05 2.108e-06 9.476 < 2e-16 ***
## displacement:year -2.190e-03 4.568e-04 -4.794 2.34e-06 ***
## year:acceleration 3.398e-03 8.900e-04 3.818 0.000157 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.904 on 385 degrees of freedom
## Multiple R-squared: 0.8637, Adjusted R-squared: 0.8615
## F-statistic: 406.5 on 6 and 385 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(auto.fit2)
(f) Try a few different transformations of the variables, such as log(X),√X, X2. Comment on your findings. First transformation I tried was squaring weight. This increased R^2 to .8528 from .8182 on the original model. The next transformation I added was to take the log of displacement. This also increased the R^2 value slightly to .8535. I decided to also try transforming the response variable so I took the log of mpg. This also increased the R^2 to .8841. I wanted to run all these transformation model with all the predictors just to see if it would improve the model. Eliminating the insignificant predictors would hopefully improve the model even further. ONe final note on these transformations, all the residuals vs fitted plots appear to have diminishing curves from teh original model.
auto.fit3 = lm(mpg~ . + I(weight^2), data = Auto2)
summary(auto.fit3)##
## Call:
## lm(formula = mpg ~ . + I(weight^2), data = Auto2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.4706 -1.6701 -0.1488 1.6383 12.5429
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.479e+00 4.614e+00 0.321 0.74867
## cylinders -2.840e-01 2.917e-01 -0.974 0.33083
## displacement 1.371e-02 6.793e-03 2.019 0.04418 *
## horsepower -2.435e-02 1.243e-02 -1.959 0.05083 .
## weight -2.049e-02 1.580e-03 -12.970 < 2e-16 ***
## acceleration 6.571e-02 8.895e-02 0.739 0.46055
## year 7.999e-01 4.615e-02 17.331 < 2e-16 ***
## origin 7.418e-01 2.603e-01 2.850 0.00461 **
## I(weight^2) 2.237e-06 2.341e-07 9.556 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.994 on 383 degrees of freedom
## Multiple R-squared: 0.8558, Adjusted R-squared: 0.8528
## F-statistic: 284.2 on 8 and 383 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(auto.fit3)
auto.fit4 = lm(mpg~ . + I(weight^2) + log(displacement), data = Auto2)
summary(auto.fit4)##
## Call:
## lm(formula = mpg ~ . + I(weight^2) + log(displacement), data = Auto2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.5028 -1.6638 -0.0858 1.5626 12.1861
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.080e+01 9.864e+00 2.109 0.03563 *
## cylinders -1.468e-01 2.968e-01 -0.495 0.62102
## displacement 3.780e-02 1.281e-02 2.950 0.00337 **
## horsepower -3.282e-02 1.295e-02 -2.535 0.01164 *
## weight -1.596e-02 2.582e-03 -6.181 1.63e-09 ***
## acceleration 3.304e-02 8.973e-02 0.368 0.71288
## year 7.991e-01 4.592e-02 17.401 < 2e-16 ***
## origin 5.452e-01 2.738e-01 1.991 0.04717 *
## I(weight^2) 1.634e-06 3.584e-07 4.560 6.91e-06 ***
## log(displacement) -5.988e+00 2.706e+00 -2.213 0.02749 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.979 on 382 degrees of freedom
## Multiple R-squared: 0.8577, Adjusted R-squared: 0.8543
## F-statistic: 255.8 on 9 and 382 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(auto.fit4)
auto.fit5 = lm(log(mpg)~ .+ I(weight^2) + log(displacement), data = Auto2)
summary(auto.fit5)##
## Call:
## lm(formula = log(mpg) ~ . + I(weight^2) + log(displacement),
## data = Auto2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.44447 -0.06736 -0.00167 0.05913 0.37335
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.873e+00 3.780e-01 7.601 2.30e-13 ***
## cylinders -1.826e-02 1.137e-02 -1.606 0.109201
## displacement 1.375e-03 4.911e-04 2.801 0.005359 **
## horsepower -1.949e-03 4.961e-04 -3.928 0.000102 ***
## weight -4.039e-04 9.894e-05 -4.082 5.44e-05 ***
## acceleration -2.871e-03 3.439e-03 -0.835 0.404309
## year 3.065e-02 1.760e-03 17.414 < 2e-16 ***
## origin 1.822e-02 1.049e-02 1.737 0.083243 .
## I(weight^2) 2.813e-08 1.374e-08 2.048 0.041237 *
## log(displacement) -2.181e-01 1.037e-01 -2.103 0.036084 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.1142 on 382 degrees of freedom
## Multiple R-squared: 0.8899, Adjusted R-squared: 0.8873
## F-statistic: 342.9 on 9 and 382 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(auto.fit5)
#QUESTION 10
library(ISLR)
attach(Carseats)(a) Fit a multiple regression model to predict Sales using Price, Urban, and US.
carseat.fit = lm(Sales~ Price + Urban + US, data = Carseats)
summary(carseat.fit)##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16b) Provide an interpretation of each coefficient in the model. The coefficients above represent that for each $1 raise in price, sales will be reduced by about $54.46. For the US(YES) variable, sales in the US are about $1,200 higher than those outside the US. The variable Urban has a high p-value and does not statistically have an impact on sales.
c) Write out the model in equation form, being careful to handle the qualitative variables properly
Sales = 13.043469 -0.054459Price-0.021916Urban_{Yes}+1.200573XUS_{Yes}
d) For which of the predictors can you reject the null hypothesis? Based on the p-values in the summary above, we can reject the null hypothesis for the Price and US variables.
On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome
carseat.fit2 = lm(Sales~ Price + US, data = Carseats)
summary(carseat.fit2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16f) How well do the model in (a) and (e) fit the data? Both of these models have a ver low R^2 value. The original is at .2335 and after eliminating the ‘Urban’ variable due to insignificance, the R^2 only improved to .2354.
g) Using the model from (e), obtain 95% confidence intervals for the coefficient(s).
confint(carseat.fit2)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632h) Is there evidence of outliers or high leverage observations in the model from (e)? There doesn’t appear to be any outliers when looking at the diagnostics plots of the model. However, we can use the influence.measures() function in R to provide a list of high leverage observations. The list appears to have 23 observations.
par(mfrow=c(2,2))
plot(carseat.fit2)
summary(influence.measures(carseat.fit2))## Potentially influential observations of
## lm(formula = Sales ~ Price + US, data = Carseats) :
##
## dfb.1_ dfb.Pric dfb.USYs dffit cov.r cook.d hat
## 26 0.24 -0.18 -0.17 0.28_* 0.97_* 0.03 0.01
## 29 -0.10 0.10 -0.10 -0.18 0.97_* 0.01 0.01
## 43 -0.11 0.10 0.03 -0.11 1.05_* 0.00 0.04_*
## 50 -0.10 0.17 -0.17 0.26_* 0.98 0.02 0.01
## 51 -0.05 0.05 -0.11 -0.18 0.95_* 0.01 0.00
## 58 -0.05 -0.02 0.16 -0.20 0.97_* 0.01 0.01
## 69 -0.09 0.10 0.09 0.19 0.96_* 0.01 0.01
## 126 -0.07 0.06 0.03 -0.07 1.03_* 0.00 0.03_*
## 160 0.00 0.00 0.00 0.01 1.02_* 0.00 0.02
## 166 0.21 -0.23 -0.04 -0.24 1.02 0.02 0.03_*
## 172 0.06 -0.07 0.02 0.08 1.03_* 0.00 0.02
## 175 0.14 -0.19 0.09 -0.21 1.03_* 0.02 0.03_*
## 210 -0.14 0.15 -0.10 -0.22 0.97_* 0.02 0.01
## 270 -0.03 0.05 -0.03 0.06 1.03_* 0.00 0.02
## 298 -0.06 0.06 -0.09 -0.15 0.97_* 0.01 0.00
## 314 -0.05 0.04 0.02 -0.05 1.03_* 0.00 0.02_*
## 353 -0.02 0.03 0.09 0.15 0.97_* 0.01 0.00
## 357 0.02 -0.02 0.02 -0.03 1.03_* 0.00 0.02
## 368 0.26 -0.23 -0.11 0.27_* 1.01 0.02 0.02_*
## 377 0.14 -0.15 0.12 0.24 0.95_* 0.02 0.01
## 384 0.00 0.00 0.00 0.00 1.02_* 0.00 0.02
## 387 -0.03 0.04 -0.03 0.05 1.02_* 0.00 0.02
## 396 -0.05 0.05 0.08 0.14 0.98_* 0.01 0.00#Question 12 This problem involves simple linear regression without an intercept
a) Recall that the coefficient estimate ˆ β for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X? The coefficient estimate will be the same for both models when y=B_0 + B_1X == x = B_0 + B_1Y
b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
diffy = rnorm(100,50,5)
diffx = rnorm(100,10,1)
diff_fit.y = lm(diffy~diffx)
diff_fit.x = lm(diffx~diffy)
summary(diff_fit.y)##
## Call:
## lm(formula = diffy ~ diffx)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.0881 -3.6996 -0.0145 3.1855 10.5794
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 53.6078 5.1268 10.46 <2e-16 ***
## diffx -0.2633 0.5069 -0.52 0.605
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.933 on 98 degrees of freedom
## Multiple R-squared: 0.002747, Adjusted R-squared: -0.007429
## F-statistic: 0.2699 on 1 and 98 DF, p-value: 0.6045summary(diff_fit.x)##
## Call:
## lm(formula = diffx ~ diffy)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.2501 -0.4954 0.0559 0.5688 2.0215
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 10.59944 1.02773 10.31 <2e-16 ***
## diffy -0.01043 0.02008 -0.52 0.605
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9818 on 98 degrees of freedom
## Multiple R-squared: 0.002747, Adjusted R-squared: -0.007429
## F-statistic: 0.2699 on 1 and 98 DF, p-value: 0.6045c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
same.y = 1:100
same.x = 100:1
samefit.y = lm(same.y~same.x)
samefit.x = lm(same.x~same.y)
summary(samefit.y)## Warning in summary.lm(samefit.y): essentially perfect fit: summary may be
## unreliable##
## Call:
## lm(formula = same.y ~ same.x)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.680e-13 -4.300e-16 2.850e-15 5.302e-15 3.575e-14
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.010e+02 5.598e-15 1.804e+16 <2e-16 ***
## same.x -1.000e+00 9.624e-17 -1.039e+16 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.778e-14 on 98 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.08e+32 on 1 and 98 DF, p-value: < 2.2e-16summary(samefit.x)## Warning in summary.lm(samefit.x): essentially perfect fit: summary may be
## unreliable##
## Call:
## lm(formula = same.x ~ same.y)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.575e-14 -5.302e-15 -2.850e-15 4.300e-16 2.680e-13
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.010e+02 5.598e-15 1.804e+16 <2e-16 ***
## same.y -1.000e+00 9.624e-17 -1.039e+16 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.778e-14 on 98 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.08e+32 on 1 and 98 DF, p-value: < 2.2e-16