Chapter 3 Homework

2. Carefully explain the differences between the KNN classifier and KNN regression methods:

The KNN Classifier method is when you attempt to classify a data value of Y given X by examining the Y values for a pre-determined amount (K) of X neighboring values in the training data. From the neighboring Y values, Y is classified as whatever the greater majority of the neighboring Y values are.

The KNN Regression method is when you attempt to make a prediction for Y given X by examining the Y values of a pre-determined amount (K) of X neighboring values in the training data and taking their averages.

9. This question involves the use of multiple linear regression on the Auto data set. (a) Produce a scatterplot matrix which includes all of the variables in the data set.

auto=read.csv("~/Grad School/Prediction Statistics/Module 2/Auto.csv", 
              na.strings='?')

library(ISLR2)
## Warning: package 'ISLR2' was built under R version 4.5.3
data(Auto)

pairs(Auto)

(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the ‘name’ variable, cor() which is qualitative.

cor(Auto[,-9])
##                     mpg  cylinders displacement horsepower     weight
## mpg           1.0000000 -0.7776175   -0.8051269 -0.7784268 -0.8322442
## cylinders    -0.7776175  1.0000000    0.9508233  0.8429834  0.8975273
## displacement -0.8051269  0.9508233    1.0000000  0.8972570  0.9329944
## horsepower   -0.7784268  0.8429834    0.8972570  1.0000000  0.8645377
## weight       -0.8322442  0.8975273    0.9329944  0.8645377  1.0000000
## acceleration  0.4233285 -0.5046834   -0.5438005 -0.6891955 -0.4168392
## year          0.5805410 -0.3456474   -0.3698552 -0.4163615 -0.3091199
## origin        0.5652088 -0.5689316   -0.6145351 -0.4551715 -0.5850054
##              acceleration       year     origin
## mpg             0.4233285  0.5805410  0.5652088
## cylinders      -0.5046834 -0.3456474 -0.5689316
## displacement   -0.5438005 -0.3698552 -0.6145351
## horsepower     -0.6891955 -0.4163615 -0.4551715
## weight         -0.4168392 -0.3091199 -0.5850054
## acceleration    1.0000000  0.2903161  0.2127458
## year            0.2903161  1.0000000  0.1815277
## origin          0.2127458  0.1815277  1.0000000

(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. 

lm.fit = lm(mpg ~ .- name ,data = Auto)
summary(lm.fit)
## 
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -9.5903 -2.1565 -0.1169  1.8690 13.0604 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -17.218435   4.644294  -3.707  0.00024 ***
## cylinders     -0.493376   0.323282  -1.526  0.12780    
## displacement   0.019896   0.007515   2.647  0.00844 ** 
## horsepower    -0.016951   0.013787  -1.230  0.21963    
## weight        -0.006474   0.000652  -9.929  < 2e-16 ***
## acceleration   0.080576   0.098845   0.815  0.41548    
## year           0.750773   0.050973  14.729  < 2e-16 ***
## origin         1.426141   0.278136   5.127 4.67e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared:  0.8215, Adjusted R-squared:  0.8182 
## F-statistic: 252.4 on 7 and 384 DF,  p-value: < 2.2e-16

For instance: i. Is there a relationship between the predictors and the response? Yes, the overall F statistic for the relationship is 252.4 and the overall p-value is below 0.5, so at least one of the predictors is statistically significant.

ii. Which predictors appear to have a statistically significant relationship to the response? From the table above, ‘displacement’, ‘weight’, ‘year’, and ‘origin’ are significant predictors of ‘mpg’.’cylinders’,‘horsepower’, and ‘acceleration’ have no effect on ‘mpg’.

For every 1 inch of cubic displacement added, mpg is expected to increase by 0.0199.

For every pound(lb) added, mpg is expected to decrease by 0.0065.

For every 1 year increase, mpg is expected to increase by 0.7508.

For every increase in origin code that occurs, mpg is expected to increase by 1.4261.

iii. What does the coefficient for the year variable suggest? For every 1 year increase, mpg is expected to increase by 0.7508.

(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?

par(mfrow=c(2,2))
plot(lm.fit)

Going off th Residuals vs. Fitted plot, there appear to be some pretty large outliers of 327, 323, and 326. The residuals vs Leverage plot also indicates there is at least one obsevation with an unusually high leverage.

(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?

lm.fit2 = lm(mpg ~ year + horsepower + year:horsepower ,data = Auto)
summary(lm.fit2)
## 
## Call:
## lm(formula = mpg ~ year + horsepower + year:horsepower, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -12.3492  -2.4509  -0.4557   2.4056  14.4437 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)     -1.266e+02  1.212e+01 -10.449   <2e-16 ***
## year             2.192e+00  1.613e-01  13.585   <2e-16 ***
## horsepower       1.046e+00  1.154e-01   9.063   <2e-16 ***
## year:horsepower -1.596e-02  1.562e-03 -10.217   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.901 on 388 degrees of freedom
## Multiple R-squared:  0.7522, Adjusted R-squared:  0.7503 
## F-statistic: 392.5 on 3 and 388 DF,  p-value: < 2.2e-16
lm.fit3 = lm(mpg ~ weight * acceleration,data = Auto)
summary(lm.fit3)
## 
## Call:
## lm(formula = mpg ~ weight * acceleration, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -10.5823  -2.6411  -0.3517   2.2611  15.6704 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## (Intercept)          2.814e+01  4.872e+00   5.776 1.57e-08 ***
## weight              -3.168e-03  1.461e-03  -2.168  0.03076 *  
## acceleration         1.117e+00  3.097e-01   3.608  0.00035 ***
## weight:acceleration -2.787e-04  9.694e-05  -2.875  0.00426 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.249 on 388 degrees of freedom
## Multiple R-squared:  0.706,  Adjusted R-squared:  0.7037 
## F-statistic: 310.5 on 3 and 388 DF,  p-value: < 2.2e-16

I decided to fit a linear regression model with interaction effects for (‘mpg’, ‘year’, and ‘horsepower’) and model with interaction effects of (‘mpg’,‘wight’,and ‘acceleration’).

From the 1st table above, ‘year’, and ‘horsepower’ are significant predictors.

If horsepower increases by 1 unit,mpg increase by 1.05. If year increases by 1 unit,mpg increase by 2.19.

The interaction between horsepower and year is statistically significant because the interaction effects between the 2 values yields a low p-value.

From the 2nd table above, ‘weight’ and ‘acceleration’ are significant predictors.

If weight increases by 1 unit,mpg decreases by -0.003168. If acceleration increases by 1 unit,mpg increase by 1.12.

The interaction between weight and acceleration is statistically significant because the interaction effects between the 2 values yields a low p-value.

(f) Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings.

lm.fitlog = lm(log(mpg) ~ .- name ,data = Auto)
summary(lm.fitlog)
## 
## Call:
## lm(formula = log(mpg) ~ . - name, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.40955 -0.06533  0.00079  0.06785  0.33925 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   1.751e+00  1.662e-01  10.533  < 2e-16 ***
## cylinders    -2.795e-02  1.157e-02  -2.415  0.01619 *  
## displacement  6.362e-04  2.690e-04   2.365  0.01852 *  
## horsepower   -1.475e-03  4.935e-04  -2.989  0.00298 ** 
## weight       -2.551e-04  2.334e-05 -10.931  < 2e-16 ***
## acceleration -1.348e-03  3.538e-03  -0.381  0.70339    
## year          2.958e-02  1.824e-03  16.211  < 2e-16 ***
## origin        4.071e-02  9.955e-03   4.089 5.28e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1191 on 384 degrees of freedom
## Multiple R-squared:  0.8795, Adjusted R-squared:  0.8773 
## F-statistic: 400.4 on 7 and 384 DF,  p-value: < 2.2e-16
lm.fitsqrt = lm(sqrt(mpg) ~ .- name ,data = Auto)
summary(lm.fitsqrt)
## 
## Call:
## lm(formula = sqrt(mpg) ~ . - name, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.98891 -0.18946  0.00505  0.16947  1.02581 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   1.075e+00  4.290e-01   2.506   0.0126 *  
## cylinders    -5.942e-02  2.986e-02  -1.990   0.0474 *  
## displacement  1.752e-03  6.942e-04   2.524   0.0120 *  
## horsepower   -2.512e-03  1.274e-03  -1.972   0.0493 *  
## weight       -6.367e-04  6.024e-05 -10.570  < 2e-16 ***
## acceleration  2.738e-03  9.131e-03   0.300   0.7644    
## year          7.381e-02  4.709e-03  15.675  < 2e-16 ***
## origin        1.217e-01  2.569e-02   4.735 3.09e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.3074 on 384 degrees of freedom
## Multiple R-squared:  0.8561, Adjusted R-squared:  0.8535 
## F-statistic: 326.3 on 7 and 384 DF,  p-value: < 2.2e-16

When I take the log of mpg, the overall F statistic is 400.4 and the p-value is low. The model also explains 87.95% of the variability in the response variable using the predictor variables. All of the predictor variables, minus acceleration, appear to be statistically significant due to their low p-values.

When I take the square root of mpg, the overall F statistic is 326.3 and the p-value is still low. The model also explains 85.6% of the variability in the response variable using the predictor variables. All of the predictor variables, minus acceleration, appear to be statistically significant due to their low p-values. However, it appears that the ‘intercept’ and ‘horsepower’ predictor variables decrease in significance.

Key results from the original fit: Overall F: 252.4 Multiple R-squared: 82.15% Significant predictors: Intercept, displacement, weight, year, and origin.

Comparing these 2 new models to the results from the original fit, taking the log value appears to yield the higher F value, explains more of the variability, and has a greater amount of statistically significant predictor variables.

10. This question should be answered using

the Carseats data set.

library(ISLR2)
attach(Carseats)

(a) Fit a multiple regression model to predict ‘Sales’ using ‘Price’, ‘Urban’, and ‘US’.

fit <-lm(Sales~Price+Urban+US)
summary(fit)
## 
## Call:
## lm(formula = Sales ~ Price + Urban + US)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9206 -1.6220 -0.0564  1.5786  7.0581 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.043469   0.651012  20.036  < 2e-16 ***
## Price       -0.054459   0.005242 -10.389  < 2e-16 ***
## UrbanYes    -0.021916   0.271650  -0.081    0.936    
## USYes        1.200573   0.259042   4.635 4.86e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2335 
## F-statistic: 41.52 on 3 and 396 DF,  p-value: < 2.2e-16

(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
From the table above, ‘Price’ and ‘US’ are significant predictors of ‘Sales’. For every $1 increase in my price, sales decrease by $14. Sales inside the US are $1,200 higher than Sales outside of the US.Urban has no effect on Sales.

(c) Write out the model in equation form, being careful to handle the qualitative variables properly.
\(Sales = 13.043469 - 0.054459(Price) - 0.021916(UrbanYes) + 1.200573(USYes)\)

(d) For which of the predictors can you reject the null hypothesis \(H_0 : \beta_j =0\)?
‘Price’ and ‘US’

(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.

fit <-lm(Sales~Price+US)
summary(fit)
## 
## Call:
## lm(formula = Sales ~ Price + US)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9269 -1.6286 -0.0574  1.5766  7.0515 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.03079    0.63098  20.652  < 2e-16 ***
## Price       -0.05448    0.00523 -10.416  < 2e-16 ***
## USYes        1.19964    0.25846   4.641 4.71e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2354 
## F-statistic: 62.43 on 2 and 397 DF,  p-value: < 2.2e-16

(f) How well do the models in (a) and (e) fit the data?
Not well, each model explains around 23% of the variance in Sales.

(g) Using the model from (e), obtain 95% confidence intervals for the coefficient(s).

confint(fit)
##                   2.5 %      97.5 %
## (Intercept) 11.79032020 14.27126531
## Price       -0.06475984 -0.04419543
## USYes        0.69151957  1.70776632

(h) Is there evidence of outliers or high leverage observations in the model from (e)? 

par(mfrow=c(2,2))
plot(fit)

Yes, there is evidence of outliers in the model.

Based on the Residuals vs. Fitted plot, 377, 69 and 51 appear to be outliers to note. However, these outliers are not too extreme/significant.

Based on the Residuals vs Leverage plot, there does not appear to be any significant high leverage observations in the model. The highest value is slightly greater then 0.04, while most others remain lower then that.

12. This problem involves simple linear regression without an intercept. (a) Recall that the coefficient estimate ˆβ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X? When the total sum of squares is equal for both X and Y OR if the total sum of products \(\sum_{i=1}^{n} x_i y_i\) equals 0.

(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X. 

set.seed(500)

# Generate n = 100 observations
n <- 100
x <- rnorm(n)
y <- 2 * x + rnorm(n)

# Regression of Y onto X (no intercept)
coef(lm(y ~ x - 1))
##        x 
## 2.009522
# Regression of X onto Y (no intercept)
coef(lm(x ~ y - 1))
##         y 
## 0.4007431

(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X

set.seed(500)

n <- 100
x <- rnorm(n)
y <- x

# Regression of Y onto X (no intercept)
coef(lm(y ~ x - 1))
## x 
## 1
# Regression of X onto Y (no intercept)
coef(lm(x ~ y - 1))
## y 
## 1