Exercise 2

Carefully explain the differences between the KNN classifier and KNN regression methods.
The KKN classifier is used when the outcome is qualitative, and the classifier predicts the outcome to be the class with most representations among the k nearest neighbors. On the other hand, KNN regression methods are used when the outcome is quantitative. The prediction is made based on the average value of the k nearest neighbors.

Exercise 9

This question involves the use of multiple linear regression on the Auto data set.

(a) Produce a scatterplot matrix which includes all of the variables in the data set.

pairs(Auto)

(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative.

cor(subset(Auto, select = -name))
##                     mpg  cylinders displacement horsepower     weight
## mpg           1.0000000 -0.7776175   -0.8051269 -0.7784268 -0.8322442
## cylinders    -0.7776175  1.0000000    0.9508233  0.8429834  0.8975273
## displacement -0.8051269  0.9508233    1.0000000  0.8972570  0.9329944
## horsepower   -0.7784268  0.8429834    0.8972570  1.0000000  0.8645377
## weight       -0.8322442  0.8975273    0.9329944  0.8645377  1.0000000
## acceleration  0.4233285 -0.5046834   -0.5438005 -0.6891955 -0.4168392
## year          0.5805410 -0.3456474   -0.3698552 -0.4163615 -0.3091199
## origin        0.5652088 -0.5689316   -0.6145351 -0.4551715 -0.5850054
##              acceleration       year     origin
## mpg             0.4233285  0.5805410  0.5652088
## cylinders      -0.5046834 -0.3456474 -0.5689316
## displacement   -0.5438005 -0.3698552 -0.6145351
## horsepower     -0.6891955 -0.4163615 -0.4551715
## weight         -0.4168392 -0.3091199 -0.5850054
## acceleration    1.0000000  0.2903161  0.2127458
## year            0.2903161  1.0000000  0.1815277
## origin          0.2127458  0.1815277  1.0000000

(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:
- Is there a relationship between the predictors and the response?
- Which predictors appear to have a statistically significant relationship to the response?
- What does the coefficient for the year variable suggest?

lm.fit9 <- lm(mpg ~ .-name, Auto)
summary(lm.fit9)
## 
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -9.5903 -2.1565 -0.1169  1.8690 13.0604 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -17.218435   4.644294  -3.707  0.00024 ***
## cylinders     -0.493376   0.323282  -1.526  0.12780    
## displacement   0.019896   0.007515   2.647  0.00844 ** 
## horsepower    -0.016951   0.013787  -1.230  0.21963    
## weight        -0.006474   0.000652  -9.929  < 2e-16 ***
## acceleration   0.080576   0.098845   0.815  0.41548    
## year           0.750773   0.050973  14.729  < 2e-16 ***
## origin         1.426141   0.278136   5.127 4.67e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared:  0.8215, Adjusted R-squared:  0.8182 
## F-statistic: 252.4 on 7 and 384 DF,  p-value: < 2.2e-16

Given the significance of the overall model, there appears to be a relationship between the predictors and the response. The variables that are statistically significant predictors of mpg are displacement, weight, year, and origin. The coefficient of the year variable suggests that for each unit increase in a car’s model year, its miles per gallon is estimated to increase by 0.75.

(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?

par(mfrow = c(2,2))
plot(lm.fit9)

The Residuals vs Fitted plot indicates that the data violates the assumption of homoscedasticity. Additionally, the plot indicates that the data is non-linear given the curved pattern in the residuals. From the Residuals vs Leverage plot, there appears to be no leverage points seeing that no observations cross the long-dashed grey line, which represents Cook’s distance.

(e) Use the star and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?

lm.fit9i1 = lm(mpg ~ horsepower*displacement, Auto)
summary(lm.fit9i1) 
## 
## Call:
## lm(formula = mpg ~ horsepower * displacement, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -10.9391  -2.3373  -0.5816   2.1698  17.5771 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)              5.305e+01  1.526e+00   34.77   <2e-16 ***
## horsepower              -2.343e-01  1.959e-02  -11.96   <2e-16 ***
## displacement            -9.805e-02  6.682e-03  -14.67   <2e-16 ***
## horsepower:displacement  5.828e-04  5.193e-05   11.22   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.944 on 388 degrees of freedom
## Multiple R-squared:  0.7466, Adjusted R-squared:  0.7446 
## F-statistic:   381 on 3 and 388 DF,  p-value: < 2.2e-16
lm.fit9i2 = lm(mpg ~ displacement*weight, Auto)
summary(lm.fit9i2) 
## 
## Call:
## lm(formula = mpg ~ displacement * weight, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -13.8664  -2.4801  -0.3355   1.8071  17.9429 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## (Intercept)          5.372e+01  1.940e+00  27.697  < 2e-16 ***
## displacement        -7.831e-02  1.131e-02  -6.922 1.85e-11 ***
## weight              -8.931e-03  8.474e-04 -10.539  < 2e-16 ***
## displacement:weight  1.744e-05  2.789e-06   6.253 1.06e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.097 on 388 degrees of freedom
## Multiple R-squared:  0.7265, Adjusted R-squared:  0.7244 
## F-statistic: 343.6 on 3 and 388 DF,  p-value: < 2.2e-16
lm.fit9i3 = lm(mpg ~ acceleration*cylinders, Auto)
summary(lm.fit9i3) 
## 
## Call:
## lm(formula = mpg ~ acceleration * cylinders, data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -13.2257  -3.1788  -0.7045   2.4031  17.4642 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)    
## (Intercept)            31.37192    5.27599   5.946 6.13e-09 ***
## acceleration            0.73498    0.33724   2.179   0.0299 *  
## cylinders              -1.84692    0.85564  -2.159   0.0315 *  
## acceleration:cylinders -0.11179    0.05806  -1.926   0.0549 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.895 on 388 degrees of freedom
## Multiple R-squared:  0.6097, Adjusted R-squared:  0.6067 
## F-statistic:   202 on 3 and 388 DF,  p-value: < 2.2e-16

When assessing the models above, we observe that the interaction terms between horsepower and displacement and between displacement and weight are statistically significant, while the interaction term between acceleration and cylinders is insignificant. However, it’s worth noting that the significance levels of these interactions may change when including additional predictors in the models.

(f) Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings.

reference9 <- lm(mpg ~ acceleration, Auto) 
summary(reference9)
## 
## Call:
## lm(formula = mpg ~ acceleration, data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -17.989  -5.616  -1.199   4.801  23.239 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    4.8332     2.0485   2.359   0.0188 *  
## acceleration   1.1976     0.1298   9.228   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.08 on 390 degrees of freedom
## Multiple R-squared:  0.1792, Adjusted R-squared:  0.1771 
## F-statistic: 85.15 on 1 and 390 DF,  p-value: < 2.2e-16
lm.fit9t1 <- lm(mpg ~ poly(acceleration, 5), Auto) 
summary(lm.fit9t1) 
## 
## Call:
## lm(formula = mpg ~ poly(acceleration, 5), data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -17.2209  -5.2976  -0.9565   4.7597  22.7506 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)    
## (Intercept)             23.4459     0.3516  66.689  < 2e-16 ***
## poly(acceleration, 5)1  65.3340     6.9608   9.386  < 2e-16 ***
## poly(acceleration, 5)2 -18.7482     6.9608  -2.693  0.00738 ** 
## poly(acceleration, 5)3   6.0643     6.9608   0.871  0.38418    
## poly(acceleration, 5)4  20.7577     6.9608   2.982  0.00304 ** 
## poly(acceleration, 5)5  -5.3550     6.9608  -0.769  0.44218    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 6.961 on 386 degrees of freedom
## Multiple R-squared:  0.2148, Adjusted R-squared:  0.2046 
## F-statistic: 21.12 on 5 and 386 DF,  p-value: < 2.2e-16
lm.fit9t2 <- lm(mpg ~ log(acceleration), Auto) 
summary(lm.fit9t2)
## 
## Call:
## lm(formula = mpg ~ log(acceleration), data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -18.0234  -5.6231  -0.9787   4.5943  23.0872 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        -27.834      5.373  -5.180 3.56e-07 ***
## log(acceleration)   18.801      1.966   9.565  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.033 on 390 degrees of freedom
## Multiple R-squared:   0.19,  Adjusted R-squared:  0.1879 
## F-statistic: 91.49 on 1 and 390 DF,  p-value: < 2.2e-16
lm.fit9t3 <- lm(mpg ~ sqrt(acceleration), Auto) 
summary(lm.fit9t3) 
## 
## Call:
## lm(formula = mpg ~ sqrt(acceleration), data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -18.034  -5.601  -1.027   4.713  23.184 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)         -14.177      4.008  -3.537 0.000453 ***
## sqrt(acceleration)    9.582      1.017   9.424  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.053 on 390 degrees of freedom
## Multiple R-squared:  0.1855, Adjusted R-squared:  0.1834 
## F-statistic: 88.81 on 1 and 390 DF,  p-value: < 2.2e-16

When comparing the reference model to the three models testing various transformation methods on acceleration, we notice that the transformations improve the multiple R-square in all three instances. From the polynomial transformation, we also see that the significance level of accelaration fluctuates as we increment the power to which it is raised. However, no polynomial terms beyond the 4th order have significant p-values.

Exercise 10

This question should be answered using the Carseats data set.

(a) Fit a multiple regression model to predict Sales using Price, Urban, and US.

lm.fit10a <- lm(Sales ~ Price + Urban + US, Carseats)
summary(lm.fit10a)
## 
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9206 -1.6220 -0.0564  1.5786  7.0581 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.043469   0.651012  20.036  < 2e-16 ***
## Price       -0.054459   0.005242 -10.389  < 2e-16 ***
## UrbanYes    -0.021916   0.271650  -0.081    0.936    
## USYes        1.200573   0.259042   4.635 4.86e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2335 
## F-statistic: 41.52 on 3 and 396 DF,  p-value: < 2.2e-16

(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
The output reveals that Price and US are significant predictors of Sales, unlike Urban. More specifically, for every $1000 increase in price, sales at each location decrease by 54 units. Additionally, locations in the US sell about 1,201 units more than locations outside the US.

(c) Write out the model in equation form, being careful to handle the qualitative variables properly.
The estimated regression equation is: \(Sales = 13.043469 - 0.054459 \cdot Price - 0.021916 \cdot Urban_{Yes} + 1.200573 \cdot US_{Yes}\)

(d) For which of the predictors can you reject the null hypothesis \(H_0: β_j=0\)?
As mentioned in (a), Price and US are significant in the model, why we can reject the null hypothesis for these two predictors.

(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.

lm.fit10e <- lm(Sales ~ Price + US, Carseats)
summary(lm.fit10e)
## 
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9269 -1.6286 -0.0574  1.5766  7.0515 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.03079    0.63098  20.652  < 2e-16 ***
## Price       -0.05448    0.00523 -10.416  < 2e-16 ***
## USYes        1.19964    0.25846   4.641 4.71e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2354 
## F-statistic: 62.43 on 2 and 397 DF,  p-value: < 2.2e-16

(f) How well do the models in (a) and (e) fit the data?
Both models have a multiple R-squared of 0.2393, meaning that the models can explain 23.93% of the variance in Sales. This is generally not a great fit.

(g) Using the model from (e), obtain 95% confidence intervals for the coefficient(s).

confint(lm.fit10e)
##                   2.5 %      97.5 %
## (Intercept) 11.79032020 14.27126531
## Price       -0.06475984 -0.04419543
## USYes        0.69151957  1.70776632

(h) Is there evidence of outliers or high leverage observations in the model from (e)?

par(mfrow = c(1,1))
plot(hatvalues(lm.fit10e))

From the plot above we see an outlier with an index around 50. Using the which.max() function below, we find that the exact index for the outlier is 43.

which.max(hatvalues(lm.fit10e))
## 43 
## 43

Exercise 12

This problem involves simple linear regression without an intercept.

(a) Recall that the coefficient estimate βˆ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?
The coefficient estimates for the regression of X onto Y and Y onto X respectively will be the same if the relationship between them is perfectly linear. In other words, if \(\sum x^{2} = \sum y^{2}\), the coefficients will be the same.

(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.

set.seed(123)
n <- 100
X <- rnorm(n)
Y <- 2*X + rnorm(n)

lm_XY <- lm(X ~ Y + 0)  
coef(lm_XY) 
##         Y 
## 0.3975655
lm_YX <- lm(Y ~ X + 0) 
coef(lm_YX) 
##        X 
## 1.936372
sum(X^2)
## [1] 83.30737
sum(Y^2)
## [1] 405.7547

We observe different coefficients as a result of the difference between \(\sum X^{2}\) and \(\sum Y^{2}\)

(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.

set.seed(123)
n <- 100
X <- rnorm(n)

X_sum_sq <- sum(X^2) 
Y_sum_sq <- X_sum_sq / n 
Y <- sqrt(rep(Y_sum_sq, n)) 

lm_XY <- lm(X ~ Y + 0)
coef(lm_XY) 
##          Y 
## 0.09905014
lm_YX <- lm(Y ~ X + 0)
coef(lm_YX) 
##          X 
## 0.09905014
sum(X^2)
## [1] 83.30737
sum(Y^2)
## [1] 83.30737

This time, the coefficients are the same (0.09905014). As explained in (a), this is because \(\sum x^{2}\) is equivalent to \(\sum y^{2}\), which in this case are both 83.30737.