Chapter 3

Question 2

Carefully explain the differences between the KNN classifier and KNN regression methods.

KNN classifier solves classification problems where the response variable is categorical/qualitative. While KNN regression solves a regression problem where the response variable is continuous/quantitative. Both are methods that identify the K closest training observations to a target test point and differ fundamentally in their objectives and mathematical aggregations. As KNN classifier estimates the conditional probability for each class as the target point is assigned to the class with the highest probability. KNN regression predicts the quantitative value by taking the average of the response values.

Question 9

This question involves the use of multiple linear regression on the Auto data set.

library(ISLR2)
  1. Produce a scatterplot matrix which includes all of the variables in the data set.
pairs(Auto[, 1:8])

  1. Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, cor() which is qualitative.
cor(subset(Auto, select = -name))
##                     mpg  cylinders displacement horsepower     weight
## mpg           1.0000000 -0.7776175   -0.8051269 -0.7784268 -0.8322442
## cylinders    -0.7776175  1.0000000    0.9508233  0.8429834  0.8975273
## displacement -0.8051269  0.9508233    1.0000000  0.8972570  0.9329944
## horsepower   -0.7784268  0.8429834    0.8972570  1.0000000  0.8645377
## weight       -0.8322442  0.8975273    0.9329944  0.8645377  1.0000000
## acceleration  0.4233285 -0.5046834   -0.5438005 -0.6891955 -0.4168392
## year          0.5805410 -0.3456474   -0.3698552 -0.4163615 -0.3091199
## origin        0.5652088 -0.5689316   -0.6145351 -0.4551715 -0.5850054
##              acceleration       year     origin
## mpg             0.4233285  0.5805410  0.5652088
## cylinders      -0.5046834 -0.3456474 -0.5689316
## displacement   -0.5438005 -0.3698552 -0.6145351
## horsepower     -0.6891955 -0.4163615 -0.4551715
## weight         -0.4168392 -0.3091199 -0.5850054
## acceleration    1.0000000  0.2903161  0.2127458
## year            0.2903161  1.0000000  0.1815277
## origin          0.2127458  0.1815277  1.0000000
  1. Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results.
lmfunction = lm(mpg ~ . - name, data = Auto)
summary(lmfunction)
## 
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -9.5903 -2.1565 -0.1169  1.8690 13.0604 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -17.218435   4.644294  -3.707  0.00024 ***
## cylinders     -0.493376   0.323282  -1.526  0.12780    
## displacement   0.019896   0.007515   2.647  0.00844 ** 
## horsepower    -0.016951   0.013787  -1.230  0.21963    
## weight        -0.006474   0.000652  -9.929  < 2e-16 ***
## acceleration   0.080576   0.098845   0.815  0.41548    
## year           0.750773   0.050973  14.729  < 2e-16 ***
## origin         1.426141   0.278136   5.127 4.67e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared:  0.8215, Adjusted R-squared:  0.8182 
## F-statistic: 252.4 on 7 and 384 DF,  p-value: < 2.2e-16

Comment on the output. For instance:

  1. Is there a relationship between the predictors and the response?

Yes, the p-value is < 2.2e-16 is highly significant indicating that at least one predictor has a non-zero linear relationship with mpg

  1. Which predictors appear to have a statistically significant relationship to the response?

The p < 0.05 are highly significant in displacement, weight, year and origin. horsepower is no longer significant when accounting for these other features.

  1. What does the coefficient for the year variable suggest?

The coefficient is positive implying that holding all other features increases efficiency on average every year

  1. Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow = c(2, 2))
plot(lmfunction)

In this model, Residuals vs Fitted a distinct non-linear curved pattern strongly suggests that the true relationship is non-linear, and a purely linear model isn’t fully capturing the trend. Normal Q-Q the points generally fall along the straight line but curl away at the upper right end, indicating some mild right-skewness or outliers in the upper range of residuals. Scale-Location model the spread of the residuals widens slightly for lower fitted values, meaning the variance isn’t perfectly constant. Residuals vs Leverage the plot identifies a few observations sitting unusually far to the right in the predictor space.

  1. Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
lm_interact <- lm(mpg ~ displacement * weight + horsepower * acceleration, data = Auto)
summary(lm_interact)
## 
## Call:
## lm(formula = mpg ~ displacement * weight + horsepower * acceleration, 
##     data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -12.3265  -2.2605  -0.3629   1.9423  16.3116 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)              4.726e+01  4.476e+00  10.557  < 2e-16 ***
## displacement            -7.323e-02  1.093e-02  -6.701 7.39e-11 ***
## weight                  -6.509e-03  1.133e-03  -5.746 1.86e-08 ***
## horsepower               1.899e-02  3.159e-02   0.601  0.54821    
## acceleration             5.131e-01  2.052e-01   2.500  0.01282 *  
## displacement:weight      1.757e-05  3.027e-06   5.805 1.35e-08 ***
## horsepower:acceleration -7.593e-03  2.284e-03  -3.325  0.00097 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.906 on 385 degrees of freedom
## Multiple R-squared:  0.7534, Adjusted R-squared:  0.7495 
## F-statistic:   196 on 6 and 385 DF,  p-value: < 2.2e-16

The interaction of displacement:weight is highly significant this confirms the impact of placement with the weight

  1. Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings.
lm_trans <- lm(mpg ~ log(horsepower) + I(weight^2) + sqrt(displacement), data = Auto)
summary(lm_trans)
## 
## Call:
## lm(formula = mpg ~ log(horsepower) + I(weight^2) + sqrt(displacement), 
##     data = Auto)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -10.9791  -2.8081  -0.3314   2.1889  15.4260 
## 
## Coefficients:
##                      Estimate Std. Error t value Pr(>|t|)    
## (Intercept)         7.416e+01  5.117e+00  14.492  < 2e-16 ***
## log(horsepower)    -8.533e+00  1.356e+00  -6.292 8.45e-10 ***
## I(weight^2)        -2.799e-07  1.055e-07  -2.653 0.008299 ** 
## sqrt(displacement) -6.599e-01  1.687e-01  -3.911 0.000108 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.158 on 388 degrees of freedom
## Multiple R-squared:  0.7184, Adjusted R-squared:  0.7162 
## F-statistic:   330 on 3 and 388 DF,  p-value: < 2.2e-16

The transformations successfully improved the linearity of the engine variables and reduced visible structure in the residuals. However, this transformed model yields an R^2 of 0.7184 , which is lower than the full baseline model’s R^2 of 0.8215. This occurs because the transformed model completely excludes the highly significant predictors year and origin. Thus, while the transformations better satisfy our linear assumptions, the model loses predictive power by omitting those critical structural variables

Question 10

This question should be answered using the Carseats data set.

  1. Fit a multiple regression model to predict Sales using Price, Urban, and US.
lm_car <- lm(Sales ~ Price + Urban + US, data = Carseats)
summary(lm_car)
## 
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9206 -1.6220 -0.0564  1.5786  7.0581 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.043469   0.651012  20.036  < 2e-16 ***
## Price       -0.054459   0.005242 -10.389  < 2e-16 ***
## UrbanYes    -0.021916   0.271650  -0.081    0.936    
## USYes        1.200573   0.259042   4.635 4.86e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2335 
## F-statistic: 41.52 on 3 and 396 DF,  p-value: < 2.2e-16
  1. Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!

Intercept (\(\hat{\beta}_0 \approx 13.04\))- The theoretical average of child car seat sales. When price is 0 it’s unphysical, so this primarily serves to anchor the regression line

Price (\(\hat{\beta}_1 \approx -0.0545\))- Holding all other variables constant, for every $1 increase in the price of car seats, sales decrease by approximately 54.5 units on average. This shows a clear negative relationship between price and sales.

UrbanYes (\(\hat{\beta}_2 \approx -0.0219\))- Holding all other variables constant, stores located in urban areas sell about 21.9 fewer car seats on average than stores in rural areas. However, this effect is exceptionally small.

USYes (\(\hat{\beta}_3 \approx 1.2006\))- Holding all other variables constant, stores located within the United States sell about 1.2006 more car seats on average than stores located outside the US.

  1. Write out the model in equation form, being careful to handle the qualitative variables properly.

\[\widehat{\text{Sales}} = 13.0435 - 0.0545(\text{Price}) - 0.0219(\text{UrbanYes}) + 1.2006(\text{USYes})\]

  1. For which of the predictors can you reject the null hypothesis H0 : βj =0?

We reject the null hypothesis for Price and US as the p-values < 0.05. We fail to reject the null hypothesis for Urban.

  1. On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
lm_small_car <- lm(Sales ~ Price + US, data = Carseats)
summary(lm_small_car)
## 
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9269 -1.6286 -0.0574  1.5766  7.0515 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.03079    0.63098  20.652  < 2e-16 ***
## Price       -0.05448    0.00523 -10.416  < 2e-16 ***
## USYes        1.19964    0.25846   4.641 4.71e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2354 
## F-statistic: 62.43 on 2 and 397 DF,  p-value: < 2.2e-16

The resulting model equation becomes:

\[\widehat{\text{Sales}} = 13.0308 - 0.0545(\text{Price}) + 1.1996(\text{USYes})\]

  1. How well do the models in (a) and (e) fit the data?

Both configurations perform nearly identically as model (a) [Full] \(R^2 \approx 0.2393\), \(\text{RSE} = 2.472\) with 396 degrees of freedom, and model (e) [Reduced] \(R^2 \approx 0.2393\), \(\text{RSE} = 2.469\) with 397 degrees of freedom. However, because the reduced model RSE slightly decreases provides a marginally better, more parsimonious fit by eliminating unnecessary parameter variance.

  1. Using the model from (e), obtain 95% confidence intervals for the coefficient(s).
confint(lm_small_car)
##                   2.5 %      97.5 %
## (Intercept) 11.79032020 14.27126531
## Price       -0.06475984 -0.04419543
## USYes        0.69151957  1.70776632
  1. Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow = c(2, 2))
plot(lm_small_car)

Looking at the Residuals vs Fitted or Scale-Location plots, almost all residuals fall comfortably between the normal range of [-3, 3]. There are a couple of points right on the boundary line, but there is no strong evidence of extreme outliers disrupting the model fit. Inspecting the Residuals vs Leverage plot reveals a few extreme data coordinates with leverage values exceeding 0.03 or 0.04. These observations have highly unusual values in the predictor space and represent true high-leverage points.

Question 12

This problem involves simple linear regression without an intercept.

  1. Recall that the coefficient estimate ˆβ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?

For the two coefficient estimates to be mathematically identical (\(\hat{\beta}_{Y \sim X} = \hat{\alpha}_{X \sim Y}\)), we set their analytical expressions equal to one another. Assuming that the numerator is non-zero, we can divide both sides by \(\sum_{i=1}^n x_i y_i\). Resulting in, \[\frac{1}{\sum_{i=1}^n x_i^2} = \frac{1}{\sum_{i=1}^n y_i^2} \implies \sum_{i=1}^n x_i^2 = \sum_{i=1}^n y_i^2\] The coefficient estimate for the regression of \(X\) onto \(Y\) will be exactly the same as the coefficient estimate for the regression of \(Y\) onto \(X\) if and only if the sum of squares of the \(X\) observations equals the sum of squares of the \(Y\) observations

  1. Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
x <- rnorm(100, mean = 0, sd = 1)

y <- 3 * x + rnorm(100, mean = 0, sd = 1)

fit_yx <- lm(y ~ x + 0)

fit_xy <- lm(x ~ y + 0)

coef(fit_yx) 
##        x 
## 3.027824
coef(fit_xy) 
##        y 
## 0.291617
  1. Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
x <- rnorm(100, mean = 0, sd = 2)

y <- rev(x)

sum(x^2) == sum(y^2)
## [1] TRUE
fit.yx_same <- lm(y ~ x + 0)

fit.xy_same <- lm(x ~ y + 0)

print(coef(fit.yx_same))
##         x 
## 0.0734215
print(coef(fit.xy_same))
##         y 
## 0.0734215