Conceptual Exercises

Q2. Carefully explain the differences between the KNN classifier and KNN regression methods?**

KNN classifier is used to predict qualitative responses by attempting to to estimate the conditional distribution between Y and X, and then classifying observations to the class with the highest estimated probability to predict f(x). KNN regression is used to predict quantitative responses by approximating the conditional distribution between Y and X to predict f(x) by averaging observations closes to x(0). In both instances, the number of observation is determined by a positive integer K. The fit of either model is determined by K. In other words, the fit of the model is determined by the bias-variance tradeoff of K.

Applied Exercises

Q9. This question involves the use of multiple linear regression on the Auto data set. **

(A) Produce a scatter plot matrix which includes all of the variables in the data set.

pairs(Auto)

(B) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, cor() which is qualitative.

names(Auto)
## [1] "mpg"          "cylinders"    "displacement" "horsepower"   "weight"      
## [6] "acceleration" "year"         "origin"       "name"
cor(Auto[1:8])
##                     mpg  cylinders displacement horsepower     weight
## mpg           1.0000000 -0.7776175   -0.8051269 -0.7784268 -0.8322442
## cylinders    -0.7776175  1.0000000    0.9508233  0.8429834  0.8975273
## displacement -0.8051269  0.9508233    1.0000000  0.8972570  0.9329944
## horsepower   -0.7784268  0.8429834    0.8972570  1.0000000  0.8645377
## weight       -0.8322442  0.8975273    0.9329944  0.8645377  1.0000000
## acceleration  0.4233285 -0.5046834   -0.5438005 -0.6891955 -0.4168392
## year          0.5805410 -0.3456474   -0.3698552 -0.4163615 -0.3091199
## origin        0.5652088 -0.5689316   -0.6145351 -0.4551715 -0.5850054
##              acceleration       year     origin
## mpg             0.4233285  0.5805410  0.5652088
## cylinders      -0.5046834 -0.3456474 -0.5689316
## displacement   -0.5438005 -0.3698552 -0.6145351
## horsepower     -0.6891955 -0.4163615 -0.4551715
## weight         -0.4168392 -0.3091199 -0.5850054
## acceleration    1.0000000  0.2903161  0.2127458
## year            0.2903161  1.0000000  0.1815277
## origin          0.2127458  0.1815277  1.0000000

(C) Use the lm() function to perform a multiple linear regressionwith mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:

lm.fit=lm(mpg ~. - name, data=Auto)
summary(lm.fit)
## 
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -9.5903 -2.1565 -0.1169  1.8690 13.0604 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -17.218435   4.644294  -3.707  0.00024 ***
## cylinders     -0.493376   0.323282  -1.526  0.12780    
## displacement   0.019896   0.007515   2.647  0.00844 ** 
## horsepower    -0.016951   0.013787  -1.230  0.21963    
## weight        -0.006474   0.000652  -9.929  < 2e-16 ***
## acceleration   0.080576   0.098845   0.815  0.41548    
## year           0.750773   0.050973  14.729  < 2e-16 ***
## origin         1.426141   0.278136   5.127 4.67e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared:  0.8215, Adjusted R-squared:  0.8182 
## F-statistic: 252.4 on 7 and 384 DF,  p-value: < 2.2e-16

i. Is there a relationship between the predictors and the response?

The p-values is less the the significance value of .05 indicating that there is a relationship between the predictors and the response.

ii. Which predictors appear to have a statistically significant relationship to the response?

A review of the t-stats p-vales for each predictor identifies that displacement, weight, year, and origin have a statistically significant relationship to the response.

iii. What does the coefficient for the year variable suggest?

The coefficient for the year variable suggest, with all other predictors remaining constant, that for each year that passes fuel efficiency increase by .75 mpg.

(D) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?

par (mfrow = c (2,2))
plot(lm.fit)

  • The Residuals vs Fitted plot suggest that there is a non-linear relationship between the predictors and the response variables
  • The Normal Q-Q plot suggest the residuals as being normally distributed.
  • The Scale-Location plot suggest that heteroskedasticity is present
  • The Residuals vs Leverage plot suggest that there is on leverage point identified as observation #14 and that there are quite a few outliers based on the Cook’s distance line.

(E) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?

lm.fit2 = lm(mpg ~. - name + weight:acceleration, data=Auto)
summary(lm.fit2)
## 
## Call:
## lm(formula = mpg ~ . - name + weight:acceleration, data = Auto)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -8.247 -2.048 -0.045  1.619 12.193 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## (Intercept)         -4.364e+01  5.811e+00  -7.511 4.18e-13 ***
## cylinders           -2.141e-01  3.078e-01  -0.696 0.487117    
## displacement         3.138e-03  7.495e-03   0.419 0.675622    
## horsepower          -4.141e-02  1.348e-02  -3.071 0.002287 ** 
## weight               4.027e-03  1.636e-03   2.462 0.014268 *  
## acceleration         1.629e+00  2.422e-01   6.726 6.36e-11 ***
## year                 7.821e-01  4.833e-02  16.184  < 2e-16 ***
## origin               1.033e+00  2.686e-01   3.846 0.000141 ***
## weight:acceleration -5.826e-04  8.408e-05  -6.928 1.81e-11 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.141 on 383 degrees of freedom
## Multiple R-squared:  0.8414, Adjusted R-squared:  0.838 
## F-statistic: 253.9 on 8 and 383 DF,  p-value: < 2.2e-16

With the inclusion of an interaction there has been an signification change in the statistically significant variables. All but two variables, cylinders and displacement, are now significant.

(F) Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings

lm.fit3 = lm(mpg ~ . - name + log(weight) + sqrt(horsepower) + I(displacement^2) + I(cylinders^2),data=Auto)
summary(lm.fit3)
## 
## Call:
## lm(formula = mpg ~ . - name + log(weight) + sqrt(horsepower) + 
##     I(displacement^2) + I(cylinders^2), data = Auto)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -9.2216 -1.4972 -0.1142  1.4184 11.9541 
## 
## Coefficients:
##                     Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        1.228e+02  4.381e+01   2.803  0.00532 ** 
## cylinders          3.360e-01  1.451e+00   0.232  0.81697    
## displacement      -3.765e-02  2.175e-02  -1.731  0.08421 .  
## horsepower         2.197e-01  6.684e-02   3.287  0.00111 ** 
## weight             1.181e-03  2.074e-03   0.569  0.56949    
## acceleration      -2.036e-01  1.004e-01  -2.028  0.04323 *  
## year               7.654e-01  4.526e-02  16.911  < 2e-16 ***
## origin             5.497e-01  2.679e-01   2.052  0.04088 *  
## log(weight)       -1.493e+01  6.714e+00  -2.223  0.02678 *  
## sqrt(horsepower)  -5.998e+00  1.493e+00  -4.018 7.06e-05 ***
## I(displacement^2)  6.788e-05  3.773e-05   1.799  0.07279 .  
## I(cylinders^2)    -1.067e-02  1.164e-01  -0.092  0.92702    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.904 on 380 degrees of freedom
## Multiple R-squared:  0.8654, Adjusted R-squared:  0.8615 
## F-statistic: 222.2 on 11 and 380 DF,  p-value: < 2.2e-16

The variable transformations that had a significant effect are sqrt(horsepower) I(displacement^2), and log(weight). The addition of these variable decreased the significance of some of the original significant variables.

Q10. This question should be answered using the Carseats data set.

(A) Fit a multiple regression model to predict Sales using Price, Urban, and US.

lm.fit10A = lm(Sales ~ Price+Urban+US, data=Carseats)
summary(lm.fit10A)
## 
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9206 -1.6220 -0.0564  1.5786  7.0581 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.043469   0.651012  20.036  < 2e-16 ***
## Price       -0.054459   0.005242 -10.389  < 2e-16 ***
## UrbanYes    -0.021916   0.271650  -0.081    0.936    
## USYes        1.200573   0.259042   4.635 4.86e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2335 
## F-statistic: 41.52 on 3 and 396 DF,  p-value: < 2.2e-16

(B) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!

  • With all other predictors held constant, the effect of Price increasing by $1 will effectively decrease sales by 54.459 units.
  • With all other predictors held constant, the average unit of sales in Urban areas is 21.916 units less than Rural areas.
  • With all other predictors held constant, the average unit of sales in a US based store are 1200.573 units more than non US stores.

(C) Write out the model in equation form, being careful to handle the qualitative variables properly.

With Urban and US being qualitative variables the assumption is yes = 1 and no = 0 then the equation for the model would be:

Sales = 13.043469 + (-0.54459 X Price) + (-0.021916 X Urban) + (1.200573 X US) + e.

(D) For which of the predictors can you reject the null hypothesis H0 : βj = 0?

The H(0) can be rejected for the predictors Price and US.

(E) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.

lm.fit10E = lm(Sales ~ Price + US, data=Carseats)
summary(lm.fit10E)
## 
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.9269 -1.6286 -0.0574  1.5766  7.0515 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.03079    0.63098  20.652  < 2e-16 ***
## Price       -0.05448    0.00523 -10.416  < 2e-16 ***
## USYes        1.19964    0.25846   4.641 4.71e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared:  0.2393, Adjusted R-squared:  0.2354 
## F-statistic: 62.43 on 2 and 397 DF,  p-value: < 2.2e-16

(F) How well do the models in (a) and (e) fit the data?

The R-squared values for both of models indicate that they are poor in explaining the variability in the models. Only 24% of the variability is explained by the models.

(G) Using the model from (e), obtain 95% confidence intervals for the coefficient(s)

confint.lm(lm.fit10E, level=.95)
##                   2.5 %      97.5 %
## (Intercept) 11.79032020 14.27126531
## Price       -0.06475984 -0.04419543
## USYes        0.69151957  1.70776632

(H) Is there evidence of outliers or high leverage observations in the model from (e)?

par(mfrow = c(2,2))
plot(lm.fit10E)

The Standardized residuals versus Leverage plot suggest the presence of outliers. These are the plotted potins that exceed the Cook’s distance of -2 to 2. There are no leverage points.

Q12. This problem involves simple linear regression without an intercept.

(A) Recall that the coefficient estimate ˆ β for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?

The circumstance where the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X is when the sum of squares for X is equal to the sum of squares for Y.

(B) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.

x=rnorm(100)
y=8*x+rnorm(100,0,0.0001)
lm.fit12B1 = lm(y~x)
summary(lm.fit12B1)
## 
## Call:
## lm(formula = y ~ x)
## 
## Residuals:
##        Min         1Q     Median         3Q        Max 
## -2.664e-04 -7.252e-05 -6.075e-06  6.089e-05  2.381e-04 
## 
## Coefficients:
##               Estimate Std. Error    t value Pr(>|t|)    
## (Intercept) -3.836e-06  1.003e-05     -0.382    0.703    
## x            8.000e+00  1.052e-05 760541.463   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 9.984e-05 on 98 degrees of freedom
## Multiple R-squared:      1,  Adjusted R-squared:      1 
## F-statistic: 5.784e+11 on 1 and 98 DF,  p-value: < 2.2e-16
lm.fit12B2 = lm(x~y)
summary(lm.fit12B2)
## 
## Call:
## lm(formula = x ~ y)
## 
## Residuals:
##        Min         1Q     Median         3Q        Max 
## -2.976e-05 -7.612e-06  7.600e-07  9.065e-06  3.329e-05 
## 
## Coefficients:
##              Estimate Std. Error   t value Pr(>|t|)    
## (Intercept) 4.796e-07  1.254e-06 3.820e-01    0.703    
## y           1.250e-01  1.644e-07 7.605e+05   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.248e-05 on 98 degrees of freedom
## Multiple R-squared:      1,  Adjusted R-squared:      1 
## F-statistic: 5.784e+11 on 1 and 98 DF,  p-value: < 2.2e-16

(C) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.

set.seed(234)
x <- rnorm(100)
y <- sample(x,100)
if (sum(x^2) == sum(y^2)) {'Sum of Squares for x and y are equal'}
## [1] "Sum of Squares for x and y are equal"
lm.fit12C1 = lm(y~x + 0)
lm.fit12C2 = lm(x~y + 0)
summary(lm.fit12C1)
## 
## Call:
## lm(formula = y ~ x + 0)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.0269 -0.6529  0.1558  0.5911  2.0742 
## 
## Coefficients:
##   Estimate Std. Error t value Pr(>|t|)
## x  0.03347    0.10045   0.333     0.74
## 
## Residual standard error: 0.9186 on 99 degrees of freedom
## Multiple R-squared:  0.00112,    Adjusted R-squared:  -0.00897 
## F-statistic: 0.111 on 1 and 99 DF,  p-value: 0.7397
summary(lm.fit12C2)
## 
## Call:
## lm(formula = x ~ y + 0)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.0175 -0.6635  0.1507  0.6183  2.1045 
## 
## Coefficients:
##   Estimate Std. Error t value Pr(>|t|)
## y  0.03347    0.10045   0.333     0.74
## 
## Residual standard error: 0.9186 on 99 degrees of freedom
## Multiple R-squared:  0.00112,    Adjusted R-squared:  -0.00897 
## F-statistic: 0.111 on 1 and 99 DF,  p-value: 0.7397