Assignment 3
2023-02-17
Question 2
Carefully explain the differences between the KNN classifier and KNN regression methods.
The KNN classifier first and the regresion methods both identify the \(K\) points in the training data that are closest to \(x_0\), represented by \(Ν_0\). But the KNN classifier estimates the conditional probability for class \(j\) as the fraction of points in \(N_0\) whose response values equal \(j\) while the KNN regression estimates \(f(x_0)\) using the average of all the training responses in \(N_0\).
Question 9
9. This question involves the use of multiple linear regression on the Auto data set.
library(ISLR)
head(Auto)## mpg cylinders displacement horsepower weight acceleration year origin
## 1 18 8 307 130 3504 12.0 70 1
## 2 15 8 350 165 3693 11.5 70 1
## 3 18 8 318 150 3436 11.0 70 1
## 4 16 8 304 150 3433 12.0 70 1
## 5 17 8 302 140 3449 10.5 70 1
## 6 15 8 429 198 4341 10.0 70 1
## name
## 1 chevrolet chevelle malibu
## 2 buick skylark 320
## 3 plymouth satellite
## 4 amc rebel sst
## 5 ford torino
## 6 ford galaxie 500(a) Produce a scatterplot matrix which includes all of the variables in the data set.
pairs(Auto)
(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative.
Correlation_of_Auto <- cor(Auto[-9])
Correlation_of_Auto## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:
COF_model <- lm(mpg~. -name , data = Auto)
summary(COF_model)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the
response?
Yes there is since we recieved an R square of .8215
ii. Which predictors appear to have a statistically significant
relationship to the response?
Here we can see that displacement, year,
origin, and weight have there p-values lower
than the significance level of .05 thus we can say there is significance
relationship there.
iii. What does the coefficient for the year variable
suggest?
If we increase by one unit while keeping all the others the same then
mpg will decrease.
(d) Use the plot() function to produce diagnostic plots of the
linear regression fit. Comment on any problems you see with the fit. Do
the residual plots suggest any unusually large outliers? Does the
leverage plot identify any observations with unusually high
leverage?
plot(COF_model)
The plots suggest there is a couple outliers that are significantly large. The leverage plot has marked two points with unusually high leverage.
(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
horsepower_displacement <- lm(mpg~.-name + horsepower*displacement, data=Auto)
origin_horsepower <- lm(mpg~.-name + horsepower*origin, data=Auto)
summary(horsepower_displacement)##
## Call:
## lm(formula = mpg ~ . - name + horsepower * displacement, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.7010 -1.6009 -0.0967 1.4119 12.6734
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.894e+00 4.302e+00 -0.440 0.66007
## cylinders 6.466e-01 3.017e-01 2.143 0.03275 *
## displacement -7.487e-02 1.092e-02 -6.859 2.80e-11 ***
## horsepower -1.975e-01 2.052e-02 -9.624 < 2e-16 ***
## weight -3.147e-03 6.475e-04 -4.861 1.71e-06 ***
## acceleration -2.131e-01 9.062e-02 -2.351 0.01921 *
## year 7.379e-01 4.463e-02 16.534 < 2e-16 ***
## origin 6.891e-01 2.527e-01 2.727 0.00668 **
## displacement:horsepower 5.236e-04 4.813e-05 10.878 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.912 on 383 degrees of freedom
## Multiple R-squared: 0.8636, Adjusted R-squared: 0.8608
## F-statistic: 303.1 on 8 and 383 DF, p-value: < 2.2e-16summary(origin_horsepower)##
## Call:
## lm(formula = mpg ~ . - name + horsepower * origin, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.277 -1.875 -0.225 1.570 12.080
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -2.196e+01 4.396e+00 -4.996 8.94e-07 ***
## cylinders -5.275e-01 3.028e-01 -1.742 0.0823 .
## displacement -1.486e-03 7.607e-03 -0.195 0.8452
## horsepower 8.173e-02 1.856e-02 4.404 1.38e-05 ***
## weight -4.710e-03 6.555e-04 -7.186 3.52e-12 ***
## acceleration -1.124e-01 9.617e-02 -1.168 0.2434
## year 7.327e-01 4.780e-02 15.328 < 2e-16 ***
## origin 7.695e+00 8.858e-01 8.687 < 2e-16 ***
## horsepower:origin -7.955e-02 1.074e-02 -7.405 8.44e-13 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.116 on 383 degrees of freedom
## Multiple R-squared: 0.8438, Adjusted R-squared: 0.8406
## F-statistic: 258.7 on 8 and 383 DF, p-value: < 2.2e-16Yes horsepower and origin interaction seem to be significant while the horsepower displacement interaction doesnt change.
(f) Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings.
summary(lm(mpg ~ . -name + log(origin), data=Auto))##
## Call:
## lm(formula = mpg ~ . - name + log(origin), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.0095 -2.0785 -0.0982 1.9856 13.3608
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.479e+01 4.722e+00 -3.131 0.00187 **
## cylinders -4.897e-01 3.212e-01 -1.524 0.12821
## displacement 2.398e-02 7.653e-03 3.133 0.00186 **
## horsepower -1.818e-02 1.371e-02 -1.326 0.18549
## weight -6.710e-03 6.551e-04 -10.243 < 2e-16 ***
## acceleration 7.910e-02 9.822e-02 0.805 0.42110
## year 7.770e-01 5.178e-02 15.005 < 2e-16 ***
## origin -3.169e+00 1.907e+00 -1.661 0.09743 .
## log(origin) 8.366e+00 3.436e+00 2.435 0.01535 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.307 on 383 degrees of freedom
## Multiple R-squared: 0.8242, Adjusted R-squared: 0.8205
## F-statistic: 224.5 on 8 and 383 DF, p-value: < 2.2e-16summary(lm(mpg ~ . -name + log(acceleration), data=Auto))##
## Call:
## lm(formula = mpg ~ . - name + log(acceleration), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.7931 -2.0052 -0.1279 1.9299 13.1085
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.552e+01 1.479e+01 3.077 0.00224 **
## cylinders -2.796e-01 3.193e-01 -0.876 0.38172
## displacement 8.042e-03 7.805e-03 1.030 0.30344
## horsepower -3.434e-02 1.401e-02 -2.450 0.01473 *
## weight -5.343e-03 6.854e-04 -7.795 6.15e-14 ***
## acceleration 2.167e+00 4.782e-01 4.532 7.82e-06 ***
## year 7.560e-01 4.978e-02 15.186 < 2e-16 ***
## origin 1.329e+00 2.724e-01 4.877 1.58e-06 ***
## log(acceleration) -3.513e+01 7.886e+00 -4.455 1.10e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.249 on 383 degrees of freedom
## Multiple R-squared: 0.8303, Adjusted R-squared: 0.8267
## F-statistic: 234.2 on 8 and 383 DF, p-value: < 2.2e-16summary(lm(mpg ~ . -name + I(horsepower^2), data=Auto))##
## Call:
## lm(formula = mpg ~ . - name + I(horsepower^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.5497 -1.7311 -0.2236 1.5877 11.9955
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.3236564 4.6247696 0.286 0.774872
## cylinders 0.3489063 0.3048310 1.145 0.253094
## displacement -0.0075649 0.0073733 -1.026 0.305550
## horsepower -0.3194633 0.0343447 -9.302 < 2e-16 ***
## weight -0.0032712 0.0006787 -4.820 2.07e-06 ***
## acceleration -0.3305981 0.0991849 -3.333 0.000942 ***
## year 0.7353414 0.0459918 15.989 < 2e-16 ***
## origin 1.0144130 0.2545545 3.985 8.08e-05 ***
## I(horsepower^2) 0.0010060 0.0001065 9.449 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.001 on 383 degrees of freedom
## Multiple R-squared: 0.8552, Adjusted R-squared: 0.8522
## F-statistic: 282.8 on 8 and 383 DF, p-value: < 2.2e-16summary(lm(mpg ~ . -name + I(weight^2), data=Auto))##
## Call:
## lm(formula = mpg ~ . - name + I(weight^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.4706 -1.6701 -0.1488 1.6383 12.5429
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.479e+00 4.614e+00 0.321 0.74867
## cylinders -2.840e-01 2.917e-01 -0.974 0.33083
## displacement 1.371e-02 6.793e-03 2.019 0.04418 *
## horsepower -2.435e-02 1.243e-02 -1.959 0.05083 .
## weight -2.049e-02 1.580e-03 -12.970 < 2e-16 ***
## acceleration 6.571e-02 8.895e-02 0.739 0.46055
## year 7.999e-01 4.615e-02 17.331 < 2e-16 ***
## origin 7.418e-01 2.603e-01 2.850 0.00461 **
## I(weight^2) 2.237e-06 2.341e-07 9.556 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.994 on 383 degrees of freedom
## Multiple R-squared: 0.8558, Adjusted R-squared: 0.8528
## F-statistic: 284.2 on 8 and 383 DF, p-value: < 2.2e-16summary(lm(mpg ~ . -name + sqrt(horsepower), data=Auto))##
## Call:
## lm(formula = mpg ~ . - name + sqrt(horsepower), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.5402 -1.6717 -0.0778 1.4861 11.9754
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.299e+01 7.251e+00 5.929 6.82e-09 ***
## cylinders 6.037e-02 2.928e-01 0.206 0.836748
## displacement -5.870e-03 7.156e-03 -0.820 0.412560
## horsepower 4.239e-01 4.532e-02 9.353 < 2e-16 ***
## weight -3.285e-03 6.604e-04 -4.975 9.87e-07 ***
## acceleration -3.342e-01 9.705e-02 -3.443 0.000638 ***
## year 7.398e-01 4.536e-02 16.308 < 2e-16 ***
## origin 9.159e-01 2.526e-01 3.626 0.000326 ***
## sqrt(horsepower) -1.050e+01 1.039e+00 -10.104 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.961 on 383 degrees of freedom
## Multiple R-squared: 0.8591, Adjusted R-squared: 0.8561
## F-statistic: 291.8 on 8 and 383 DF, p-value: < 2.2e-16Using log does change the significance of the variable but not by that much. While the other two methods do not change the significance.
Question 10
This question should be answered using the Carseats data set.
(a) Fit a multiple regression model to predict Sales using Price, Urban, and US.
library(ISLR)
head(Carseats)## Sales CompPrice Income Advertising Population Price ShelveLoc Age Education
## 1 9.50 138 73 11 276 120 Bad 42 17
## 2 11.22 111 48 16 260 83 Good 65 10
## 3 10.06 113 35 10 269 80 Medium 59 12
## 4 7.40 117 100 4 466 97 Medium 55 14
## 5 4.15 141 64 3 340 128 Bad 38 13
## 6 10.81 124 113 13 501 72 Bad 78 16
## Urban US
## 1 Yes Yes
## 2 Yes Yes
## 3 Yes Yes
## 4 Yes Yes
## 5 Yes No
## 6 No Yesregression_model <- lm(Sales~Price+Urban+US, data = Carseats)
summary(regression_model)##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16(b) Provide an interpretation of each coefficient in the
model. Be careful—some of the variables in the model are
qualitative!
Since the price is neggative that means for every increase in a dollar
sales drop by 54.5 units. Also if the carseat is in Urban area it is
22.9 units lower than in rural locations. And if the carseat is in the
US then it is 1200.6 units higher than if it wasn’t.
(c) Write out the model in equation form, being careful to
handle the qualitative variables properly.
Sales = 13.043469 -.054459\(*\)Price-.021916\(*\)Uban+1.2006\(*\)US
(d) For which of the predictors can you reject the null
hypothesis \(H0 :βj
=0\)?
Since both p-values of Price and US are less than the significance level
we reject the null hypothesis. (e) On the basis of your response
to the previous question, fit a smaller model that only uses the
predictors for which there is evidence of association with the
outcome.
regression_model2 <- lm(Sales~Price+US, data = Carseats)
summary(regression_model2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16(f) How well do the models in (a) and (e) fit the
data?
Both models have the same r squared but the model in e has a higher
adjusted r squard so it is technically better.
(g) Using the model from (e), obtain 95% confidence intervals
for the coefficient(s).
confidence_interval<-confint(regression_model2,level = 0.95)
sprintf('95%% confidence interval for the intercept is [%.4f,%.4f]',confidence_interval[1,1],confidence_interval[1,2])## [1] "95% confidence interval for the intercept is [11.7903,14.2713]"sprintf('95%% confidence interval for the coefficient of Price is [%.4f,%.4f]',confidence_interval[2,1],confidence_interval[2,2])## [1] "95% confidence interval for the coefficient of Price is [-0.0648,-0.0442]"sprintf('95%% confidence interval for the coefficient of US:Yes is [%.4f,%.4f]',confidence_interval[3,1],confidence_interval[3,2])## [1] "95% confidence interval for the coefficient of US:Yes is [0.6915,1.7078]"(h) Is there evidence of outliers or high leverage observations in the model from (e)?
plot(rstudent(regression_model), predict(regression_model2))
From what I see it doesn’t seem to be any outliers in the data since its all from -3 to 3.
Question 12
This problem involves simple linear regression without an
intercept.
(a) Recall that the coefficient estimate \(βˆ\) for the linear regression of \(Y\) onto \(X\) without an intercept is given by
(3.38). Under what circumstance is the coefficient estimate for the
regression of \(X\) onto \(Y\) the same as the coefficient estimate
for the regression of \(Y\) onto \(X\)?
The circumstance where the regression of \(X\) onto \(Y\) is the same as \(Y\) onto \(X\) is when \(Σ_j
x^2_j = Σ_j y^2_j\)
(b) Generate an example in R with \(n
= 100\) observations in which the coefficient estimate for the
regression of \(X\) onto \(Y\) is different from the coefficient
estimate for the regression of \(Y\)
onto \(X\).
x <- 1:100
sum(x^2)## [1] 338350y <- 2 * x + rnorm(100, sd = 0.1)
sum(y^2)## [1] 1353583(c) Generate an example in R with \(n = 100\) observations in which the coefficient estimate for the regression of \(X\) onto \(Y\) is the same as the coefficient estimate for the regression of \(Y\) onto \(X\).
x <- 1:100
sum(x^2)## [1] 338350y <- 100:1
sum(y^2)## [1] 338350