Assignment2
Xiaoqi Jiang
2025-06-20
2. Carefully explain the differences between the KNN
classifier and KNN regression methods.
Classification is used for discrete values, whereas regression is used
with continuous ones.
KNN Classification: Predicts the class of a data point by majority vote of its k nearest neighbors. Each neighbor casts a vote for their class, and the class with the most votes is assigned to the new observation.
KNN Regression: Predicts a continuous value by taking the average (or weighted average) of the target values of its k nearest neighbors.
Key differences:
Output: KNN Classifier: Class labels (classification); KNN Regression: continuous values (regression)
Decision Mechanism: KNN Classifier: Uses majority voting (or weighted voting based on distance) to assign a class. KNN Regression: Uses averaging (or weighted averaging) of the neighbors’ target values.
Evaluation metrics: KNN Classifier: Evaluated using metrics like accuracy, precision, recall, or F1-score. KNN Regression: Evaluated using metrics like mean squared error (MSE), mean absolute error (MAE), or R-squared.
9. This question involves the use of multiple linear
regression on the Auto data set.
(a) Produce a scatterplot matrix which includes all of the
variables in the data set.
library(ISLR2)
data(Auto)
pairs(Auto)
(b) Compute the matrix of correlations between the variables
using the function cor(). You will need to exclude the
name variable, which is qualitative.
Auto_num <- Auto[, sapply(Auto, is.numeric)]
cor(Auto_num)## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000(c) Use the lm() function to perform a multiple
linear regression with mpg as the response and all other
variables except name as the predictors. Use the summary()
function to print the results. Comment on the output. For
instance:
auto_lm <- lm(mpg ~ . - name, data = Auto)
summary(auto_lm)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the
response?
Yes, there is a statistically significant relationship between the
predictors and the response (mpg). The F-statistic is
252.4 with a p-value < 2.2e-16, which is
far below the typical significance threshold of 0.05. This
indicates that at least one predictor is significantly associated with
mpg. Additionally, the Multiple R-squared value of
0.8215 suggests that approximately 82.15% of
the variability in mpg is explained by the model,
indicating a strong relationship between the predictors and the
response.
ii. Which predictors appear to have a statistically
significant relationship to the response?
displacement, weight, year, and
origin
iii. What does the coefficient for the year
variable suggest?
This suggests that newer cars tend to have higher fuel efficiency, as
the positive coefficient indicates a positive relationship between the
model year and mpg.
(d) Use the plot() function to produce
diagnostic plots of the linear regression fit. Comment on any problems
you see with the fit. Do the residual plots suggest any unusually large
outliers? Does the leverage plot identify any observations with
unusually high leverage?
par(mfrow = c(2, 2))
plot(auto_lm)
Residuals vs. Fitted: Random scatter with slight curvature; no clear
funnel shape, but residuals 13.06 and -9.59
suggest outliers (check standardized residuals > 3).
Normal Q-Q: Points mostly follow the line, but tail deviations (e.g.,
323, 330) indicate potential outliers.
Scale-Location: Flat line with slight spread increase; no strong heteroscedasticity.
Residuals vs. Leverage: 14O outside Cook’s distance is influential.
(e) Use the * and : symbols to fit
linear regression models with interaction effects. Do any interactions
appear to be statistically significant?
lm_interaction <- lm(mpg ~ weight * horsepower + weight * year, data = Auto)
summary(lm_interaction)##
## Call:
## lm(formula = mpg ~ weight * horsepower + weight * year, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -7.4858 -1.8374 -0.0902 1.4265 12.1980
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -5.065e+01 1.453e+01 -3.487 0.000545 ***
## weight 8.191e-03 5.210e-03 1.572 0.116736
## horsepower -1.944e-01 2.162e-02 -8.991 < 2e-16 ***
## year 1.451e+00 1.802e-01 8.052 1.02e-14 ***
## weight:horsepower 4.726e-05 5.645e-06 8.372 1.06e-15 ***
## weight:year -2.498e-04 6.453e-05 -3.871 0.000127 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.911 on 386 degrees of freedom
## Multiple R-squared: 0.8627, Adjusted R-squared: 0.8609
## F-statistic: 485 on 5 and 386 DF, p-value: < 2.2e-16Significant interactions: weight:horsepower (p = 1.06e-15) and weight:year (p = 0.000127) are statistically significant (p < 0.05), indicating their effects on mpg vary with each other. R-squared (0.8627) shows improved fit over the base model (0.8215).
(f) Try a few different transformations of the variables, such as \(\log(X)\), \(\sqrt{\smash[b]{X}}\), \(X^{\smash{2}}\). Comment on your findings.
lm_transform <- lm(mpg ~ log(weight) + sqrt(horsepower) + I(year^2), data = Auto)
summary(lm_transform)##
## Call:
## lm(formula = mpg ~ log(weight) + sqrt(horsepower) + I(year^2),
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.9902 -1.9582 -0.0019 1.7657 13.7262
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.490e+02 7.551e+00 19.727 <2e-16 ***
## log(weight) -1.909e+01 1.135e+00 -16.827 <2e-16 ***
## sqrt(horsepower) -2.623e-01 1.872e-01 -1.401 0.162
## I(year^2) 5.027e-03 3.133e-04 16.044 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.137 on 388 degrees of freedom
## Multiple R-squared: 0.8397, Adjusted R-squared: 0.8385
## F-statistic: 677.5 on 3 and 388 DF, p-value: < 2.2e-16Significant terms: log(weight) (p < 2e-16) and I(year^2) (p < 2e-16) are significant; sqrt(horsepower) (p = 0.162) is not. R-squared (0.8397) is higher than the base model, suggesting transformations improve fit, especially for weight and year. Residuals (max 13.7262) indicate possible outliers still present.
10. This question should be answered using the
Carseats data set.
(a) Fit a multiple regression model to predict
Sales using Price, Urban, and
US.
attach(Carseats)fit<-lm(Sales~Price+Urban+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + Urban + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16(b) Provide an interpretation of each coefficient in the
model. Be careful—some of the variables in the model are
qualitative!
From the table above, Price and US are
significant predictors of Sales, for every $1 increase in
my price, sales decrease by $53. Sales inside the US are $1,200 higher
than sales outside of the US. Urban has no effect on Sales.
(c) Write out the model in equation form, being careful to
handle the qualitative variables properly.
\(Sales = 13.043469 - 0.054459Price -
0.021916Urban_{Yes} + 1.200573US_{Yes}\)
(d) For which of the predictors can you reject the null
hypothesis \(H_0 :\beta_j
=0\)?
Price and US
(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
fit<-lm(Sales~Price+US)
summary(fit)##
## Call:
## lm(formula = Sales ~ Price + US)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16(f) How well do the models in (a) and (e) fit the
data?
Not well, each model explains around 23% of the variances in sales.
(g) Using the model from (e), obtain 95% confidence intervals for the coefficient(s).
confint(fit)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632(h) Is there evidence of outliers or high leverage
observations in the model from (e)?
Yes, there is evidence of outliers (observations with unusually large
residuals) visible in the Residuals vs. Fitted and Normal Q-Q plots.
Some observations are labeled as having high leverage in the Residuals
vs. Leverage plot, indicating unusual predictor values. However, none of
these points appear to be extremely influential.
12. This problem involves simple linear regression without an
intercept.
(a) Recall that the coefficient estimate \(\hat{\beta}\) for the linear regression of
\(Y\) onto \(X\) without an intercept is given by \((3.38)\). Under what circumstance is the
coefficient estimate for the regression of \(X\) onto \(Y\) the same as the coefficient estimate for
the regression of \(Y\) onto \(X\)?
For Y on X:
\(\hat{\beta}_{Y \sim X} = \frac{\sum x_i
y_i}{\sum x_i^2}\)
For X on Y:
\(\hat{\beta}_{X \sim Y} = \frac{\sum x_i
y_i}{\sum y_i^2}\)
Therefore, if \(\sum x_i^2 = \sum y_i^2\), we can get the same coefficient estimate.
(b) Generate an example in R with \(n = 100\) observations in which the
coefficient estimate for the regression of \(X\) onto \(Y\) is different from the
coefficient estimate for the regression of \(Y\) onto \(X\).
set.seed(1)
x <- 1:100
sum(x^2)## [1] 338350y <- 2 * x + rnorm(100, sd = 0.1)
sum(y^2)## [1] 1353606fit.Y <- lm(y ~ x + 0)
fit.X <- lm(x ~ y + 0)
summary(fit.Y)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.223590 -0.062560 0.004426 0.058507 0.230926
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 2.0001514 0.0001548 12920 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.09005 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.669e+08 on 1 and 99 DF, p-value: < 2.2e-16summary(fit.X)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.115418 -0.029231 -0.002186 0.031322 0.111795
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 5.00e-01 3.87e-05 12920 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.04502 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 1.669e+08 on 1 and 99 DF, p-value: < 2.2e-16(c) Generate an example in R with\(n = 100\) observations in which the
coefficient estimate for the regression of \(X\) onto \(Y\) is the same as the coefficient
estimate for the regression of \(Y\)
onto \(X\).
x <- 1:100
sum(x^2)## [1] 338350y <- x
sum(y^2)## [1] 338350fit.Y <- lm(y ~ x + 0)
fit.X <- lm(x ~ y + 0)
summary(fit.Y)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.338e-13 -1.589e-15 2.900e-17 1.540e-15 7.683e-15
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 1.000e+00 4.072e-17 2.456e+16 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.369e-14 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 6.03e+32 on 1 and 99 DF, p-value: < 2.2e-16summary(fit.X)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.338e-13 -1.589e-15 2.900e-17 1.540e-15 7.683e-15
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 1.000e+00 4.072e-17 2.456e+16 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.369e-14 on 99 degrees of freedom
## Multiple R-squared: 1, Adjusted R-squared: 1
## F-statistic: 6.03e+32 on 1 and 99 DF, p-value: < 2.2e-16