STA-6543 Assignment2
Anbumani Dhanapal
2025-02-27
2. Carefully explain the differences between the KNN classifier and KNN regression methods.
Though KNN Classifier and KNN Regression both are supervised learning algorithms under the concept of nearest neighbors, they differ in their output and application,
Output: KNN Classifier: Predicts a categorical (class label) output based on the majority vote of KNN. KNN Regression: Predicts a continuous value (numeric) by averaging target values of KNN.
Application: KNN Classifier is used for classification tasks (Ex: spam detection, heart disease Y/N). KNN Regression is used for regression tasks (Ex: predicting house prices, temperature).
Evaluation: KNN Classifier is evaluated using metrics like accuracy and confusion matrix. KNN Regression is evaluated using metrics like Mean Squared Error (MSE) and R-squared.
Measuring Performance: KNN Classifier: Metrics like accuracy, precision, recall, and F1-score (a balance of precision and recall). KNN Regression: Metrics like Mean Squared Error (MSE) or R-squared, which tells you how well the model explains the data.
#9. This question involves the use of multiple linear regression on the Auto data set.
##(a) Produce a scatterplot matrix which includes all of the variables in the data set.
library("ISLR")## Warning: package 'ISLR' was built under R version 4.4.2pairs(Auto)
##(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative. cor()
cor(Auto[, names(Auto) !="name"])## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000##(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:
###i. Is there a relationship between the predictors and the response?
model = lm(mpg ~. -name, data = Auto)
summary(model)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16Yes, there is significant relationship with few predictors and response. However, some predictors do not have a statistically significant effect on the response. R-squared value implies that 82% of the changes in the response can be explained by the predictors in this regression model
###ii. Which predictors appear to have a statistically significant relationship to the response? Referring to coefficients p-value, & p=0.05 as significance threshold. Displacement, weight, year, origin are 4 predictors that are statistically having significant relationship to mpg.
###iii. What does the coefficient for the year variable suggest? When every other predictor held constant, The coefficient for year is 0.750773, meaning that for each additional year, the mpg increases by approximately 0.75. This suggests that newer cars are more fuel-efficient, due to technology advancements resulting in mileage improvements over time.
##(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow = c(2,2))
plot(model)
Residual vs Fitted graph - This plot checks for non-linearity and heteroscedasticity. Ideally, residuals should be randomly scattered around zero. In this plot, a slight curve is visible, suggesting a possible non-linearity in the model. Some large residuals (outliers) are present, as seen by points far from the center. Observation –> shows that there is possible non-linearity in the data, indicating that a transformation might improve the model
QQ graph - This plot checks whether residuals follow a normal distribution. If points follow the diagonal line, the normality assumption holds. In this plot, deviation at the ends (tails) suggests the presence of outliers Observation –> shows that the residuals are right skewed; The model might not fully satisfy the normality assumption due to outliers
Scale Location - This checks for homoscedasticity (constant variance of residuals). Ideally, residuals should be spread evenly across fitted values. In this plot, the residuals appear somewhat evenly spread, though there are a few large residuals at the right. Observation –> graph shows that Potential heteroscedasticity (unequal variance), log(mpg) transformation might be required
Residuals vs leverage - This plot helps identify high-leverage points, which can have a strong influence on the model. Points beyond Cook’s distance lines are highly influential. Observations 327, 394, and 14 have high leverage and may significantly affect the regression. Observation –> shows that there are observations that stands out as a potential leverage point (labeled 14 on the graph)
##(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
summary(lm(formula = mpg ~ . * ., data = Auto[, -9]))##
## Call:
## lm(formula = mpg ~ . * ., data = Auto[, -9])
##
## Residuals:
## Min 1Q Median 3Q Max
## -7.6303 -1.4481 0.0596 1.2739 11.1386
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.548e+01 5.314e+01 0.668 0.50475
## cylinders 6.989e+00 8.248e+00 0.847 0.39738
## displacement -4.785e-01 1.894e-01 -2.527 0.01192 *
## horsepower 5.034e-01 3.470e-01 1.451 0.14769
## weight 4.133e-03 1.759e-02 0.235 0.81442
## acceleration -5.859e+00 2.174e+00 -2.696 0.00735 **
## year 6.974e-01 6.097e-01 1.144 0.25340
## origin -2.090e+01 7.097e+00 -2.944 0.00345 **
## cylinders:displacement -3.383e-03 6.455e-03 -0.524 0.60051
## cylinders:horsepower 1.161e-02 2.420e-02 0.480 0.63157
## cylinders:weight 3.575e-04 8.955e-04 0.399 0.69000
## cylinders:acceleration 2.779e-01 1.664e-01 1.670 0.09584 .
## cylinders:year -1.741e-01 9.714e-02 -1.793 0.07389 .
## cylinders:origin 4.022e-01 4.926e-01 0.816 0.41482
## displacement:horsepower -8.491e-05 2.885e-04 -0.294 0.76867
## displacement:weight 2.472e-05 1.470e-05 1.682 0.09342 .
## displacement:acceleration -3.479e-03 3.342e-03 -1.041 0.29853
## displacement:year 5.934e-03 2.391e-03 2.482 0.01352 *
## displacement:origin 2.398e-02 1.947e-02 1.232 0.21875
## horsepower:weight -1.968e-05 2.924e-05 -0.673 0.50124
## horsepower:acceleration -7.213e-03 3.719e-03 -1.939 0.05325 .
## horsepower:year -5.838e-03 3.938e-03 -1.482 0.13916
## horsepower:origin 2.233e-03 2.930e-02 0.076 0.93931
## weight:acceleration 2.346e-04 2.289e-04 1.025 0.30596
## weight:year -2.245e-04 2.127e-04 -1.056 0.29182
## weight:origin -5.789e-04 1.591e-03 -0.364 0.71623
## acceleration:year 5.562e-02 2.558e-02 2.174 0.03033 *
## acceleration:origin 4.583e-01 1.567e-01 2.926 0.00365 **
## year:origin 1.393e-01 7.399e-02 1.882 0.06062 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.695 on 363 degrees of freedom
## Multiple R-squared: 0.8893, Adjusted R-squared: 0.8808
## F-statistic: 104.2 on 28 and 363 DF, p-value: < 2.2e-16Using the standard threshold of 0.05, the significant interaction terms are given by:
cylinders:acceleration acceleration:year acceleration:originEuropean acceleration:originJapanese year:originEuropean year:originJapanese
##(f) Try a few different transformations of the variables, such as log(X), √X, X2. Comment on your findings. Log Transformation for weight
summary(lm(mpg ~ log(weight), data=Auto))##
## Call:
## lm(formula = mpg ~ log(weight), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.4315 -2.6752 -0.2888 1.9429 16.0136
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 209.9433 6.0002 34.99 <2e-16 ***
## log(weight) -23.4317 0.7534 -31.10 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.189 on 390 degrees of freedom
## Multiple R-squared: 0.7127, Adjusted R-squared: 0.7119
## F-statistic: 967.3 on 1 and 390 DF, p-value: < 2.2e-16Log Transformation: mpg ~ log(weight) Coefficient: -23.43, meaning a logarithmic increase in weight significantly decreases mpg. Multiple R-squared: 0.7127 Adjusted R-squared: 0.7119 Residual Standard Error (RSE): 4.189
Square Transform Weight
summary(lm(mpg ~ sqrt(weight), data = Auto))##
## Call:
## lm(formula = mpg ~ sqrt(weight), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.2402 -2.9005 -0.3708 2.0791 16.2296
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 69.67218 1.52649 45.64 <2e-16 ***
## sqrt(weight) -0.85560 0.02797 -30.59 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.239 on 390 degrees of freedom
## Multiple R-squared: 0.7058, Adjusted R-squared: 0.705
## F-statistic: 935.4 on 1 and 390 DF, p-value: < 2.2e-16Square Root Transformation: mpg ~ sqrt(weight) Coefficient: -0.85560, meaning an increase in sqrt(weight) leads to a decrease in mpg. Multiple R-squared: 0.7058 Adjusted R-squared: 0.705 Residual Standard Error (RSE): 4.239
Slightly lower fit than the log model (R² = 0.7058)
summary(lm(mpg ~ I(weight^2), data = Auto))##
## Call:
## lm(formula = mpg ~ I(weight^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -11.2813 -3.1744 -0.4708 2.2708 17.2506
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.447e+01 4.708e-01 73.22 <2e-16 ***
## I(weight^2) -1.150e-06 4.266e-08 -26.96 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.619 on 390 degrees of freedom
## Multiple R-squared: 0.6507, Adjusted R-squared: 0.6498
## F-statistic: 726.6 on 1 and 390 DF, p-value: < 2.2e-16Squared Transformation: mpg ~ I(weight^2) has the least of the R² value as 0.6507.
hence we can say log transformation model had the best results of these 3 comparisons
#10. This question should be answered using the Carseats data set. ##(a) Fit a multiple regression model to predict Sales using Price,Urban, and US.
library("ISLR")
?Carseats## starting httpd help server ... donehead(Carseats)## Sales CompPrice Income Advertising Population Price ShelveLoc Age Education
## 1 9.50 138 73 11 276 120 Bad 42 17
## 2 11.22 111 48 16 260 83 Good 65 10
## 3 10.06 113 35 10 269 80 Medium 59 12
## 4 7.40 117 100 4 466 97 Medium 55 14
## 5 4.15 141 64 3 340 128 Bad 38 13
## 6 10.81 124 113 13 501 72 Bad 78 16
## Urban US
## 1 Yes Yes
## 2 Yes Yes
## 3 Yes Yes
## 4 Yes Yes
## 5 Yes No
## 6 No Yesstr(Carseats)## 'data.frame': 400 obs. of 11 variables:
## $ Sales : num 9.5 11.22 10.06 7.4 4.15 ...
## $ CompPrice : num 138 111 113 117 141 124 115 136 132 132 ...
## $ Income : num 73 48 35 100 64 113 105 81 110 113 ...
## $ Advertising: num 11 16 10 4 3 13 0 15 0 0 ...
## $ Population : num 276 260 269 466 340 501 45 425 108 131 ...
## $ Price : num 120 83 80 97 128 72 108 120 124 124 ...
## $ ShelveLoc : Factor w/ 3 levels "Bad","Good","Medium": 1 2 3 3 1 1 3 2 3 3 ...
## $ Age : num 42 65 59 55 38 78 71 67 76 76 ...
## $ Education : num 17 10 12 14 13 16 15 10 10 17 ...
## $ Urban : Factor w/ 2 levels "No","Yes": 2 2 2 2 2 1 2 2 1 1 ...
## $ US : Factor w/ 2 levels "No","Yes": 2 2 2 2 1 2 1 2 1 2 ...lm.fit = lm(Sales ~ Price+Urban+US, data= Carseats)
summary(lm.fit)##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16##(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
Sales = 13.04 - 0.0545(Price) - 0.0219(UrbanYes) + 1.2057(USYes)
Each coefficient represents the expected change in Sales given a one-unit change in the predictor while holding other variables constant.
Interpretation of Each Coefficient:
-> Intercept (β₀ = 13.04) When Price = 0, Urban = No, and US = No, the expected Sales would be 13.04 units. Price of 0 is only for theoretical comparison.
-> Price (β₁ = -0.0545, p < 2e-16 → Significant) For each $1 increase in Price, Sales decrease by 0.0545 units. iN other words - When price increases by $1000, the number of carseats sold decrease by 54.459 units. P-value is extremely small, it has a strong negative effect on Sales.
-> UrbanYes (β₂ = -0.0219, p = 0.936 → Not Significant) A store’s sale is not affected by whether or not it is in a Urban area. (Insignificant). The high p-value (0.936) means we cannot conclude that being in an urban area affects Sales.
-> USYes (β₃ = 1.2057, p = 4.86e-06 → Significant) Stores in the US tend to have 1.2057 more units in Sales than non-US stores. Since the p-value is very small, this effect is also statistically significant.
##(c) Write out the model in equation form, being careful to handle the qualitative variables properly. Regression Model: Sales=13.0434689+(−0.0544588)×Price+(−0.0219162)×UrbanYes+(1.2005727)×USYes
##(d) For which of the predictors can you reject the null hypothesis H0 : βj = 0? We can reject the null hypothesis for the “Price” and “US” variable, as they have Positive impact on Sales
##(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
lm.fit2 = lm(Sales ~ Price+US, data= Carseats)
summary(lm.fit2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16Reduced Model Equation: Sales=13.03 − 0.05448×Price + 1.12 USYes
##(f) How well do the models in (a) and (e) fit the data?
summary(lm.fit)$adj.r.squared## [1] 0.2335123summary(lm.fit2)$adj.r.squared## [1] 0.2354305The adjusted R2 of the reduced model is slightly better than the full model.
##(g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
confint(lm.fit2)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632->Intercept (11.79 to 14.27): We are 95% confident that the true intercept lies between 12.40 and 14.89.
->Price (-0.0647 to -0.0442):
We are 95% confident that the effect of Price on Sales is between -0.0647 and -0.0442. Since the interval is entirely negative, this confirms that Price has a significant negative impact on Sales. A $1 increase in Price is expected to reduce Sales by approximately 0.0442 to 0.0647 units.
##(h) Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow=c(2,2))
plot(lm.fit2)
Based on Residuals vs Fitted The plot looks reasonable. However, a few
extreme residuals (e.g., point 377) might indicate outliers. Based on QQ
plot The tails deviate slightly, suggesting some amount of
non-normality. This might indicate outliers affecting the distribution.
Based on Scale-Location plot few points (e.g., 377) show higher residual
variance, possibly reinforcing the presence of outliers. Based on the
Residual vs Levrage Some points (e.g., 311, 377, 175) are close to
Cook’s distance lines, suggesting influential observations.
##########################################
#12. This problem involves simple linear regression without an intercept. ##(a) Recall that the coefficient estimate ˆ β for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X? For the regression coefficients to be equal, the following must hold: ∑xi^2 = ∑yi^2. Meaning the sum of squares of X must be equal to the sum of squares of Y. This occurs when X and Y have equal variance. When all points touch on a line through the origin (e.g.,Y=±X), the variances of X and Y will be the same, making the two regression coefficients equal.
##(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
# Set seed for reproducibility
set.seed(25)
# Generate X and Y with different variances
n <- 100
X <- rnorm(n, mean = 10, sd = 5)
Y <- 2 * X + rnorm(n, mean = 0, sd = 10) # Different variance
# Regression of Y on X (without intercept)
beta_y_on_x <- sum(X * Y) / sum(X^2)
# Regression of X on Y (without intercept)
beta_x_on_y <- sum(X * Y) / sum(Y^2)
# Print coefficients
cat("β (Y ~ X):", beta_y_on_x, "\n")## β (Y ~ X): 2.027536cat("β (X ~ Y):", beta_x_on_y, "\n")## β (X ~ Y): 0.4154969##(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
# Set seed for reproducibility
set.seed(35)
# Generate X
n <- 100
X <- rnorm(n, mean = 0, sd = 5)
# Set Y equal to X (ensuring equal variance)
Y <- X
# Regression of Y on X (without intercept)
beta_y_on_x <- sum(X * Y) / sum(X^2)
# Regression of X on Y (without intercept)
beta_x_on_y <- sum(X * Y) / sum(Y^2)
# Print coefficients
cat("β (Y ~ X):", beta_y_on_x, "\n")## β (Y ~ X): 1cat("β (X ~ Y):", beta_x_on_y, "\n")## β (X ~ Y): 1We could create a relationship as strong or as weak as desired, simply by permuting less/more the elements of x, and still we would see that βa=βb holds