Hw2-RPubs
NastaranTaabodi -dew050
2026-07-07
QUESTION 2:
#The the differences between KNN Classifier and KNN Regression: both KNN Classifier and KNN Regression work the same basic way: given a new data point, both methods find the K nearest neighbors in the training data. The difference lies in what they do with those neighbors and the type of outcome they predict.
library(ISLR2)
head(Auto)## mpg cylinders displacement horsepower weight acceleration year origin
## 1 18 8 307 130 3504 12.0 70 1
## 2 15 8 350 165 3693 11.5 70 1
## 3 18 8 318 150 3436 11.0 70 1
## 4 16 8 304 150 3433 12.0 70 1
## 5 17 8 302 140 3449 10.5 70 1
## 6 15 8 429 198 4341 10.0 70 1
## name
## 1 chevrolet chevelle malibu
## 2 buick skylark 320
## 3 plymouth satellite
## 4 amc rebel sst
## 5 ford torino
## 6 ford galaxie 500pairs(Auto)
The scatterplot matrix shows the relationships between all variables
in the Auto dataset. We can observe that mpg has a negative relationship with weight, displacement and horsepower, and a positive relationship with year.
Compute the matrix of correlations between the variables.
Auto_numeric <- Auto[, !names(Auto) %in% c("name")]
cor(Auto_numeric)## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000Through the the correlation matrix I found these :
displacement (-0.81), cylinders (-0.78) and horsepower (-0.78).
This suggests that heavier, larger engine cars tend to have worse fuel efficiency,
while newer cars tend to have better fuel efficiency.
library(ISLR2)
lm.fit <- lm(mpg ~ . - name, data = Auto)
summary(lm.fit)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16The F-statistic is 252.4 with p-value < 2.2e-16, indicating a strong relationship between predictors and mpg.
R-squared = 0.8182, meaning the model explains 81.82% of variance in mpg.
Statistically significant predictors: weight, year, origin, displacement.
The year coefficient (0.75) suggests newer cars have better fuel efficiency.
par(mar = c(4, 4, 2, 1))
par(mfrow = c(2, 2)) # arrange plots in a 2x2 grid
plot(lm.fit)
Diagnostic plots interpretation:
1. Residuals vs Fitted: slight non-linearity detected in the data
2. Q-Q Plot: residuals are mostly normal but with slight deviation at upper tail
3. Scale-Location: variance is mostly constant, observation 327 is a potential outlier
4. Residuals vs Leverage: observation 14 has unusually high leverage
lm.fit1 <- lm(mpg ~ weight * horsepower, data = Auto)
summary(lm.fit1)##
## Call:
## lm(formula = mpg ~ weight * horsepower, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.7725 -2.2074 -0.2708 1.9973 14.7314
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.356e+01 2.343e+00 27.127 < 2e-16 ***
## weight -1.077e-02 7.738e-04 -13.921 < 2e-16 ***
## horsepower -2.508e-01 2.728e-02 -9.195 < 2e-16 ***
## weight:horsepower 5.355e-05 6.649e-06 8.054 9.93e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.93 on 388 degrees of freedom
## Multiple R-squared: 0.7484, Adjusted R-squared: 0.7465
## F-statistic: 384.8 on 3 and 388 DF, p-value: < 2.2e-16lm.fit2 <- lm(mpg ~ weight * year, data = Auto)
summary(lm.fit2)##
## Call:
## lm(formula = mpg ~ weight * year, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -8.0397 -1.9956 -0.0983 1.6525 12.9896
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.105e+02 1.295e+01 -8.531 3.30e-16 ***
## weight 2.755e-02 4.413e-03 6.242 1.14e-09 ***
## year 2.040e+00 1.718e-01 11.876 < 2e-16 ***
## weight:year -4.579e-04 5.907e-05 -7.752 8.02e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.193 on 388 degrees of freedom
## Multiple R-squared: 0.8339, Adjusted R-squared: 0.8326
## F-statistic: 649.3 on 3 and 388 DF, p-value: < 2.2e-16lm.fit3 <- lm(mpg ~ displacement * horsepower, data = Auto)
summary(lm.fit3)##
## Call:
## lm(formula = mpg ~ displacement * horsepower, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.9391 -2.3373 -0.5816 2.1698 17.5771
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 5.305e+01 1.526e+00 34.77 <2e-16 ***
## displacement -9.805e-02 6.682e-03 -14.67 <2e-16 ***
## horsepower -2.343e-01 1.959e-02 -11.96 <2e-16 ***
## displacement:horsepower 5.828e-04 5.193e-05 11.22 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.944 on 388 degrees of freedom
## Multiple R-squared: 0.7466, Adjusted R-squared: 0.7446
## F-statistic: 381 on 3 and 388 DF, p-value: < 2.2e-16lm.log <- lm(mpg ~ log(horsepower), data = Auto)
summary(lm.log)##
## Call:
## lm(formula = mpg ~ log(horsepower), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.2299 -2.7818 -0.2322 2.6661 15.4695
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 108.6997 3.0496 35.64 <2e-16 ***
## log(horsepower) -18.5822 0.6629 -28.03 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.501 on 390 degrees of freedom
## Multiple R-squared: 0.6683, Adjusted R-squared: 0.6675
## F-statistic: 785.9 on 1 and 390 DF, p-value: < 2.2e-16lm.sqrt <- lm(mpg ~ sqrt(horsepower), data = Auto)
summary(lm.sqrt)##
## Call:
## lm(formula = mpg ~ sqrt(horsepower), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -13.9768 -3.2239 -0.2252 2.6881 16.1411
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 58.705 1.349 43.52 <2e-16 ***
## sqrt(horsepower) -3.503 0.132 -26.54 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.665 on 390 degrees of freedom
## Multiple R-squared: 0.6437, Adjusted R-squared: 0.6428
## F-statistic: 704.6 on 1 and 390 DF, p-value: < 2.2e-16lm.sq <- lm(mpg ~ I(horsepower^2), data = Auto)
summary(lm.sq)##
## Call:
## lm(formula = mpg ~ I(horsepower^2), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.529 -3.798 -1.049 3.240 18.528
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.047e+01 4.466e-01 68.22 <2e-16 ***
## I(horsepower^2) -5.665e-04 2.827e-05 -20.04 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 5.485 on 390 degrees of freedom
## Multiple R-squared: 0.5074, Adjusted R-squared: 0.5061
## F-statistic: 401.7 on 1 and 390 DF, p-value: < 2.2e-16conclusion: All three transformations are statistically significant.
The log transformation provides the best fit becuase the highest Adj R² = 0.6675 and lowest residual standard error = 4.501.
However, all transformations perform worse than the full model (Adj R² = 0.8182) because the full model includes more predictors.
The non-linear transformations confirm that horsepower has a curved relationship with mpg.
Question 10 :
library(ISLR2)
head(Carseats)## Sales CompPrice Income Advertising Population Price ShelveLoc Age Education
## 1 9.50 138 73 11 276 120 Bad 42 17
## 2 11.22 111 48 16 260 83 Good 65 10
## 3 10.06 113 35 10 269 80 Medium 59 12
## 4 7.40 117 100 4 466 97 Medium 55 14
## 5 4.15 141 64 3 340 128 Bad 38 13
## 6 10.81 124 113 13 501 72 Bad 78 16
## Urban US
## 1 Yes Yes
## 2 Yes Yes
## 3 Yes Yes
## 4 Yes Yes
## 5 Yes No
## 6 No Yesstr(Carseats)## 'data.frame': 400 obs. of 11 variables:
## $ Sales : num 9.5 11.22 10.06 7.4 4.15 ...
## $ CompPrice : num 138 111 113 117 141 124 115 136 132 132 ...
## $ Income : num 73 48 35 100 64 113 105 81 110 113 ...
## $ Advertising: num 11 16 10 4 3 13 0 15 0 0 ...
## $ Population : num 276 260 269 466 340 501 45 425 108 131 ...
## $ Price : num 120 83 80 97 128 72 108 120 124 124 ...
## $ ShelveLoc : Factor w/ 3 levels "Bad","Good","Medium": 1 2 3 3 1 1 3 2 3 3 ...
## $ Age : num 42 65 59 55 38 78 71 67 76 76 ...
## $ Education : num 17 10 12 14 13 16 15 10 10 17 ...
## $ Urban : Factor w/ 2 levels "No","Yes": 2 2 2 2 2 1 2 2 1 1 ...
## $ US : Factor w/ 2 levels "No","Yes": 2 2 2 2 1 2 1 2 1 2 ...summary(Carseats)## Sales CompPrice Income Advertising
## Min. : 0.000 Min. : 77 Min. : 21.00 Min. : 0.000
## 1st Qu.: 5.390 1st Qu.:115 1st Qu.: 42.75 1st Qu.: 0.000
## Median : 7.490 Median :125 Median : 69.00 Median : 5.000
## Mean : 7.496 Mean :125 Mean : 68.66 Mean : 6.635
## 3rd Qu.: 9.320 3rd Qu.:135 3rd Qu.: 91.00 3rd Qu.:12.000
## Max. :16.270 Max. :175 Max. :120.00 Max. :29.000
## Population Price ShelveLoc Age Education
## Min. : 10.0 Min. : 24.0 Bad : 96 Min. :25.00 Min. :10.0
## 1st Qu.:139.0 1st Qu.:100.0 Good : 85 1st Qu.:39.75 1st Qu.:12.0
## Median :272.0 Median :117.0 Medium:219 Median :54.50 Median :14.0
## Mean :264.8 Mean :115.8 Mean :53.32 Mean :13.9
## 3rd Qu.:398.5 3rd Qu.:131.0 3rd Qu.:66.00 3rd Qu.:16.0
## Max. :509.0 Max. :191.0 Max. :80.00 Max. :18.0
## Urban US
## No :118 No :142
## Yes:282 Yes:258
##
##
##
## So based on this information sale and price are numerical and both Urban and US are categorical.
lm.fit <- lm(Sales ~ Price + US + Urban , data = Carseats)
summary(lm.fit)##
## Call:
## lm(formula = Sales ~ Price + US + Urban, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16so based on the summary, residual Standard Error: 2.469, the Adjusted R-squared: 0.2354 so the model explains 23.54% of variance in Sales, the F-statistic p-value < 2.2e-16 indicates that the overall model is statistically significant, meaning at least one predictor has a significant relationship with Sales.
- Price (p < 2e-16) and US (p = 4.86e-06) are significant predictors but Urban (p = 0.936) is not a significant predictor
Coefficient Interpretation: Price (-0.054) means that for every $1 increase in Price, Sales decrease by 0.054 units. This is statistically significant (p < 2e-16). the USYes (1.200) means that the US stores sell 1,200 more units than non-US stores. This is statistically significant (p = 4.86e-06) but the UrbanYes (-0.022): Urban location has no significant effect on Sales becuase the p-value = 0.936, not significant.
part c :
#I have all the coefficient, B0, B1,B2,B3 SO and two categorical predictors:
Sales = 13.043469 - 0.054459 * Price - 0.021916 * UrbanYes + 1.200573 * USYes
Where: UrbanYes = 1 if Urban, 0 if not Urban
USYes = 1 if US, 0 if not US
Four possible equations depending on Urban and US values:
Urban=No, US=No: Sales = 13.043469 - 0.054459 * Price
Urban=Yes, US=No: Sales = 13.021553 - 0.054459 * Price
Urban=No, US=Yes: Sales = 14.244042 - 0.054459 * Price
Urban=Yes, US=Yes: Sales = 14.222126 - 0.054459 * Price
##Part D!
#So to reject the H0:Bj=0 I checked te pvalues so I will reject B0 for the one that has the pvalue < 0.05! # both price and USYes hac=ve pvalue <0.05 so I will reject the null hypothesis for these two predictor that have a statistically significant relationship with Sales but regarding the UrbanYes, the p-value = 0.936 so it fails to reject H0! Urban is NOT significant.
##Part E! smaller model using ONLY the significant predictors from part (d)!!
lm.fit2 <- lm(Sales ~ Price + US, data = Carseats)
summary(lm.fit2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16Both Price and US remain significant (p < 0.05) and the Adjusted R-squared = 0.2354 so the model explains 23.54% of the variance in Sales and this is a weak fit but our predictors are still significant. Also the Residual Standard Error = 2.469 it means that on average our model predictions are off by 2,469 units of Sales.
in addition, based on the Residual standard error: 2.469 and the mean 7.496 then the RSE of the 2.469 is 32.9% which tells me that the model is weak fit.
Part f: Comparing Model (a) and Model (e)
Model (a): Sales ~ Price + Urban + US
Adjusted R² = 0.2335
RSE = 2.472
F-statistic = 41.52
Model (e): Sales ~ Price + US
Adjusted R² = 0.2354
RSE = 2.469
F-statistic = 62.43
Conclusion: Model (e) is slightly better than Model (a) because:
1. Higher Adjusted R² (0.2354 > 0.2335)
2. Lower RSE (2.469 < 2.472)
3. Higher F-statistic (62.43 > 41.52)
4. Simpler model with fewer predictors
Removing Urban improved the model slightly since Urban was not a significant predictor.
#Part g: obtain 95% confidence intervals for the coefficients of model e.
confint(lm.fit2, level = 0.95)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632#The 95% confidence intervals show that the intercept [11.79, 14.27]: baseline Sales is between 11,790 and 14,270 units. for price , Price [-0.0648, -0.0442]:doesnt incule 0, also the both range is negative so Price has a negative effect on Sales. # USYes [0.692, 1.708]:doesnt include 0, both positive, US stores sell between 692 and 1,708 more units. Therefore all predictors are statistically significant.
#part h :
par(mar = c(4, 4, 2, 1)) # reduce margins
par(mfrow = c(2, 2)) # 2x2 grid
plot(lm.fit2)
based on the plots : In the Residuals vs Fitted plot the red line is mostly flat, no serious non-linearity detected. A few minor outliers on the left side.Also in the Q-Q Plot residuals are mostly normally distributed with slight departures at both tails.
in the Scale-Location the variance is mostly constant, homoscedasticity assumption is met. Finaly in the Residuals vs Leverag, there is no observations fall outside Cook’s distance, meaning there are no high leverage observations that significantly influence the model.Therefore there is no strong evidence of outliers or high leverage observations in model (e). The model assumptions are reasonably met.
##___________________________________________
Question 12 :
Part a: the coefficient estimate for regressing Y onto X (without intercept) is: beta_hat(Y on X) = sum(xi * yi) / sum(xi^2)
\[\hat{\beta}_{Y \text{ on } X} = \frac{\sum x_i y_i}{\sum x_i^2}\] #The coefficient estimate for regressing X onto Y is the same but we just have the sum(yi^2) in deniminator. so these two coefficients are equal when the denominators are equal, that is: sum(xi^2) = sum(yi^2). In other words, the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X when X and Y have the same sum of squares (the same total “spread” or scale).
##part b : #For this part I just need to have different variance for X and Y :
set.seed(1)
x <- rnorm(100)
y <- 2 * x + rnorm(100)lm.y.on.x <- lm(y ~ x + 0)
summary(lm.y.on.x)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.9154 -0.6472 -0.1771 0.5056 2.3109
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 1.9939 0.1065 18.73 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9586 on 99 degrees of freedom
## Multiple R-squared: 0.7798, Adjusted R-squared: 0.7776
## F-statistic: 350.7 on 1 and 99 DF, p-value: < 2.2e-16lm.x.on.y <- lm(x ~ y + 0)
summary(lm.x.on.y)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.8699 -0.2368 0.1030 0.2858 0.8938
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.39111 0.02089 18.73 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4246 on 99 degrees of freedom
## Multiple R-squared: 0.7798, Adjusted R-squared: 0.7776
## F-statistic: 350.7 on 1 and 99 DF, p-value: < 2.2e-16#while both models has teh same r-square and tvalues but the coefficients are clearly different! X onto Y :0.39111 and Y onto X is 1.99390. It happens becuase they have differnt variance/scale becuase we defined Y as y <- 2 * x + rnorm(100).
#Part c : Generate an example in R with n=100 observations where the coefficient for regressing X onto Y is the SAME as regressing Y onto X.
set.seed(1)
x <- rnorm(100)
y <- sample(x, 100)
lm.y.on.x2 <- lm(y ~ x + 0)
summary(lm.y.on.x2)##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.1665 -0.4995 0.1140 0.6945 2.2833
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x -0.07768 0.10020 -0.775 0.44
##
## Residual standard error: 0.9021 on 99 degrees of freedom
## Multiple R-squared: 0.006034, Adjusted R-squared: -0.004006
## F-statistic: 0.601 on 1 and 99 DF, p-value: 0.4401lm.x.on.y2 <- lm(x ~ y + 0)
summary(lm.x.on.y2)##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.2182 -0.4969 0.1595 0.6782 2.4017
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y -0.07768 0.10020 -0.775 0.44
##
## Residual standard error: 0.9021 on 99 degrees of freedom
## Multiple R-squared: 0.006034, Adjusted R-squared: -0.004006
## F-statistic: 0.601 on 1 and 99 DF, p-value: 0.4401