Assignment2
Karthik Munnam
2024-02-22
library(ISLR)
library(tidyverse)## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.3 ✔ readr 2.1.4
## ✔ forcats 1.0.0 ✔ stringr 1.5.0
## ✔ ggplot2 3.4.3 ✔ tibble 3.2.1
## ✔ lubridate 1.9.2 ✔ tidyr 1.3.0
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(car)## Loading required package: carData
##
## Attaching package: 'car'
##
## The following object is masked from 'package:dplyr':
##
## recode
##
## The following object is masked from 'package:purrr':
##
## somelibrary(ggplot2)R Markdown
This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:
2. Carefully explain the differences between the KNN classifier and KNN regression methods?
Answer:
The K-Nearest Neighbors (KNN) classification method is a machine learning technique that groups data points into predefined categories. The K data points that are closest to the new observation, expressed in Manhattan or Euclidean distances, are found using an algorithm in this method. Next, the class label of the majority of these closest neighbors is assigned to the new observation, making it a member of that class. KNN classification yields discrete class labels, and metrics such as accuracy, precision, recall, and F1-score are used to evaluate the model’s performance and give insight into its classification capabilities.
On the other hand, K-Nearest Neighbors (KNN) regression is a method for making continuous value predictions based on the traits of the K nearest neighbors. In order to determine the K nearest neighbors, the method computes the distances between each new data point and every existing data point in the training set, much like in KNN classification. To forecast the continuous value for the new observation, KNN regression, on the other hand, computes the average (or weighted average) of the target values of these neighbors, as opposed to identifying the majority class label. The accuracy and goodness-of-fit of the regression model are evaluated using common metrics for KNN regression, such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R-squared.
9. This question involves the use of multiple linear regression on the Auto data set.
(a) Produce a scatterplot matrix which includes all of the variables in the data set.
data("Auto", package = "ISLR") pairs(Auto[,1:9])
It is evident from the plot above that there is a strong correlation between displacement and weight and horsepower.
(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative.
cor(Auto[,!colnames(Auto) %in% c("name")]) ## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output.
Auto1 = lm(mpg ~ . -name, data=Auto)
summary(Auto1)##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16i. Is there a relationship between the predictors and the response?
By testing the null hypothesis that all regression coefficients are zero, it is evident that there is a substantial link between the predictors (all the variables except “name”) and the response (mpg). With a p-value of less than 2.2e-16 and an F-statistic of 252.4, the model appears to have significant predictive potential and provides strong evidence against the null hypothesis.
ii. Which predictors appear to have a statistically significant relationship to the response?
The predictors (weight, displacement, origin, and year) and response (mpg) have a statistically significant relationship.
iii. What does the coefficient for the year variable suggest?
Answer: For the year variable, the co-efficient is 0.750773(0.751).
(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?
par(mfrow=c(2,2))
plot(Auto1)
It is clear that the data is normally distributed from the QQ Plot. There is one high leverage point (observation 14) and a small number of outliers (between 2 and -2) in the Standardized Residuals, indicating that the data is somewhat non-linear.
(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?
Auto2 <- lm(mpg ~ cylinders * displacement+ weight *displacement, data = Auto[, 1:8])
summary(Auto2)##
## Call:
## lm(formula = mpg ~ cylinders * displacement + weight * displacement,
## data = Auto[, 1:8])
##
## Residuals:
## Min 1Q Median 3Q Max
## -13.2934 -2.5184 -0.3476 1.8399 17.7723
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 5.262e+01 2.237e+00 23.519 < 2e-16 ***
## cylinders 7.606e-01 7.669e-01 0.992 0.322
## displacement -7.351e-02 1.669e-02 -4.403 1.38e-05 ***
## weight -9.888e-03 1.329e-03 -7.438 6.69e-13 ***
## cylinders:displacement -2.986e-03 3.426e-03 -0.872 0.384
## displacement:weight 2.128e-05 5.002e-06 4.254 2.64e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.103 on 386 degrees of freedom
## Multiple R-squared: 0.7272, Adjusted R-squared: 0.7237
## F-statistic: 205.8 on 5 and 386 DF, p-value: < 2.2e-16It is clear from the summary above that displacement and weight have statistical significance.cylinders:displacement is not statistically significant, though.
Auto3 = lm(mpg ~ . + displacement:cylinders + displacement:
+ displacement:weight,
data=Auto[, 1:8])
summary(Auto3)##
## Call:
## lm(formula = mpg ~ . + displacement:cylinders + displacement:+displacement:weight,
## data = Auto[, 1:8])
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.0609 -1.7589 -0.0494 1.5790 12.1496
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -4.795e+00 4.515e+00 -1.062 0.28883
## cylinders -1.091e-01 5.965e-01 -0.183 0.85502
## displacement -7.186e-02 1.363e-02 -5.273 2.25e-07 ***
## horsepower -3.457e-02 1.304e-02 -2.651 0.00836 **
## weight -1.030e-02 1.064e-03 -9.680 < 2e-16 ***
## acceleration 6.618e-02 8.817e-02 0.751 0.45334
## year 7.840e-01 4.566e-02 17.171 < 2e-16 ***
## origin 5.475e-01 2.643e-01 2.071 0.03901 *
## cylinders:displacement 1.186e-03 2.715e-03 0.437 0.66251
## displacement:weight 2.141e-05 3.712e-06 5.768 1.66e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.967 on 382 degrees of freedom
## Multiple R-squared: 0.8588, Adjusted R-squared: 0.8555
## F-statistic: 258.2 on 9 and 382 DF, p-value: < 2.2e-16It is evident from the summary above that displacement:weight has statistical significance.Displacement:cylinders, however, does not have statistical significance.
(f) Try a few different transformations of the variables, such as log(X), ’X, X2. Comment on your findings.
Auto.lm4 <- lm(mpg~ log(displacement) + sqrt(cylinders) + I(horsepower^2), data = Auto[, 1:8])
summary(Auto.lm4)##
## Call:
## lm(formula = mpg ~ log(displacement) + sqrt(cylinders) + I(horsepower^2),
## data = Auto[, 1:8])
##
## Residuals:
## Min 1Q Median 3Q Max
## -15.1056 -2.5077 -0.5845 2.1013 18.8673
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 8.040e+01 3.498e+00 22.985 < 2e-16 ***
## log(displacement) -1.170e+01 1.325e+00 -8.827 < 2e-16 ***
## sqrt(cylinders) 1.905e+00 1.914e+00 0.995 0.32040
## I(horsepower^2) -1.115e-04 3.787e-05 -2.945 0.00342 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.338 on 388 degrees of freedom
## Multiple R-squared: 0.6934, Adjusted R-squared: 0.6911
## F-statistic: 292.6 on 3 and 388 DF, p-value: < 2.2e-16(horsepower^2) is significant & horsepower is not significant, log(displacement) & displacement both remain significant, and sart(cylinders) & cylinders both not significant.
10. This question should be answered using the Carseats data set.
(a) Fit a multiple regression model to predict Sales using Price, Urban, and US:
data("Carseats", package = "ISLR")Carseats1<- lm(Sales ~ Price + Urban + US, data = Carseats)
summary(Carseats1)##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16Testing the null hypothesis that all regression coefficients are 0 reveals a significant relationship between the response (Sales) and the predictors (Price, Urban, and US). Significant predictive power in the model is suggested by the F-statistic (41.52) and the remarkably low p-value (< 2.2e-16), which show strong evidence against the null hypothesis.
(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!
It is evident from the prior summary that the coefficient for the variable “price” is -0.054459, meaning that sales are predicted to fall by an average of 0.054459 units for every unit rise in price.Regarding the “Urban” variable, the coefficient is -0.021916, indicating that being in an urban region is linked to a 0.021916 unit decline in average sales when compared to rural areas.The coefficient for the “US” variable is 1.200573, meaning that, in comparison to locations outside the US, being located in the US is linked to an average sales increase of 1.200573 units.
(c) Write out the model in equation form, being careful to handle the qualitative variables properly.
Sales=13.043469-0.054459∗Price-0.021916∗UrbanYes+1.200573* USYes
(d) For which of the predictors can you reject the null hypothesis H0 : #j = 0?
Price and US based on the p-value
(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.
Carseats2 <- lm(Sales ~ Price + US, data = Carseats)
summary(Carseats2)##
## Call:
## lm(formula = Sales ~ Price + US, data = Carseats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16Testing the null hypothesis that all regression coefficients are 0 reveals a substantial association between the response (sales) and the predictors (price). Significant predictive power in the model is suggested by the F-statistic (41.52) and the remarkably low p-value (< 2.2e-16), which show strong evidence against the null hypothesis.
(f) How well do the models in (a) and (e) fit the data?
Just over 23% of the variance in the response variable (sales) is explained by models (a) and (e); these models perform poorly in this situation due to their low r-square values.
(g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
confint(Carseats2)## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632(h) Is there evidence of outliers or high leverage observations in the model from (e)?
par(mfrow=c(2,2))
plot(Carseats2)
The model from (e) shows no evidence of outliers or high leverage observations.
12. This problem involves simple linear regression without an intercept.
(a) Recall that the coefficient estimate ˆ # for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?
Only when there is a perfect one-to-one link between X and Y does the coefficient stay the same in both scenarios. This scenario suggests that there is an equal increase or reduction of one unit in Y for every unit increase or decrease in X.Further predicated on: ∑ x^2=∑ y^2
(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.
set.seed(123)
n <- 100
X <- rnorm(n)
Y <- 2*X + rnorm(n) linear regression of X onto Y
XY <- lm(X ~ Y)
XY_coef <- coef(XY)[2]XY_coef## Y
## 0.3964576linear regression of Y onto X
YX <- lm(Y ~ X)
YX_coef <- coef(YX)[2] YX_coef## X
## 1.947528The coefficient estimate for X in the regression of X onto Y is roughly 1.9475. This suggests that we should expect X to increase by roughly 1.9475 units for every unit increase in Y.The coefficient estimate for Y in the regression of Y onto X is roughly 0.3965. This implies that we anticipate an average rise in Y of about 0.3965 units for every unit increase in X.
(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.
set.seed(100)
n2 <- 100
X2 <- rnorm(n2)
Y2 <- X2 linear regression of X onto Y
XY2 <- lm(X2 ~ Y2)
XY2_coef <- coef(XY2)[2]XY2_coef## Y2
## 1linear regression of Y onto X
YX2 <- lm(Y2 ~ X2)
YX2_coef <- coef(YX2)[2]YX2_coef## X2
## 1Interpretation:
X (or Y) has a coefficient estimate of 1. This suggests that we anticipate X (or Y) to increase by one unit on average for every unit increase in Y (or X).