KNN classifier and KNN regression models are very similar, both using
the principle of finding nearest neighbors. However, they are used for
different types of problems(classification vs regression).
KNN classification is used for classifying data points into
categories or classes based on the majority class of their nearest
neighbors. The output is that of a class label representing the
predicted category. These models are more flexible than KNN regression
as it is non-linear.
KNN regression is to predict a continuous value for each data point
based on the weighted average of the target values of its nearest
neighbors. The output for KNN regression models is a continuous value
representing the predicted target for each data point.
pairs(Auto)
drops <- c("name")
cor(Auto[ , !(names(Auto) %in% drops)])
## mpg cylinders displacement horsepower weight
## mpg 1.0000000 -0.7776175 -0.8051269 -0.7784268 -0.8322442
## cylinders -0.7776175 1.0000000 0.9508233 0.8429834 0.8975273
## displacement -0.8051269 0.9508233 1.0000000 0.8972570 0.9329944
## horsepower -0.7784268 0.8429834 0.8972570 1.0000000 0.8645377
## weight -0.8322442 0.8975273 0.9329944 0.8645377 1.0000000
## acceleration 0.4233285 -0.5046834 -0.5438005 -0.6891955 -0.4168392
## year 0.5805410 -0.3456474 -0.3698552 -0.4163615 -0.3091199
## origin 0.5652088 -0.5689316 -0.6145351 -0.4551715 -0.5850054
## acceleration year origin
## mpg 0.4233285 0.5805410 0.5652088
## cylinders -0.5046834 -0.3456474 -0.5689316
## displacement -0.5438005 -0.3698552 -0.6145351
## horsepower -0.6891955 -0.4163615 -0.4551715
## weight -0.4168392 -0.3091199 -0.5850054
## acceleration 1.0000000 0.2903161 0.2127458
## year 0.2903161 1.0000000 0.1815277
## origin 0.2127458 0.1815277 1.0000000
auto.model <- lm(mpg ~ . -name, data = Auto)
summary(auto.model)
##
## Call:
## lm(formula = mpg ~ . - name, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.5903 -2.1565 -0.1169 1.8690 13.0604
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -17.218435 4.644294 -3.707 0.00024 ***
## cylinders -0.493376 0.323282 -1.526 0.12780
## displacement 0.019896 0.007515 2.647 0.00844 **
## horsepower -0.016951 0.013787 -1.230 0.21963
## weight -0.006474 0.000652 -9.929 < 2e-16 ***
## acceleration 0.080576 0.098845 0.815 0.41548
## year 0.750773 0.050973 14.729 < 2e-16 ***
## origin 1.426141 0.278136 5.127 4.67e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.328 on 384 degrees of freedom
## Multiple R-squared: 0.8215, Adjusted R-squared: 0.8182
## F-statistic: 252.4 on 7 and 384 DF, p-value: < 2.2e-16
It seems like every predictor has a relationship with the response
variable, however some relationships are stronger than others. We can
see the strength of these relationships as represented by the Estimate
value for each predictor in the model results.
There are 4 predictors that have a relationship with the response
variable, mpg:
- With every one unit increase of displacement we can notice an increase
in the response variable by 0.019896 units.
- With every one unit increase of weight we can notice an decrease in
the response variable by 0.006474 units.
- With every one unit increase of year we can notice an increase in the
response variable by 0.750773 units.
- With every one unit increase of origin we can notice an increase in
the response variable by 1.426141 units.
The coefficient for the year variable suggests that newer cars have
better gas mileage than older cars.
par(mfrow=c(2,2)) # Change the panel layout to 2 x 2
plot(auto.model)
From the residuals vs fitted plot we don’t notice any particular
pattern, however it does not illustrate any problems.
If we look at the Q-Q Residuals plot we can notice a few outliers on the
right side of the plot, this could possibly create a right skew in the
distribution of the data.
The residuals vs leverage plot seems to indicate that data point ‘14’ is
causing issues and creating unusually high leverage.
auto2.model <- lm(mpg ~ displacement * weight * year * origin, data = Auto)
summary(auto2.model)
##
## Call:
## lm(formula = mpg ~ displacement * weight * year * origin, data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.2148 -1.5712 -0.0295 1.3200 14.3708
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.799e+02 1.174e+02 -1.532 0.1262
## displacement -1.019e+00 1.047e+00 -0.973 0.3312
## weight 8.198e-02 5.106e-02 1.605 0.1092
## year 3.017e+00 1.540e+00 1.959 0.0508 .
## origin 1.984e+01 9.933e+01 0.200 0.8418
## displacement:weight 2.297e-04 3.996e-04 0.575 0.5656
## displacement:year 1.249e-02 1.371e-02 0.911 0.3629
## weight:year -1.177e-03 6.687e-04 -1.760 0.0793 .
## displacement:origin 1.421e+00 1.002e+00 1.417 0.1572
## weight:origin -3.674e-02 4.435e-02 -0.828 0.4080
## year:origin -2.073e-01 1.297e+00 -0.160 0.8731
## displacement:weight:year -2.856e-06 5.197e-06 -0.550 0.5830
## displacement:weight:origin -3.598e-04 3.932e-04 -0.915 0.3608
## displacement:year:origin -1.874e-02 1.307e-02 -1.434 0.1524
## weight:year:origin 4.531e-04 5.779e-04 0.784 0.4335
## displacement:weight:year:origin 4.824e-06 5.106e-06 0.945 0.3454
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.901 on 376 degrees of freedom
## Multiple R-squared: 0.8672, Adjusted R-squared: 0.8619
## F-statistic: 163.6 on 15 and 376 DF, p-value: < 2.2e-16
auto3.model <- lm(mpg ~ displacement * weight * cylinders * origin, data = Auto)
summary(auto3.model)
##
## Call:
## lm(formula = mpg ~ displacement * weight * cylinders * origin,
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.6298 -2.4463 -0.3365 1.7266 18.6207
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.758e+02 7.391e+01 2.379 0.0178 *
## displacement -7.982e-01 5.128e-01 -1.557 0.1204
## weight -4.327e-02 2.975e-02 -1.454 0.1467
## cylinders -3.555e+01 1.757e+01 -2.024 0.0437 *
## origin -1.258e+02 6.952e+01 -1.810 0.0711 .
## displacement:weight 2.160e-04 1.995e-04 1.083 0.2795
## displacement:cylinders 2.016e-01 1.148e-01 1.756 0.0798 .
## weight:cylinders 1.106e-02 6.983e-03 1.583 0.1142
## displacement:origin 6.852e-01 4.855e-01 1.411 0.1590
## weight:origin 3.854e-02 2.831e-02 1.361 0.1743
## cylinders:origin 3.512e+01 1.690e+01 2.078 0.0384 *
## displacement:weight:cylinders -6.208e-05 4.513e-05 -1.376 0.1698
## displacement:weight:origin -2.008e-04 1.938e-04 -1.036 0.3009
## displacement:cylinders:origin -1.943e-01 1.129e-01 -1.722 0.0860 .
## weight:cylinders:origin -1.137e-02 6.838e-03 -1.662 0.0973 .
## displacement:weight:cylinders:origin 6.167e-05 4.480e-05 1.376 0.1695
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.008 on 376 degrees of freedom
## Multiple R-squared: 0.7464, Adjusted R-squared: 0.7362
## F-statistic: 73.76 on 15 and 376 DF, p-value: < 2.2e-16
auto4.model <- lm(mpg ~ acceleration * weight * cylinders * origin, data = Auto)
summary(auto4.model)
##
## Call:
## lm(formula = mpg ~ acceleration * weight * cylinders * origin,
## data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.3852 -2.3927 -0.3423 1.8985 16.4051
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 5.063e+02 1.610e+02 3.146 0.00179 **
## acceleration -2.732e+01 1.067e+01 -2.561 0.01082 *
## weight -1.444e-01 5.683e-02 -2.541 0.01147 *
## cylinders -1.106e+02 3.761e+01 -2.942 0.00347 **
## origin -3.736e+02 1.496e+02 -2.497 0.01295 *
## acceleration:weight 8.089e-03 3.681e-03 2.197 0.02860 *
## acceleration:cylinders 6.585e+00 2.532e+00 2.601 0.00967 **
## weight:cylinders 3.388e-02 1.277e-02 2.653 0.00832 **
## acceleration:origin 2.305e+01 1.004e+01 2.296 0.02222 *
## weight:origin 1.136e-01 5.143e-02 2.208 0.02782 *
## cylinders:origin 9.402e+01 3.662e+01 2.568 0.01063 *
## acceleration:weight:cylinders -2.020e-03 8.443e-04 -2.392 0.01723 *
## acceleration:weight:origin -6.992e-03 3.388e-03 -2.064 0.03972 *
## acceleration:cylinders:origin -5.789e+00 2.468e+00 -2.345 0.01953 *
## weight:cylinders:origin -2.966e-02 1.236e-02 -2.400 0.01687 *
## acceleration:weight:cylinders:origin 1.826e-03 8.201e-04 2.226 0.02659 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.901 on 376 degrees of freedom
## Multiple R-squared: 0.7598, Adjusted R-squared: 0.7502
## F-statistic: 79.29 on 15 and 376 DF, p-value: < 2.2e-16
The model with the most interactions that are significantly significant
is ‘mpg ~ acceleration * weight * cylinders * origin’. Every interaction
in this model is below the P Value is less than 0.05 with the lowest
being weight:cylinders and acceleration:cylinders.
auto5.model <- lm(mpg ~ log(displacement) + log(weight) + log(year) + log(origin), data = Auto)
summary(auto5.model)
##
## Call:
## lm(formula = mpg ~ log(displacement) + log(weight) + log(year) +
## log(origin), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.9990 -2.0231 -0.0356 1.6133 13.1472
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -81.1209 17.6056 -4.608 5.54e-06 ***
## log(displacement) -0.2356 1.0033 -0.235 0.8145
## log(weight) -18.7195 1.7166 -10.905 < 2e-16 ***
## log(year) 58.7383 3.4907 16.827 < 2e-16 ***
## log(origin) 1.3818 0.4872 2.836 0.0048 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.139 on 387 degrees of freedom
## Multiple R-squared: 0.8399, Adjusted R-squared: 0.8383
## F-statistic: 507.6 on 4 and 387 DF, p-value: < 2.2e-16
auto5.model <- lm(mpg ~ I(displacement^2) + sqrt(weight) + sqrt(year) + sqrt(origin), data = Auto)
summary(auto5.model)
##
## Call:
## lm(formula = mpg ~ I(displacement^2) + sqrt(weight) + sqrt(year) +
## sqrt(origin), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9.8091 -1.9758 0.0239 1.6948 12.8861
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -6.033e+01 7.565e+00 -7.975 1.74e-14 ***
## I(displacement^2) 2.869e-05 7.226e-06 3.971 8.53e-05 ***
## sqrt(weight) -8.363e-01 4.683e-02 -17.859 < 2e-16 ***
## sqrt(year) 1.426e+01 8.273e-01 17.231 < 2e-16 ***
## sqrt(origin) 2.739e+00 6.728e-01 4.070 5.69e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.167 on 387 degrees of freedom
## Multiple R-squared: 0.8371, Adjusted R-squared: 0.8354
## F-statistic: 497.1 on 4 and 387 DF, p-value: < 2.2e-16
auto6.model <- lm(mpg ~ I(displacement^2) * sqrt(weight) * sqrt(year) * sqrt(origin), data = Auto)
summary(auto6.model)
##
## Call:
## lm(formula = mpg ~ I(displacement^2) * sqrt(weight) * sqrt(year) *
## sqrt(origin), data = Auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -10.2921 -1.5660 0.0372 1.3243 13.8466
##
## Coefficients:
## Estimate Std. Error
## (Intercept) -1.175e+03 6.295e+02
## I(displacement^2) -9.086e-02 5.047e-02
## sqrt(weight) 2.561e+01 1.347e+01
## sqrt(year) 1.419e+02 7.209e+01
## sqrt(origin) 6.858e+02 5.780e+02
## I(displacement^2):sqrt(weight) 1.592e-03 9.724e-04
## I(displacement^2):sqrt(year) 1.041e-02 5.756e-03
## sqrt(weight):sqrt(year) -3.023e+00 1.542e+00
## I(displacement^2):sqrt(origin) 9.380e-02 5.030e-02
## sqrt(weight):sqrt(origin) -1.800e+01 1.251e+01
## sqrt(year):sqrt(origin) -7.772e+01 6.610e+01
## I(displacement^2):sqrt(weight):sqrt(year) -1.823e-04 1.108e-04
## I(displacement^2):sqrt(weight):sqrt(origin) -1.645e-03 9.708e-04
## I(displacement^2):sqrt(year):sqrt(origin) -1.077e-02 5.736e-03
## sqrt(weight):sqrt(year):sqrt(origin) 2.048e+00 1.430e+00
## I(displacement^2):sqrt(weight):sqrt(year):sqrt(origin) 1.888e-04 1.106e-04
## t value Pr(>|t|)
## (Intercept) -1.866 0.0628 .
## I(displacement^2) -1.800 0.0726 .
## sqrt(weight) 1.901 0.0580 .
## sqrt(year) 1.968 0.0498 *
## sqrt(origin) 1.187 0.2362
## I(displacement^2):sqrt(weight) 1.637 0.1025
## I(displacement^2):sqrt(year) 1.809 0.0713 .
## sqrt(weight):sqrt(year) -1.960 0.0508 .
## I(displacement^2):sqrt(origin) 1.865 0.0630 .
## sqrt(weight):sqrt(origin) -1.439 0.1510
## sqrt(year):sqrt(origin) -1.176 0.2405
## I(displacement^2):sqrt(weight):sqrt(year) -1.646 0.1007
## I(displacement^2):sqrt(weight):sqrt(origin) -1.694 0.0911 .
## I(displacement^2):sqrt(year):sqrt(origin) -1.878 0.0611 .
## sqrt(weight):sqrt(year):sqrt(origin) 1.432 0.1529
## I(displacement^2):sqrt(weight):sqrt(year):sqrt(origin) 1.707 0.0886 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.915 on 376 degrees of freedom
## Multiple R-squared: 0.8659, Adjusted R-squared: 0.8606
## F-statistic: 161.9 on 15 and 376 DF, p-value: < 2.2e-16
It seems like taking the square root of each variable making the
relationship between the predictor and response variable stronger.
Taking the square of ‘displacement’ while taking the square root of the
rest makes all of the predictors very statistically significant.
df <- Carseats
df$Urban <- as.factor(df$Urban)
df$US <- as.factor(df$US)
carseat.model <- lm(Sales ~ Price + Urban + US, data = df)
summary(carseat.model)
##
## Call:
## lm(formula = Sales ~ Price + Urban + US, data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9206 -1.6220 -0.0564 1.5786 7.0581
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.043469 0.651012 20.036 < 2e-16 ***
## Price -0.054459 0.005242 -10.389 < 2e-16 ***
## UrbanYes -0.021916 0.271650 -0.081 0.936
## USYes 1.200573 0.259042 4.635 4.86e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.472 on 396 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2335
## F-statistic: 41.52 on 3 and 396 DF, p-value: < 2.2e-16
The results of the the linear regression model suggest that Price and USYes seem to have the strongest relationship with the ‘Sales’ variable. Both of these predictors are statistically significant. Urban does not seem to have a significant effect on Sales.
- For every unit increase in Price there is an decrease in Sales by -0.054459 units.
- For every unit increase in USYes there is an increase in Sales by 1.200573 units.
Sales^ = 13.043469 − 0.054459 ⋅ Price − 0.021916 ⋅ Urban + 1.200573 ⋅
US
The predictors which reject the null hypothesis are Price and USYes.
Urban fails to reject the null hypothesis as the P Value for this
predictor is not less than 0.05.
carseat2.model <- lm(Sales ~ Price + US, data = df)
summary(carseat2.model)
##
## Call:
## lm(formula = Sales ~ Price + US, data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -6.9269 -1.6286 -0.0574 1.5766 7.0515
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.03079 0.63098 20.652 < 2e-16 ***
## Price -0.05448 0.00523 -10.416 < 2e-16 ***
## USYes 1.19964 0.25846 4.641 4.71e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.469 on 397 degrees of freedom
## Multiple R-squared: 0.2393, Adjusted R-squared: 0.2354
## F-statistic: 62.43 on 2 and 397 DF, p-value: < 2.2e-16
While both models fit the data moderately well, the second model is
slightly better. The model is statistically significant overall as can
be by the F-statistic and its corresponding p-value. However the RSE
suggests that there is still some unexplained variability in the
model.
confint(carseat2.model)
## 2.5 % 97.5 %
## (Intercept) 11.79032020 14.27126531
## Price -0.06475984 -0.04419543
## USYes 0.69151957 1.70776632
par(mfrow=c(2,2)) # Change the panel layout to 2 x 2
plot(carseat2.model)
There do not seem to be any strong outliers in the data, however it does
seem that there is an observation that is causing high leverage in the
model based on the Residuals vs Leverage plot.
X onto Y:
β̂=∑ixiyi / ∑jy2j
Y onto X:
β̂=∑ixiyi /
∑jx2j
x = rnorm(100)
y = rbinom(100,2,0.3)
estiReg <- lm(y~x+0)
summary(estiReg)
##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.18300 0.02035 0.83044 1.05747 2.13703
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.09892 0.10030 0.986 0.326
##
## Residual standard error: 0.9697 on 99 degrees of freedom
## Multiple R-squared: 0.00973, Adjusted R-squared: -0.0002731
## F-statistic: 0.9727 on 1 and 99 DF, p-value: 0.3264
estiReg2 <- lm(x~y+0)
summary(estiReg2)
##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.4497 -0.6794 -0.1926 0.4977 2.3449
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.09836 0.09973 0.986 0.326
##
## Residual standard error: 0.9669 on 99 degrees of freedom
## Multiple R-squared: 0.00973, Adjusted R-squared: -0.0002731
## F-statistic: 0.9727 on 1 and 99 DF, p-value: 0.3264
x = 1:100
y = 100:1
estiReg3 <- lm(y~x+0)
summary(estiReg3)
##
## Call:
## lm(formula = y ~ x + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## x 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08
estiReg4 <- lm(x~y+0)
summary(estiReg4)
##
## Call:
## lm(formula = x ~ y + 0)
##
## Residuals:
## Min 1Q Median 3Q Max
## -49.75 -12.44 24.87 62.18 99.49
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## y 0.5075 0.0866 5.86 6.09e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 50.37 on 99 degrees of freedom
## Multiple R-squared: 0.2575, Adjusted R-squared: 0.25
## F-statistic: 34.34 on 1 and 99 DF, p-value: 6.094e-08