Mtcars Dataset

Load the data:

data(mtcars)
attach(mtcars)

And load the package for data visualization:

library(ggplot2)
## 
## Attaching package: 'ggplot2'
## The following object is masked from 'mtcars':
## 
##     mpg

Examine data and variable types

?mtcars
## starting httpd help server ... done
head(mtcars)
##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
str(mtcars)
## 'data.frame':    32 obs. of  11 variables:
##  $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
##  $ cyl : num  6 6 4 6 8 6 8 4 4 6 ...
##  $ disp: num  160 160 108 258 360 ...
##  $ hp  : num  110 110 93 110 175 105 245 62 95 123 ...
##  $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
##  $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
##  $ qsec: num  16.5 17 18.6 19.4 17 ...
##  $ vs  : num  0 0 1 1 0 1 0 1 1 1 ...
##  $ am  : num  1 1 1 0 0 0 0 0 0 0 ...
##  $ gear: num  4 4 4 3 3 3 3 4 4 4 ...
##  $ carb: num  4 4 1 1 2 1 4 2 2 4 ...

Questions

  1. What variables are most strongly correlated with mpg?
  2. How does the value of k in KNN affect the model’s performance?
  3. What assumptions are being made when we use linear regression? Are they met in this dataset? Just describe what you observe from the diagnostic plots.
  4. Try adding interaction terms to your linear regression model. At least try to find out one interaction term that has a statistically significant coefficient. Report the interaction term and check how do these interaction terms influence the model’s performance in terms of R^2 and how do you interpret your new model?
  5. Is there any outliers in the dataset? If yes, apply truncation or winsorization techniques to handle outliers. Compare the performance of the models before and after applying these techniques. What differences do you observe?
  6. How could feature scaling affect the KNN model?
  7. What insights can you derive from comparing the linear regression and KNN models?

Answering the Questions

  1. What variables are most strongly correlated with mpg?
data(mtcars)
attach(mtcars)
## The following object is masked from package:ggplot2:
## 
##     mpg
## The following objects are masked from mtcars (pos = 4):
## 
##     am, carb, cyl, disp, drat, gear, hp, mpg, qsec, vs, wt
Mtcars_lm <- lm(mpg ~ ., data = mtcars)
summary(Mtcars_lm)
## 
## Call:
## lm(formula = mpg ~ ., data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.4506 -1.6044 -0.1196  1.2193  4.6271 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept) 12.30337   18.71788   0.657   0.5181  
## cyl         -0.11144    1.04502  -0.107   0.9161  
## disp         0.01334    0.01786   0.747   0.4635  
## hp          -0.02148    0.02177  -0.987   0.3350  
## drat         0.78711    1.63537   0.481   0.6353  
## wt          -3.71530    1.89441  -1.961   0.0633 .
## qsec         0.82104    0.73084   1.123   0.2739  
## vs           0.31776    2.10451   0.151   0.8814  
## am           2.52023    2.05665   1.225   0.2340  
## gear         0.65541    1.49326   0.439   0.6652  
## carb        -0.19942    0.82875  -0.241   0.8122  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.65 on 21 degrees of freedom
## Multiple R-squared:  0.869,  Adjusted R-squared:  0.8066 
## F-statistic: 13.93 on 10 and 21 DF,  p-value: 3.793e-07

Based on the summary, we can see taht wt is the most correlated with mpg

  1. How does the value of k in KNN affect the model’s performance?

Increasing the k value can increase the accuracy of the model’s performance.

  1. What assumptions are being made when we use linear regression? Are they met in this dataset? Just describe what you observe from the diagnostic plots.

The linear regression assumptions are: sample is representative of the population, linear relationship between independent variable and dependent variable, and variables are normally distributed.

ggplot(mtcars, aes(x=mpg)) +
  geom_histogram(binwidth=5, fill="blue", alpha=0.7) +
  ggtitle("Distribution of MPG") +
  xlab("mpg") +
  ylab("Frequency")

Mtcars_asu <- lm(mpg ~ wt, data = mtcars)
summary(Mtcars_asu)
## 
## Call:
## lm(formula = mpg ~ wt, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528, Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

Based on the charts above, we can say that the variables fit the assumptions.

  1. Try adding interaction terms to your linear regression model. At least try to find out one interaction term that has a statistically significant coefficient. Report the interaction term and check how do these interaction terms influence the model’s performance in terms of R^2 and how do you interpret your new model?
mtcars_inter <- lm(mpg ~ wt + mpg + wt*mpg, data = mtcars)
## Warning in model.matrix.default(mt, mf, contrasts): the response appeared on
## the right-hand side and was dropped
## Warning in model.matrix.default(mt, mf, contrasts): problem with term 2 in
## model.matrix: no columns are assigned
summary(mtcars_inter)
## 
## Call:
## lm(formula = mpg ~ wt + mpg + wt * mpg, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.0798 -1.3490 -0.9543  0.6278  4.7721 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 24.05996    2.76847   8.691 1.44e-09 ***
## wt          -6.06229    0.42039 -14.421 9.24e-15 ***
## mpg:wt       0.26030    0.04767   5.460 7.06e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.175 on 29 degrees of freedom
## Multiple R-squared:  0.8781, Adjusted R-squared:  0.8697 
## F-statistic: 104.5 on 2 and 29 DF,  p-value: 5.565e-14
  1. Is there any outliers in the dataset? If yes, apply truncation or winsorization techniques to handle outliers. Compare the performance of the models before and after applying these techniques. What differences do you observe?
boxplot(mtcars, las=2, cex.axis=0.6)

Looks like we have outliers in hp, wt, qsec, and carb

# hp
lower_bound_tax <- quantile(mtcars$hp, 0.05, na.rm = TRUE)
upper_bound_tax <- quantile(mtcars$hp, 0.95, na.rm = TRUE)

mtcars$hp[mtcars$hp < lower_bound_tax] <- lower_bound_tax
mtcars$hp[mtcars$hp > upper_bound_tax] <- upper_bound_tax

# wt

lower_bound_tax <- quantile(mtcars$wt, 0.05, na.rm = TRUE)
upper_bound_tax <- quantile(mtcars$wt, 0.95, na.rm = TRUE)

mtcars$wt[mtcars$wt < lower_bound_tax] <- lower_bound_tax
mtcars$wt[mtcars$wt > upper_bound_tax] <- upper_bound_tax

# qsec

lower_bound_tax <- quantile(mtcars$qsec, 0.05, na.rm = TRUE)
upper_bound_tax <- quantile(mtcars$qsec, 0.95, na.rm = TRUE)

mtcars$qsec[mtcars$qsec < lower_bound_tax] <- lower_bound_tax
mtcars$qsec[mtcars$qsec > upper_bound_tax] <- upper_bound_tax

# carb

lower_bound_tax <- quantile(mtcars$carb, 0.05, na.rm = TRUE)
upper_bound_tax <- quantile(mtcars$carb, 0.95, na.rm = TRUE)

mtcars$carb[mtcars$carb < lower_bound_tax] <- lower_bound_tax
mtcars$carb[mtcars$carb > upper_bound_tax] <- upper_bound_tax

#Display final results

boxplot(mtcars, las=2, cex.axis=0.6)

No more outliers

  1. How could feature scaling affect the KNN model?

Featuring scaling would help the larger range variables such as disp and hp from dominating the model.

  1. What insights can you derive from comparing the linear regression and KNN models?
library(kknn)
k <- 5  
knn_fit_std <- kknn(mpg ~ ., mtcars, mtcars, k=k)

Mtcars_lm$knn_std_predicted <- knn_fit_std$fitted.values
head(Mtcars_lm)
## $coefficients
## (Intercept)         cyl        disp          hp        drat          wt 
## 12.30337416 -0.11144048  0.01333524 -0.02148212  0.78711097 -3.71530393 
##        qsec          vs          am        gear        carb 
##  0.82104075  0.31776281  2.52022689  0.65541302 -0.19941925 
## 
## $residuals
##           Mazda RX4       Mazda RX4 Wag          Datsun 710      Hornet 4 Drive 
##        -1.599505761        -1.111886079        -3.450644085         0.162595453 
##   Hornet Sportabout             Valiant          Duster 360           Merc 240D 
##         1.006565971        -2.283039036        -0.086256253         1.903988115 
##            Merc 230            Merc 280           Merc 280C          Merc 450SE 
##        -1.619089898         0.500970058        -1.391654392         2.227837890 
##          Merc 450SL         Merc 450SLC  Cadillac Fleetwood Lincoln Continental 
##         1.700426404        -0.542224699        -1.634013415        -0.536437711 
##   Chrysler Imperial            Fiat 128         Honda Civic      Toyota Corolla 
##         4.206370638         4.627094192         0.503261089         4.387630904 
##       Toyota Corona    Dodge Challenger         AMC Javelin          Camaro Z28 
##        -2.143103442        -1.443053221        -2.532181498        -0.006021976 
##    Pontiac Firebird           Fiat X1-9       Porsche 914-2        Lotus Europa 
##         2.508321011        -0.993468693        -0.152953961         2.763727417 
##      Ford Pantera L        Ferrari Dino       Maserati Bora          Volvo 142E 
##        -3.070040803         0.006171846         1.058881618        -2.968267683 
## 
## $effects
##   (Intercept)           cyl          disp            hp          drat 
## -1.136497e+02 -2.859568e+01  6.131391e+00 -3.061197e+00 -4.058009e+00 
##            wt          qsec            vs            am          gear 
## -8.802034e+00 -1.987287e+00  3.602342e-01  3.804502e+00 -9.857538e-01 
##          carb                                                         
## -6.377059e-01  2.679169e+00  2.072950e+00 -6.318915e-03 -1.254563e+00 
##                                                                       
##  2.453586e-02  4.730750e+00  5.654391e+00 -9.629859e-01  5.285476e+00 
##                                                                       
## -1.333950e+00 -9.287503e-01 -2.204646e+00 -7.723442e-01  3.228137e+00 
##                                                                       
## -2.849364e-01  2.549272e+00  4.116721e+00 -1.854365e+00  4.859340e-02 
##                             
##  9.356676e-01 -2.205753e+00 
## 
## $rank
## [1] 11
## 
## $fitted.values
##           Mazda RX4       Mazda RX4 Wag          Datsun 710      Hornet 4 Drive 
##            22.59951            22.11189            26.25064            21.23740 
##   Hornet Sportabout             Valiant          Duster 360           Merc 240D 
##            17.69343            20.38304            14.38626            22.49601 
##            Merc 230            Merc 280           Merc 280C          Merc 450SE 
##            24.41909            18.69903            19.19165            14.17216 
##          Merc 450SL         Merc 450SLC  Cadillac Fleetwood Lincoln Continental 
##            15.59957            15.74222            12.03401            10.93644 
##   Chrysler Imperial            Fiat 128         Honda Civic      Toyota Corolla 
##            10.49363            27.77291            29.89674            29.51237 
##       Toyota Corona    Dodge Challenger         AMC Javelin          Camaro Z28 
##            23.64310            16.94305            17.73218            13.30602 
##    Pontiac Firebird           Fiat X1-9       Porsche 914-2        Lotus Europa 
##            16.69168            28.29347            26.15295            27.63627 
##      Ford Pantera L        Ferrari Dino       Maserati Bora          Volvo 142E 
##            18.87004            19.69383            13.94112            24.36827 
## 
## $assign
##  [1]  0  1  2  3  4  5  6  7  8  9 10