Assignment 8

library(ISLR)
library(tidyverse)

Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
-- Attaching packages --------------------------------------- tidyverse 1.3.0 --
v ggplot2 3.3.2     v purrr   0.3.4
v tibble  3.0.3     v dplyr   1.0.1
v tidyr   1.1.1     v stringr 1.4.0
v readr   1.3.1     v forcats 0.5.0
-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

library(caret)

Loading required package: lattice

Attaching package: 㤼㸱caret㤼㸲

The following object is masked from 㤼㸱package:purrr㤼㸲:

    lift

Exercise 5

We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.

(a) Generate a data set with n = 500 and p = 2, such that the observations belong to two classes with a quadratic decision boundary between them. For instance, you can do this as follows:

x1=runif (500) -0.5
x2=runif (500) -0.5
y=1*(x1^2 - x2^2 > 0)

set.seed(1)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)

(b) Plot the observations, colored according to their class labels. Your plot should display \(X_1\) on the x-axis, and \(X_2\) on the y-axis.

df <- data.frame(X1 = x1, X2 = x2, Y = as.factor(y))
ggplot(df, aes(X1, X2, color = Y)) + geom_point(aes(shape = Y)) + 
  scale_color_brewer(palette = "Set1") + scale_shape_manual(values = c(16, 8)) + 
  theme_bw()

(c) Fit a logistic regression model to the data, using \(X_1\) and \(X_2\) as predictors.

lm_fit <- glm(Y ~ X1 + X2, family = binomial, data = df)
summary(lm_fit)

Call:
glm(formula = Y ~ X1 + X2, family = binomial, data = df)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.179  -1.139  -1.112   1.206   1.257  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.087260   0.089579  -0.974    0.330
X1           0.196199   0.316864   0.619    0.536
X2          -0.002854   0.305712  -0.009    0.993

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 692.18  on 499  degrees of freedom
Residual deviance: 691.79  on 497  degrees of freedom
AIC: 697.79

Number of Fisher Scoring iterations: 3

(d) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.

train_control = trainControl(method = "repeatedcv", number = 10, repeats = 3)
glm1 <- train(Y ~ ., data = df, method = "glm", trControl = train_control)
pred_glm1 <- predict(glm1, df)
df1 <- data.frame(df, Y_pred = as.factor(pred_glm1))
ggplot(df1, aes(X1, X2, color = Y_pred)) + geom_point(aes(shape = Y_pred)) + 
  scale_color_brewer(palette = "Set1") + scale_shape_manual(values = c(16, 8)) + 
  theme_bw()

m1 <- confusionMatrix(pred_glm1, df1$Y)
m1

Confusion Matrix and Statistics

          Reference
Prediction   0   1
         0 258 212
         1   3  27
                                          
               Accuracy : 0.57            
                 95% CI : (0.5253, 0.6139)
    No Information Rate : 0.522           
    P-Value [Acc > NIR] : 0.01754         
                                          
                  Kappa : 0.1054          
                                          
 Mcnemar's Test P-Value : < 2e-16         
                                          
            Sensitivity : 0.9885          
            Specificity : 0.1130          
         Pos Pred Value : 0.5489          
         Neg Pred Value : 0.9000          
             Prevalence : 0.5220          
         Detection Rate : 0.5160          
   Detection Prevalence : 0.9400          
      Balanced Accuracy : 0.5507          
                                          
       'Positive' Class : 0               
                                          

(e) Now fit a logistic regression model to the data using non-linear functions of \(X_1\) and \(X_2\) as predictors (e.g. \(X^2_1\) , \(X_1\)×\(X_2\), log(\(X_2\)), and so forth).

lm1_fit <- glm(Y ~ poly(X1, 2) + poly(X2, 2), family = binomial, data = df)
lm1_fit

Call:  glm(formula = Y ~ poly(X1, 2) + poly(X2, 2), family = binomial, 
    data = df)

Coefficients:
 (Intercept)  poly(X1, 2)1  poly(X1, 2)2  poly(X2, 2)1  poly(X2, 2)2  
      -94.48       3442.52      30110.74        162.82     -31383.76  

Degrees of Freedom: 499 Total (i.e. Null);  495 Residual
Null Deviance:      692.2 
Residual Deviance: 4.288e-06    AIC: 10

(f) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear.

glm2 <- train(Y ~ poly(X1, 2) + poly(X2, 2), data = df, method = "glm", 
              trControl = train_control)
pred_glm2 <- predict(glm2, df)
df2 <- data.frame(df, Y_pred = as.factor(pred_glm2))
ggplot(df2, aes(X1, X2, color = Y_pred)) + geom_point(aes(shape = Y_pred)) + 
  scale_color_brewer(palette = "Set1") + scale_shape_manual(values = c(16, 8)) + 
  theme_bw()

m2 <- confusionMatrix(pred_glm2, df2$Y)
m2

Confusion Matrix and Statistics

          Reference
Prediction   0   1
         0 261   0
         1   0 239
                                     
               Accuracy : 1          
                 95% CI : (0.9926, 1)
    No Information Rate : 0.522      
    P-Value [Acc > NIR] : < 2.2e-16  
                                     
                  Kappa : 1          
                                     
 Mcnemar's Test P-Value : NA         
                                     
            Sensitivity : 1.000      
            Specificity : 1.000      
         Pos Pred Value : 1.000      
         Neg Pred Value : 1.000      
             Prevalence : 0.522      
         Detection Rate : 0.522      
   Detection Prevalence : 0.522      
      Balanced Accuracy : 1.000      
                                     
       'Positive' Class : 0          
                                     

(g) Fit a support vector classifier to the data with \(X_1\) and \(X_2\) as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

svm_linear <- train(Y ~ ., data = df, method = "svmLinear", trControl = train_control)
lsvm_pred <- predict(svm_linear, df)
df3 <- data.frame(df, Y_pred = as.factor(lsvm_pred))
ggplot(df3, aes(X1, X2, color = Y_pred)) + geom_point(aes(shape = Y_pred)) + 
  scale_color_brewer(palette = "Set1") + scale_shape_manual(values = c(16, 8)) + 
  theme_bw()

m3 <- confusionMatrix(lsvm_pred, df3$Y)
m3

Confusion Matrix and Statistics

          Reference
Prediction   0   1
         0 261 239
         1   0   0
                                          
               Accuracy : 0.522           
                 95% CI : (0.4772, 0.5665)
    No Information Rate : 0.522           
    P-Value [Acc > NIR] : 0.5181          
                                          
                  Kappa : 0               
                                          
 Mcnemar's Test P-Value : <2e-16          
                                          
            Sensitivity : 1.000           
            Specificity : 0.000           
         Pos Pred Value : 0.522           
         Neg Pred Value :   NaN           
             Prevalence : 0.522           
         Detection Rate : 0.522           
   Detection Prevalence : 1.000           
      Balanced Accuracy : 0.500           
                                          
       'Positive' Class : 0               
                                          

(h) Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

svm_poly <- train(Y ~ ., data = df, method = "svmPoly", trControl = train_control)
psvm_pred <- predict(svm_poly, df)
df4 <- data.frame(df, Y_pred = as.factor(psvm_pred))
ggplot(df4, aes(X1, X2, color = Y_pred)) + geom_point(aes(shape = Y_pred)) + 
  scale_color_brewer(palette = "Set1") + scale_shape_manual(values = c(16, 8)) + 
  theme_bw()

m4 <- confusionMatrix(psvm_pred, df4$Y)
m4

Confusion Matrix and Statistics

          Reference
Prediction   0   1
         0 259  22
         1   2 217
                                         
               Accuracy : 0.952          
                 95% CI : (0.9294, 0.969)
    No Information Rate : 0.522          
    P-Value [Acc > NIR] : < 2.2e-16      
                                         
                  Kappa : 0.9035         
                                         
 Mcnemar's Test P-Value : 0.0001052      
                                         
            Sensitivity : 0.9923         
            Specificity : 0.9079         
         Pos Pred Value : 0.9217         
         Neg Pred Value : 0.9909         
             Prevalence : 0.5220         
         Detection Rate : 0.5180         
   Detection Prevalence : 0.5620         
      Balanced Accuracy : 0.9501         
                                         
       'Positive' Class : 0              
                                         

(i) Comment on your results.

The linear logistic regression model yielded an accuracy rate of 0.57, which is not much better than random guessing. The plot demonstrated that a linear model does not fit the data. The polynomial logistic regression reflected an extreme overfit, with an accuracy rate of 1. This would be expected since the y-value in the original dataset is a function of the squares of \(X_1\) and \(X_2\). The plot of the predictions matched the original scatter plot of the variables. No prediction other than 0 resulted from the linear SVM model. The confusion matrix showed an accuracy of 0.522 with a 95% confidence interval that included 0.5, so the model was no improvement over random guessing. The plot only showed a prediction of 0. For the polynomial SVM model, the accuracy rate was nearly 1 (0.952) and the prediction plot closely matched the original scatter plot.

Exercise 7

In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.

(a) Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.

summary(Auto)

      mpg          cylinders      displacement     horsepower        weight    
 Min.   : 9.00   Min.   :3.000   Min.   : 68.0   Min.   : 46.0   Min.   :1613  
 1st Qu.:17.00   1st Qu.:4.000   1st Qu.:105.0   1st Qu.: 75.0   1st Qu.:2225  
 Median :22.75   Median :4.000   Median :151.0   Median : 93.5   Median :2804  
 Mean   :23.45   Mean   :5.472   Mean   :194.4   Mean   :104.5   Mean   :2978  
 3rd Qu.:29.00   3rd Qu.:8.000   3rd Qu.:275.8   3rd Qu.:126.0   3rd Qu.:3615  
 Max.   :46.60   Max.   :8.000   Max.   :455.0   Max.   :230.0   Max.   :5140  
                                                                               
  acceleration        year           origin                      name    
 Min.   : 8.00   Min.   :70.00   Min.   :1.000   amc matador       :  5  
 1st Qu.:13.78   1st Qu.:73.00   1st Qu.:1.000   ford pinto        :  5  
 Median :15.50   Median :76.00   Median :1.000   toyota corolla    :  5  
 Mean   :15.54   Mean   :75.98   Mean   :1.577   amc gremlin       :  4  
 3rd Qu.:17.02   3rd Qu.:79.00   3rd Qu.:2.000   amc hornet        :  4  
 Max.   :24.80   Max.   :82.00   Max.   :3.000   chevrolet chevette:  4  
                                                 (Other)           :365  

new_var <- ifelse(Auto$mpg > median(Auto$mpg), 1, 0)
auto <- data.frame(hi_mpg = as.factor(new_var), Auto[, 2:9])
head(auto)

(b) Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results.

See below for the cross-validation accuracy rates at different costs. The selected model had a maximum accuracy of 0.8973313 for a cost level of C = 0.3157895.

# train_control is used from Exercise 3 since it is still in the global environment.
svm1_auto <- train(hi_mpg ~ ., data = auto, method = "svmLinear", trControl = 
                     train_control, preProcess = c("center", "scale"), 
                   tuneGrid = expand.grid(C = seq(0, 2, length = 20)))

svm1_auto

Support Vector Machines with Linear Kernel 

392 samples
  8 predictor
  2 classes: '0', '1' 

Pre-processing: centered (310), scaled (310) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 352, 352, 353, 354, 352, 353, ... 
Resampling results across tuning parameters:

  C          Accuracy   Kappa    
  0.0000000        NaN        NaN
  0.1052632  0.8914339  0.7827337
  0.2105263  0.8948740  0.7896672
  0.3157895  0.8973313  0.7945616
  0.4210526  0.8964541  0.7928072
  0.5263158  0.8964541  0.7928027
  0.6315789  0.8955758  0.7910846
  0.7368421  0.8938653  0.7876635
  0.8421053  0.8930319  0.7860149
  0.9473684  0.8921986  0.7843483
  1.0526316  0.8921986  0.7843708
  1.1578947  0.8896345  0.7792325
  1.2631579  0.8904453  0.7808710
  1.3684211  0.8887348  0.7774274
  1.4736842  0.8887348  0.7774274
  1.5789474  0.8888012  0.7775296
  1.6842105  0.8871559  0.7742334
  1.7894737  0.8837584  0.7674238
  1.8947368  0.8795052  0.7589055
  2.0000000  0.8794827  0.7588641

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was C = 0.3157895.

res1 <- as_tibble((svm1_auto$results[which.max(svm1_auto$results[,2]),]))
res1

(c) Now repeat (b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.

The radial SVM model had an accuracy rate of 0.8980083 with a cost level of 8 and holding sigma constant. The training procedure in caret found the optimal cost by maximizing accuracy. For the polynomial SVM, caret found the optimal degree of 3 and maximized the accuracy rate at 0.8937978 with a cost level of 0.5. All 3 models had high accuracy rates and were very close. The radial SVM model barely had the highest accuracy rate.

svm2_auto <- train(hi_mpg ~ ., data = auto, method = "svmRadial", trControl = 
                     train_control, preProcess = c("center", "scale"), 
                   tuneLength = 10)

svm2_auto

Support Vector Machines with Radial Basis Function Kernel 

392 samples
  8 predictor
  2 classes: '0', '1' 

Pre-processing: centered (310), scaled (310) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 353, 352, 353, 352, 353, 354, ... 
Resampling results across tuning parameters:

  C       Accuracy   Kappa    
    0.25  0.8825540  0.7647803
    0.50  0.8979431  0.7956449
    1.00  0.8971536  0.7940483
    2.00  0.8962989  0.7923715
    4.00  0.8971536  0.7940663
    8.00  0.8980083  0.7957701
   16.00  0.8980083  0.7957473
   32.00  0.8971098  0.7939423
   64.00  0.8962764  0.7922574
  128.00  0.8910999  0.7819152

Tuning parameter 'sigma' was held constant at a value of 0.002133085
Accuracy was used to select the optimal model using the largest value.
The final values used for the model were sigma = 0.002133085 and C = 8.

res2 <- as_tibble((svm2_auto$results[which.max(svm2_auto$results[,3]),]))
res2

svm3_auto <- train(hi_mpg ~ ., data = auto, method = "svmPoly", trControl = 
                     train_control, preProcess = c("center", "scale"))

svm3_auto

Support Vector Machines with Polynomial Kernel 

392 samples
  8 predictor
  2 classes: '0', '1' 

Pre-processing: centered (310), scaled (310) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 352, 353, 352, 352, 353, 353, ... 
Resampling results across tuning parameters:

  degree  scale  C     Accuracy   Kappa    
  1       0.001  0.25  0.7773021  0.5575208
  1       0.001  0.50  0.8403408  0.6807710
  1       0.001  1.00  0.8837101  0.7674027
  1       0.010  0.25  0.8879645  0.7758087
  1       0.010  0.50  0.8879431  0.7757899
  1       0.010  1.00  0.8879645  0.7758179
  1       0.100  0.25  0.8888192  0.7775489
  1       0.100  0.50  0.8896525  0.7792155
  1       0.100  1.00  0.8913192  0.7825535
  2       0.001  0.25  0.8394861  0.6790401
  2       0.001  0.50  0.8828554  0.7656854
  2       0.001  1.00  0.8879431  0.7757716
  2       0.010  0.25  0.8887764  0.7774656
  2       0.010  0.50  0.8896311  0.7791875
  2       0.010  1.00  0.8887978  0.7775208
  2       0.100  0.25  0.8836910  0.7672311
  2       0.100  0.50  0.8732310  0.7463442
  2       0.100  1.00  0.8605364  0.7209425
  3       0.001  0.25  0.8778340  0.7556973
  3       0.001  0.50  0.8836898  0.7673300
  3       0.001  1.00  0.8879431  0.7757899
  3       0.010  0.25  0.8904858  0.7809048
  3       0.010  0.50  0.8937978  0.7875211
  3       0.010  1.00  0.8928767  0.7856519
  3       0.100  0.25  0.6437877  0.2862411
  3       0.100  0.50  0.6429543  0.2845744
  3       0.100  1.00  0.6396210  0.2779078

Accuracy was used to select the optimal model using the largest value.
The final values used for the model were degree = 3, scale = 0.01 and C = 0.5.

res3 <- as_tibble((svm3_auto$results[which.max(svm3_auto$results[,4]),]))
res3

(d) Make some plots to back up your assertions in (b) and (c).

Hint: In the lab, we used the plot() function for svm objects only in cases with p = 2. When p > 2, you can use the plot() function to create plots displaying pairs of variables at a time. Essentially, instead of typing

plot(svmfit , dat)

where svmfit contains your fitted model and dat is a data frame containing your data, you can type

plot(svmfit , dat , x1∼x4)

in order to plot just the first and fourth variables. However, you must replace x1 and x4 with the correct variable names. To find out more, type

?plot.svm

plot(svm1_auto, main = "Linear Support Vector Classifier")

plot(svm2_auto, main = "Radial Support Vector Classifier")

plot(svm3_auto, main  = "Polynomial Support Vector Classifier")

df_svm <- tibble(Model = c("SVM Linear", "SVM Radial", "SVM Poly"), 
                 Accuracy = c(res1$Accuracy, res2$Accuracy, res3$Accuracy))
ggplot(df_svm, aes(Model, Accuracy, fill = Model)) + geom_col() + 
  scale_y_continuous(limits = c(0, 1)) + 
  scale_x_discrete(limits = c("SVM Poly", "SVM Linear", "SVM Radial")) + 
  scale_fill_brewer(palette = "Set1", limits = c("SVM Poly", "SVM Linear", 
                                                 "SVM Radial")) + 
  geom_text(aes(label = Accuracy, vjust = -0.5)) + 
  theme_bw()

Exercise 8

This problem involves the OJ data set which is part of the ISLR package.

(a) Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

summary(OJ)

 Purchase WeekofPurchase     StoreID        PriceCH         PriceMM     
 CH:653   Min.   :227.0   Min.   :1.00   Min.   :1.690   Min.   :1.690  
 MM:417   1st Qu.:240.0   1st Qu.:2.00   1st Qu.:1.790   1st Qu.:1.990  
          Median :257.0   Median :3.00   Median :1.860   Median :2.090  
          Mean   :254.4   Mean   :3.96   Mean   :1.867   Mean   :2.085  
          3rd Qu.:268.0   3rd Qu.:7.00   3rd Qu.:1.990   3rd Qu.:2.180  
          Max.   :278.0   Max.   :7.00   Max.   :2.090   Max.   :2.290  
     DiscCH            DiscMM         SpecialCH        SpecialMM     
 Min.   :0.00000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.00000   1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:0.0000  
 Median :0.00000   Median :0.0000   Median :0.0000   Median :0.0000  
 Mean   :0.05186   Mean   :0.1234   Mean   :0.1477   Mean   :0.1617  
 3rd Qu.:0.00000   3rd Qu.:0.2300   3rd Qu.:0.0000   3rd Qu.:0.0000  
 Max.   :0.50000   Max.   :0.8000   Max.   :1.0000   Max.   :1.0000  
    LoyalCH          SalePriceMM     SalePriceCH      PriceDiff       Store7   
 Min.   :0.000011   Min.   :1.190   Min.   :1.390   Min.   :-0.6700   No :714  
 1st Qu.:0.325257   1st Qu.:1.690   1st Qu.:1.750   1st Qu.: 0.0000   Yes:356  
 Median :0.600000   Median :2.090   Median :1.860   Median : 0.2300            
 Mean   :0.565782   Mean   :1.962   Mean   :1.816   Mean   : 0.1465            
 3rd Qu.:0.850873   3rd Qu.:2.130   3rd Qu.:1.890   3rd Qu.: 0.3200            
 Max.   :0.999947   Max.   :2.290   Max.   :2.090   Max.   : 0.6400            
   PctDiscMM        PctDiscCH       ListPriceDiff       STORE      
 Min.   :0.0000   Min.   :0.00000   Min.   :0.000   Min.   :0.000  
 1st Qu.:0.0000   1st Qu.:0.00000   1st Qu.:0.140   1st Qu.:0.000  
 Median :0.0000   Median :0.00000   Median :0.240   Median :2.000  
 Mean   :0.0593   Mean   :0.02731   Mean   :0.218   Mean   :1.631  
 3rd Qu.:0.1127   3rd Qu.:0.00000   3rd Qu.:0.300   3rd Qu.:3.000  
 Max.   :0.4020   Max.   :0.25269   Max.   :0.440   Max.   :4.000  

set.seed(1)
train <- sample(dim(OJ)[1], 800)
oj_train <- OJ[train, ]
oj_test <- OJ[-train, ]
dim(oj_test)

[1] 270  18

(b) Fit a support vector classifier to the training data using cost=0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.

With cost level set to 0.01, the training model had an accuracy rate of 0.8207474 and a kappa value of 0.6220058.

svm1_oj <- train(Purchase ~ ., data = oj_train, method = "svmLinear", trControl = 
                   train_control, preProcess = c("center", "scale"), 
               tuneGrid = expand.grid(C = 0.01))
svm1_oj

Support Vector Machines with Linear Kernel 

800 samples
 17 predictor
  2 classes: 'CH', 'MM' 

Pre-processing: centered (17), scaled (17) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 721, 720, 720, 720, 721, 719, ... 
Resampling results:

  Accuracy   Kappa    
  0.8207474  0.6220058

Tuning parameter 'C' was held constant at a value of 0.01

purch1_pred <- predict(svm1_oj, oj_test)
oj1_m <- confusionMatrix(purch1_pred, oj_test$Purchase)
oj1_m

Confusion Matrix and Statistics

          Reference
Prediction  CH  MM
        CH 153  35
        MM  15  67
                                          
               Accuracy : 0.8148          
                 95% CI : (0.7633, 0.8593)
    No Information Rate : 0.6222          
    P-Value [Acc > NIR] : 5.136e-12       
                                          
                  Kappa : 0.5903          
                                          
 Mcnemar's Test P-Value : 0.00721         
                                          
            Sensitivity : 0.9107          
            Specificity : 0.6569          
         Pos Pred Value : 0.8138          
         Neg Pred Value : 0.8171          
             Prevalence : 0.6222          
         Detection Rate : 0.5667          
   Detection Prevalence : 0.6963          
      Balanced Accuracy : 0.7838          
                                          
       'Positive' Class : CH              
                                          

oj1_m$overall[1]

 Accuracy 
0.8148148 

(c) What are the training and test error rates?

Training Error Rate: 0.1792526
Test Error Rate: 0.1851852

(d) Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.

svm2_oj <- train(Purchase ~ ., data = oj_train, method = "svmLinear", trControl = 
                   train_control, preProcess = c("center", "scale"), 
               tuneGrid = expand.grid(C = seq(0.01, 10, length = 20)))

svm2_oj

Support Vector Machines with Linear Kernel 

800 samples
 17 predictor
  2 classes: 'CH', 'MM' 

Pre-processing: centered (17), scaled (17) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 720, 719, 719, 719, 720, 721, ... 
Resampling results across tuning parameters:

  C           Accuracy   Kappa    
   0.0100000  0.8250671  0.6307520
   0.5357895  0.8287912  0.6392281
   1.0615789  0.8271297  0.6356709
   1.5873684  0.8267128  0.6348730
   2.1131579  0.8283847  0.6385176
   2.6389474  0.8279889  0.6379367
   3.1647368  0.8271554  0.6361990
   3.6905263  0.8279889  0.6379935
   4.2163158  0.8284160  0.6387943
   4.7421053  0.8279993  0.6378381
   5.2678947  0.8284108  0.6387841
   5.7936842  0.8288171  0.6393827
   6.3194737  0.8284004  0.6384070
   6.8452632  0.8284004  0.6384070
   7.3710526  0.8283951  0.6384575
   7.8968421  0.8275512  0.6365772
   8.4226316  0.8275512  0.6365772
   8.9484211  0.8271293  0.6356263
   9.4742105  0.8267074  0.6346643
  10.0000000  0.8267074  0.6346643

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was C = 5.793684.

svm2_oj$results$Accuracy[which.max(svm2_oj$results$Accuracy)]

[1] 0.8288171

(e) Compute the training and test error rates using this new value for cost.

Training Error Rate: 0.1711829
Test Error Rate: 0.1518519

purch2_pred <- predict(svm2_oj, oj_test)
oj2_m <- confusionMatrix(purch2_pred, oj_test$Purchase)
oj2_m

Confusion Matrix and Statistics

          Reference
Prediction  CH  MM
        CH 156  29
        MM  12  73
                                          
               Accuracy : 0.8481          
                 95% CI : (0.7997, 0.8888)
    No Information Rate : 0.6222          
    P-Value [Acc > NIR] : 2.513e-16       
                                          
                  Kappa : 0.6661          
                                          
 Mcnemar's Test P-Value : 0.01246         
                                          
            Sensitivity : 0.9286          
            Specificity : 0.7157          
         Pos Pred Value : 0.8432          
         Neg Pred Value : 0.8588          
             Prevalence : 0.6222          
         Detection Rate : 0.5778          
   Detection Prevalence : 0.6852          
      Balanced Accuracy : 0.8221          
                                          
       'Positive' Class : CH              
                                          

oj2_m$overall[1]

 Accuracy 
0.8481481 

(f) Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.

Training Error Rate: 0.1733135
Test Error Rate: 0.1814815

svm3_oj <- train(Purchase ~ ., data = oj_train, method = "svmRadial", trControl = 
                   train_control, preProcess = c("center", "scale"))

svm3_oj

Support Vector Machines with Radial Basis Function Kernel 

800 samples
 17 predictor
  2 classes: 'CH', 'MM' 

Pre-processing: centered (17), scaled (17) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 720, 721, 721, 719, 719, 720, ... 
Resampling results across tuning parameters:

  C     Accuracy   Kappa    
  0.25  0.8246283  0.6240238
  0.50  0.8262749  0.6293929
  1.00  0.8266865  0.6299419

Tuning parameter 'sigma' was held constant at a value of 0.05870232
Accuracy was used to select the optimal model using the largest value.
The final values used for the model were sigma = 0.05870232 and C = 1.

svm3_oj$results$Accuracy[which.max(svm3_oj$results$Accuracy)]

[1] 0.8266865

purch3_pred <- predict(svm3_oj, oj_test)
oj3_m <- confusionMatrix(purch3_pred, oj_test$Purchase)
oj3_m

Confusion Matrix and Statistics

          Reference
Prediction  CH  MM
        CH 151  32
        MM  17  70
                                          
               Accuracy : 0.8185          
                 95% CI : (0.7673, 0.8626)
    No Information Rate : 0.6222          
    P-Value [Acc > NIR] : 1.887e-12       
                                          
                  Kappa : 0.6025          
                                          
 Mcnemar's Test P-Value : 0.0455          
                                          
            Sensitivity : 0.8988          
            Specificity : 0.6863          
         Pos Pred Value : 0.8251          
         Neg Pred Value : 0.8046          
             Prevalence : 0.6222          
         Detection Rate : 0.5593          
   Detection Prevalence : 0.6778          
      Balanced Accuracy : 0.7925          
                                          
       'Positive' Class : CH              
                                          

oj3_m$overall[1]

 Accuracy 
0.8185185 

(g) Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree=2.

The training procedure in caret will be allowed to select the optimal degree.
Training Error Rate: 0.1683105
Test Error Rate: 0.1777778

svm4_oj <- train(Purchase ~ ., data = oj_train, method = "svmPoly", trControl = 
                   train_control, preProcess = c("center", "scale"))

svm4_oj

Support Vector Machines with Polynomial Kernel 

800 samples
 17 predictor
  2 classes: 'CH', 'MM' 

Pre-processing: centered (17), scaled (17) 
Resampling: Cross-Validated (10 fold, repeated 3 times) 
Summary of sample sizes: 720, 720, 721, 719, 720, 720, ... 
Resampling results across tuning parameters:

  degree  scale  C     Accuracy   Kappa    
  1       0.001  0.25  0.6062589  0.0000000
  1       0.001  0.50  0.6062589  0.0000000
  1       0.001  1.00  0.6979597  0.2782855
  1       0.010  0.25  0.8245951  0.6252818
  1       0.010  0.50  0.8212303  0.6234578
  1       0.010  1.00  0.8191885  0.6188475
  1       0.100  0.25  0.8246058  0.6303219
  1       0.100  0.50  0.8266944  0.6350564
  1       0.100  1.00  0.8250225  0.6319884
  2       0.001  0.25  0.6062589  0.0000000
  2       0.001  0.50  0.6975430  0.2774730
  2       0.001  1.00  0.8150475  0.6000696
  2       0.010  0.25  0.8208398  0.6214581
  2       0.010  0.50  0.8233660  0.6267510
  2       0.010  1.00  0.8258557  0.6325540
  2       0.100  0.25  0.8291529  0.6375090
  2       0.100  0.50  0.8271007  0.6340759
  2       0.100  1.00  0.8246004  0.6278044
  3       0.001  0.25  0.6408399  0.1094171
  3       0.001  0.50  0.8050833  0.5707678
  3       0.001  1.00  0.8258450  0.6306579
  3       0.010  0.25  0.8225221  0.6249101
  3       0.010  0.50  0.8316895  0.6442259
  3       0.010  1.00  0.8316842  0.6446164
  3       0.100  0.25  0.8175057  0.6091842
  3       0.100  0.50  0.8129064  0.5992662
  3       0.100  1.00  0.8083226  0.5907770

Accuracy was used to select the optimal model using the largest value.
The final values used for the model were degree = 3, scale = 0.01 and C = 0.5.

svm4_oj$results$Accuracy[which.max(svm4_oj$results$Accuracy)]

[1] 0.8316895

purch4_pred <- predict(svm4_oj, oj_test)
oj4_m <- confusionMatrix(purch4_pred, oj_test$Purchase)
oj4_m

Confusion Matrix and Statistics

          Reference
Prediction  CH  MM
        CH 153  33
        MM  15  69
                                          
               Accuracy : 0.8222          
                 95% CI : (0.7713, 0.8659)
    No Information Rate : 0.6222          
    P-Value [Acc > NIR] : 6.769e-13       
                                          
                  Kappa : 0.6083          
                                          
 Mcnemar's Test P-Value : 0.01414         
                                          
            Sensitivity : 0.9107          
            Specificity : 0.6765          
         Pos Pred Value : 0.8226          
         Neg Pred Value : 0.8214          
             Prevalence : 0.6222          
         Detection Rate : 0.5667          
   Detection Prevalence : 0.6889          
      Balanced Accuracy : 0.7936          
                                          
       'Positive' Class : CH              
                                          

oj4_m$overall[1]

 Accuracy 
0.8222222 

(h) Overall, which approach seems to give the best results on this data?

Based on the results below, the linear SVM model with automatic selection by highest accuracy using multiple cost values performed slightly better than the rest of the models.

Model	Test Accuracy Rate
Linear SVM with C = 0.01	0.8148148
Linear SVM Autoselected C	0.8481481
Radial SVM	0.8185185
Polynomial SVM	0.8222222