Support Vector Machines

Introduction

In this lab I will be experimenting with Support Vector Machine models. Utilizing the SVM() function to create both linear, radial, and polynomial support vector models. I will also use the caret package to compare the SVM models to other types of models and perform resampling splits. Both the OJ and the Auto dataset that I will be using are from the ISLR2 package and graphics will be performed using base, ggplot, and plotly.

library(e1071)
library(caret)
library(tidyverse)
library(plotly)
library(ISLR2)

Q5 - Fitting SVMs.

We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.

(a) Creating the Data

Generate a data set with n = 500 and p =2, such that the observations belong to two classes with a quadratic decision boundary between them. For instance, you can do this as follows:

x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)

data <- data.frame(
  x1 = x1,
  x2 = x2, 
  y = as.factor(y)
  )

(b) EDA

Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the yaxis.

d <- data |> ggplot(aes(
    x = x1,
    y = x2,
    color = y)) +
    geom_point(
      size =1
      ) + 
  labs(
    title = "x1 and x2 by Class"
    ) +
  theme_grey() + 
  scale_color_manual(values = c("#264653", "#E76F51"))


ggplotly(d)

(c) Linear Model

Fit a logistic regression model to the data, using X1 and X2 as predictors.

fit <- train(
  y ~ ., 
  data = data,
  method = "glm",
  family = binomial
)

(d) Linear Model Predictions

Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.

# Predicted class labels
preds <- predict(fit, newdata = data)

data$preds <- preds

# Coefficients from the underlying glm model
b <- coef(fit$finalModel)

d <- ggplot(data, aes(x = x1, y = x2, color = preds)) +
  geom_point(size = 1) +
  geom_abline(
    intercept = -b[1] / b[3],
    slope = -b[2] / b[3],
    color = "black",
    linewidth = 1
  ) +
  labs(
    title = "x1 and x2 by Predicted Class",
    color = "Predicted Class"
  ) +
  theme_grey() + 
  scale_color_manual(values = c("#264653", "#E76F51"))


ggplotly(d)

(e) Non-linear Model

Now fit a logistic regression model to the data using non-linear functions of X1 and X2 as predictors (e.g. X2 1 , X1×X2, log(X2), and so forth).

fit1 <- train(
  y ~ x1 + x2 + I(x1^2) + I(x2^2) + x1:x2,
  data = data,
  method = "glm",
  family = binomial
)

(f) Non-linear Predictions

Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear.

data$preds <- predict(fit1, newdata = data)

# Create a grid of x1 and x2 values
grid <- expand.grid(
  x1 = seq(min(data$x1), max(data$x1), length.out = 200),
  x2 = seq(min(data$x2), max(data$x2), length.out = 200)
)

# Predict probability of class "1"
grid$prob <- predict(fit1, newdata = grid, type = "prob")[, "1"]

# Plot
d <- ggplot(data, aes(x = x1, y = x2)) +
  geom_point(aes(color = preds), size = 1) +
  geom_contour(
    data = grid,
    aes(x = x1, y = x2, z = prob),
    breaks = 0.5,
    color = "black",
    linewidth = 1
  ) +
  labs(
    title = "Nonlinear Logistic Regression",
    color = "Predicted Class"
  ) +
  theme_grey() + 
  scale_color_manual(values = c("#264653", "#E76F51"))


ggplotly(d)

(g) Linear Support Vector

Fit a support vector classifier to the data with X1 and X2 as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

svm.fit <- svm(y ~ .,
               data = data, 
               kernel = "linear", 
               cost = 1e5)
summary(svm.fit)

## 
## Call:
## svm(formula = y ~ ., data = data, kernel = "linear", cost = 1e+05)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  1e+05 
## 
## Number of Support Vectors:  9
## 
##  ( 5 4 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  0 1

# Predicted class labels
data$preds <- predict(svm.fit, newdata = data)

# Plot
d <- ggplot(data, aes(x = x1, y = x2, color = preds)) +
  geom_point(size = 1) +
  labs(
    title = "Support Vector Classifier",
    color = "Predicted Class"
  ) +
  theme_grey() + 
  scale_color_manual(values = c("#264653", "#E76F51"))


ggplotly(d)

(h) Non-linear Supper Vector

Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

svm.fit1 <- svm(y ~ .,
               data = data, 
               kernel = "radial", 
               gamma = 1,
               cost = 1e5)
summary(svm.fit1)

## 
## Call:
## svm(formula = y ~ ., data = data, kernel = "radial", gamma = 1, cost = 1e+05)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  1e+05 
## 
## Number of Support Vectors:  33
## 
##  ( 15 18 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  0 1

# Predicted class labels
data$preds <- predict(svm.fit1, newdata = data)

# Plot
d <- ggplot(data, aes(x = x1, y = x2, color = preds)) +
  geom_point(size = 1) +
  labs(
    title = "Support Vector Classifier",
    color = "Predicted Class"
  ) +
  theme_grey() + 
  scale_color_manual(values = c("#264653", "#E76F51"))


ggplotly(d)

(i) Comments

Comment on your results. The linear model produced a linear decision boundary between two classes dividing both into class 0 and class 1. The non-linear model produced a nonlinear V-shaped boundary. Two of them for each class. Similar to the non-linear regression model - both of the SVM models produced v shaped decision boundaries. I was supprised to see the linear SVM model with a non-linear boundary on the predictions.

---

Q7 - Predicting MPG

In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.

(a) Variable Creation

Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.

#Look at the data the data
head(Auto)

# Create the binary variable
Auto$mpg_bin <- as.factor(ifelse(Auto$mpg > 23.45, 1, 0))

(b) Support Vector Classifier Models

Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results.

# remove mpg as instructed
Auto1 <- as.data.frame(subset(Auto, select = -mpg))

sv.class <- svm(mpg_bin ~ .,
               data = Auto1, 
               kernel = "linear", 
               cost = 10,
               cross = 5)
summary(sv.class)

## 
## Call:
## svm(formula = mpg_bin ~ ., data = Auto1, kernel = "linear", cost = 10, 
##     cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  10 
## 
## Number of Support Vectors:  82
## 
##  ( 41 41 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  0 1
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 88.52041 
## Single Accuracies:
##  84.61538 87.17949 89.87342 89.74359 91.13924

# Predicted class labels
Auto1$preds <- predict(sv.class, newdata = Auto1)

# compute the cv error
cv_error <- 1 - (sv.class$tot.accuracy / 100)
print(cv_error)

## [1] 0.1147959

Comment on your results. So far I have fit the model with MPG and am using MPG1 as a binary response variable in my SVC model. My CV error came out to around 10%. Meaning that around 90% of the time this model is predicting MPG correctly.

(c) Support Vector Radial and Polynomial Models

Now repeat(b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.

#SV radial
sv.radial <- svm(mpg_bin ~ .,
               data = Auto1, 
               kernel = "radial", 
               cost = 1,
               gamma = 1,
               degree = 2,
               cross = 5)
summary(sv.radial)

## 
## Call:
## svm(formula = mpg_bin ~ ., data = Auto1, kernel = "radial", cost = 1, 
##     gamma = 1, degree = 2, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  1 
## 
## Number of Support Vectors:  375
## 
##  ( 197 178 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  0 1
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 97.19388 
## Single Accuracies:
##  96.15385 97.4359 97.46835 98.71795 96.20253

# Predicted class labels
Auto1$preds <- predict(sv.radial, newdata = Auto1)

# compute the cv error
cv_error <- 1 - (sv.radial$tot.accuracy / 100)
print(cv_error)

## [1] 0.02806122

#SV Polynomial
sv.polynomial <- svm(mpg_bin ~ .,
               data = Auto1, 
               kernel = "polynomial", 
               cost = 2,
               gamma = 3,
               degree = 5,
               cross = 5)
summary(sv.polynomial)

## 
## Call:
## svm(formula = mpg_bin ~ ., data = Auto1, kernel = "polynomial", cost = 2, 
##     gamma = 3, degree = 5, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  polynomial 
##        cost:  2 
##      degree:  5 
##      coef.0:  0 
## 
## Number of Support Vectors:  104
## 
##  ( 67 37 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  0 1
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 98.21429 
## Single Accuracies:
##  100 98.71795 98.73418 94.87179 98.73418

# Predicted class labels
Auto1$preds <- predict(sv.polynomial, newdata = Auto1)

# compute the cv error
cv_error <- 1 - (sv.polynomial$tot.accuracy / 100)
print(cv_error)

## [1] 0.01785714

Comment on your results. For both the radial and polynomial model the CV error is around 2%. Meaning that around 98% of the time the predictions are accurately predicted. See the comparison below. NOTE: Because I am using CV the folds are selected at random so the numbers may be slightly off from the comments. .

# AI assistance was used on this specific plot
errors <- c(Linear = 1 - (sv.class$tot.accuracy / 100),
            Radial = 1 - (sv.radial$tot.accuracy / 100),
            Polynomial = 1 - (sv.polynomial$tot.accuracy / 100))

# 2. Plot a clean bar chart
barplot(errors * 100, 
        main = "Cross-Validation Error Comparison",
        ylab = "Error Rate (%)", 
        xlab = "SVM Kernel Type",
        col = c("#0072B2", "#E69F00", "#009E73"),
        ylim = c(0, 15))

# Add the exact percentage text on top of each bar
text(x = c(0.7, 1.9, 3.1), y = (errors * 100) + 0.8, 
     labels = paste0(round(errors * 100, 2), "%"), font = 2)

(d) Plots

Make some plots to back up your assertions in (b) and (c). Hint: In the lab, we used the plot() function for svm objects only in cases with p =2. When p> 2, you can use the plot() function to create plots displaying pairs of variables at a time.

# Plot the affect of weight and horsepower on MPG

# Linear SV
plot(sv.class, Auto1, weight ~ horsepower)

plot(sv.class, Auto1, displacement ~ cylinders)

# Radial SV
plot(sv.radial, Auto1, weight ~ horsepower)

plot(sv.radial, Auto1, displacement ~ cylinders)

# Polynomial SV
plot(sv.polynomial, Auto1, weight ~ horsepower)

plot(sv.polynomial, Auto1, displacement ~ cylinders)

Comment on your results. The Linear support vector machine models had the highest CV error, with radial in second and polynomial being the best model. Weight, horsepower, displacement, and cylinders are appeared to be significant predictors in MPG. The graphs show clear evidence that cars with lower MPG have lower horsepower and weight than cars with higher MPG. Also, the cars with lower cylinders and displacement tend to have lower MPG. .

Q8 - Predicting OJ loyalty

This problem involves the OJ data set which is part of the ISLR2 package.

OJ

(a) Test/Train split

Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

# Set the seed for reproduceablity
set.seed(189) 

# create the test/train split. Here I perform a 70/30 split
train <- createDataPartition(
  y = OJ$Purchase, 
  p=800/nrow(OJ), 
  list = FALSE)

train_data <- OJ[train, ]
test_data <- OJ[-train, ]

(b) Support Vector (linear)

Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.

# Support Vector Classifier model

# training data run
lin_sv_train <- svm(Purchase ~ .,
               data = train_data, 
               kernel = "linear", 
               cost = .01,
               cross = 5)
summary(lin_sv_train)

## 
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "linear", 
##     cost = 0.01, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.01 
## 
## Number of Support Vectors:  443
## 
##  ( 223 220 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 81.02372 
## Single Accuracies:
##  83.75 80.625 78.75 81.875 80.12422

# testing data run
lin_sv_test <- svm(Purchase ~ .,
               data = test_data, 
               kernel = "linear", 
               cost = .01,
               cross = 5)
summary(lin_sv_test)

## 
## Call:
## svm(formula = Purchase ~ ., data = test_data, kernel = "linear", 
##     cost = 0.01, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.01 
## 
## Number of Support Vectors:  177
## 
##  ( 89 88 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 84.01487 
## Single Accuracies:
##  83.01887 83.33333 88.88889 85.18519 79.62963

Comment on your results. The model accurately predicted OJ class around 81% of the time shown in the CV total accuracy. 443 Support vectors were used between the MM and CH class to fit the decision boundary with 223 CH records and 220 MM records. .

(c) Test/Train Error Rates

What are the training and test error rates?

# compute the cv error

# train error
train_cv_error_lin <- 1 - (lin_sv_train$tot.accuracy / 100)
print(train_cv_error_lin)

## [1] 0.1897628

# test error
test_cv_error_lin <- 1 - (lin_sv_test$tot.accuracy / 100)
print(test_cv_error_lin)

## [1] 0.1598513

• Train Error .19 showing that 19% of the time the model misclassifies the response

• Test Error: .16 showing that 16% of the time the model misclassifies the response .

(d) Tuning the SV cost

Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.

set.seed(55)

# Run a tune() function to see what the best cost value is
tune.out_train <- tune(svm, 
                 Purchase ~ ., 
                 data = train_data, 
                 kernel = "linear",
                 ranges = list(cost = 
                                 c(0.001, 0.81, 0.8, 4, 7, 10))
                 )

summary(tune.out_train)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##  0.81
## 
## - best performance: 0.1759259 
## 
## - Detailed performance results:
##     cost     error dispersion
## 1  0.001 0.3308333 0.06849912
## 2  0.810 0.1759259 0.05126570
## 3  0.800 0.1759259 0.05126570
## 4  4.000 0.1796605 0.05308830
## 5  7.000 0.1821605 0.05565997
## 6 10.000 0.1809105 0.05597810

• Thoughts The optimal cost value was 0.81 between the six costs I tested on the training data. This model produced the lowest CV error right around 17.6% showing that over 82% of the time this model would predict the response variable accurately .

(e) Optimal SV Model

Compute the training and test error rates using this new value for cost.

# train data run
lin_sv_train <- svm(Purchase ~ .,
               data = train_data, 
               kernel = "linear", 
               degree = 2,
               cost = .81,
               cross = 5)
summary(lin_sv_train)

## 
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "linear", 
##     degree = 2, cost = 0.81, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.81 
## 
## Number of Support Vectors:  340
## 
##  ( 171 169 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 82.27216 
## Single Accuracies:
##  82.5 81.875 82.5 82.5 81.98758

# test data run
lin_sv_test <- svm(Purchase ~ .,
               data = test_data, 
               kernel = "linear", 
               cost = .81,
               cross = 5)
summary(lin_sv_test)

## 
## Call:
## svm(formula = Purchase ~ ., data = test_data, kernel = "linear", 
##     cost = 0.81, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.81 
## 
## Number of Support Vectors:  103
## 
##  ( 52 51 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 84.01487 
## Single Accuracies:
##  81.13208 85.18519 90.74074 85.18519 77.77778

#calculate the error
train_cv_error_lin <- 1 - (lin_sv_train$tot.accuracy / 100)
print(train_cv_error_lin)

## [1] 0.1772784

test_cv_error_lin <- 1 - (lin_sv_test$tot.accuracy / 100)
print(test_cv_error_lin)

## [1] 0.1598513

• Train Error The training error was 17.7%

• Test Error The test error was 15.9% .

(f) Support Vector (radial)

Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.

# Support Vector Classifier model

# Tune to find the best cost value
tune.out_train <- tune(svm, 
                 Purchase ~ ., 
                 data = train_data, 
                 kernel = "radial",
                 ranges = list(cost = 
                                 c(0.001, 0.81, 0.8, 4, 7, 10))
                 )

summary(tune.out_train)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##  0.81
## 
## - best performance: 0.1735494 
## 
## - Detailed performance results:
##     cost     error dispersion
## 1  0.001 0.3895679 0.04562831
## 2  0.810 0.1735494 0.03312751
## 3  0.800 0.1735494 0.03312751
## 4  4.000 0.1822840 0.03451909
## 5  7.000 0.1860340 0.03206926
## 6 10.000 0.1860185 0.02972496

# training data run
rad_sv_train <- svm(Purchase ~ .,
               data = train_data, 
               kernel = "radial", 
               cost = .8,
               cross = 5)
summary(rad_sv_train)

## 
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "radial", 
##     cost = 0.8, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  0.8 
## 
## Number of Support Vectors:  388
## 
##  ( 196 192 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 82.89638 
## Single Accuracies:
##  85.625 84.375 83.75 77.5 83.22981

train_cv_error_rad <- 1 - (rad_sv_train$tot.accuracy / 100)
print(train_cv_error_rad)

## [1] 0.1710362

# test data run
rad_sv_test <- svm(Purchase ~ .,
               data = test_data, 
               kernel = "radial", 
               cost = .8,
               cross = 5)
summary(rad_sv_test)

## 
## Call:
## svm(formula = Purchase ~ ., data = test_data, kernel = "radial", 
##     cost = 0.8, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  0.8 
## 
## Number of Support Vectors:  148
## 
##  ( 76 72 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 83.27138 
## Single Accuracies:
##  92.45283 75.92593 83.33333 83.33333 81.48148

test_cv_error_rad <- 1 - (rad_sv_test$tot.accuracy / 100)
print(test_cv_error_rad)

## [1] 0.1672862

• Train Error The training error was 17.1%

• Test Error The test error was 16.7%

Note The optimal cost for these models was .8 and they produced CV error that was very close. This is a positive sign that shows that around 82% of the time this model predicts accurately. .

(g) Support Vector (polynomial)

Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.

# Support Vector Classifier model

# Tune to find the best cost value
tune.out_train <- tune(svm, 
                 Purchase ~ ., 
                 data = train_data, 
                 kernel = "polynomial",
                 degree = 2,
                 ranges = list(cost = 
                                 c(0.001, 0.81, 0.8, 4, 7, 10))
                 )

summary(tune.out_train)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##    10
## 
## - best performance: 0.1923148 
## 
## - Detailed performance results:
##     cost     error dispersion
## 1  0.001 0.3894907 0.04213955
## 2  0.810 0.2072840 0.04953353
## 3  0.800 0.2072840 0.04953353
## 4  4.000 0.1985494 0.04163625
## 5  7.000 0.1960648 0.04433460
## 6 10.000 0.1923148 0.04237464

# training data run
pol_sv_train <- svm(Purchase ~ .,
               data = train_data, 
               kernel = "polynomial", 
               degree = 2,
               cost = 4,
               cross = 5)
summary(pol_sv_train)

## 
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "polynomial", 
##     degree = 2, cost = 4, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  polynomial 
##        cost:  4 
##      degree:  2 
##      coef.0:  0 
## 
## Number of Support Vectors:  382
## 
##  ( 195 187 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 79.77528 
## Single Accuracies:
##  76.875 78.75 78.125 82.5 82.6087

train_cv_error_pol <- 1 - (pol_sv_train$tot.accuracy / 100)
print(train_cv_error_pol)

## [1] 0.2022472

# test data run
pol_sv_test <- svm(Purchase ~ .,
               data = test_data, 
               kernel = "polynomial", 
               degree = 2,
               cost = 4,
               cross = 5)
summary(pol_sv_test)

## 
## Call:
## svm(formula = Purchase ~ ., data = test_data, kernel = "polynomial", 
##     degree = 2, cost = 4, cross = 5)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  polynomial 
##        cost:  4 
##      degree:  2 
##      coef.0:  0 
## 
## Number of Support Vectors:  143
## 
##  ( 76 67 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## 
## 5-fold cross-validation on training data:
## 
## Total Accuracy: 81.41264 
## Single Accuracies:
##  79.24528 81.48148 85.18519 75.92593 85.18519

test_cv_error_pol <- 1 - (pol_sv_test$tot.accuracy / 100)
print(test_cv_error_pol)

## [1] 0.1858736

• Train Error The training error was 20.2%

• Test Error The test error was 18.6%

Note The optimal cost for these models was 4 and they produced CV error that was very close. This is a positive sign that shows that around 81% of the time this model predicts accurately. .

(h) Final Results

Overall, which approach seems to give the best results on this data?

errors <- c(Linear = 1 - (lin_sv_train$tot.accuracy / 100),
            Radial = 1 - (rad_sv_train$tot.accuracy / 100),
            Polynomial = 1 - (pol_sv_train$tot.accuracy / 100))

# 2. Plot a clean bar chart
barplot(errors * 100, 
        main = "Training Set Cross-Validation Error Comparison",
        ylab = "Error Rate (%)", 
        xlab = "SVM Kernel Type",
        col = c("#0072B2", "#E69F00", "#009E73"),
        ylim = c(0,30))

# Add the exact percentage text on top of each bar
text(x = c(0.7, 1.9, 3.1), y = (errors * 100) + 0.8, 
     labels = paste0(round(errors * 100, 2), "%"), font = 2)

errors <- c(Linear = 1 - (lin_sv_test$tot.accuracy / 100),
            Radial = 1 - (rad_sv_test$tot.accuracy / 100),
            Polynomial = 1 - (pol_sv_test$tot.accuracy / 100))

# 2. Plot a clean bar chart
barplot(errors * 100, 
        main = "Test Set Cross-Validation Error Comparison",
        ylab = "Error Rate (%)", 
        xlab = "SVM Kernel Type",
        col = c("#0072B2", "#E69F00", "#009E73"),
        ylim = c(0,30))

# Add the exact percentage text on top of each bar
text(x = c(0.7, 1.9, 3.1), y = (errors * 100) + 0.8, 
     labels = paste0(round(errors * 100, 2), "%"), font = 2)

Final thoughts Among the training data the radial support vector model yielded the best results with a 17.% error rate. Among the test data the linear support vector yielded the best result with a 15.99% error rate. If I had to choose the best support vector between the three I would choose the radial model due to how closely it ranked in the top 2 amongst the other model types. .