SLlab10

Question 4

Generate a simulated two-class data set with 100 observations and two features in which there is a visible but non-linear separation between the two classes. Show that in this setting, a support vector machine with a polynomial kernel (with degree greater than 1) or a radial kernel will outperform a support vector classifier on the training data. Which technique performs best on the test data? Make plots and report training and test error rates in order to back up your assertions.

# Load required libraries
library(e1071)

## Warning: package 'e1071' was built under R version 4.4.3

library(ggplot2)

set.seed(10)
data <- data.frame(
  x = runif(100),
  y = runif(100)
)
score <- (2 * data$x - 0.5)^2 + (data$y)^2 - 0.5
data$class <- factor(ifelse(score > 0, "red", "blue"))

p <- ggplot(data, aes(x = x, y = y, color = class)) +
  geom_point(size = 2) +
  scale_colour_identity()
p

train <- 1:50
test <- 51:100

fits <- list(
  "Radial" = svm(class ~ ., data = data[train, ], kernel = "radial"),
  "Polynomial" = svm(class ~ ., data = data[train, ], kernel = "polynomial", degree = 2),
  "Linear" = svm(class ~ ., data = data[train, ], kernel = "linear")
)

err <- function(model, data) {
  out <- table(predict(model, data), data$class)
  (out[1, 2] + out[2, 1]) / sum(out)
}
plot(fits[[1]], data)

plot(fits[[2]], data)

plot(fits[[3]], data)

sapply(fits, err, data = data[train, ])

##     Radial Polynomial     Linear 
##       0.04       0.30       0.10

sapply(fits, err, data = data[test, ])

##     Radial Polynomial     Linear 
##       0.06       0.48       0.14

The Radial Basis Function (RBF) kernel outperforms the other methods both in training and generalization, making it the best choice for this non-linear classification task.

Question 7

In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.

Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.

library(ISLR2)
data <- Auto
data$high_mpg <- as.factor(as.numeric(data$mpg > median(data$mpg)))

Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results.

set.seed(42)
costs <- 10^seq(-4, 3, by = 0.5)
results <- list()
f <- high_mpg ~ displacement + horsepower + weight
results$linear <- tune(svm, f,
  data = data, kernel = "linear",
  ranges = list(cost = costs)
)
summary(results$linear)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##        cost
##  0.03162278
## 
## - best performance: 0.1019231 
## 
## - Detailed performance results:
##            cost     error dispersion
## 1  1.000000e-04 0.5967949 0.05312225
## 2  3.162278e-04 0.5967949 0.05312225
## 3  1.000000e-03 0.2199359 0.08718077
## 4  3.162278e-03 0.1353846 0.06058195
## 5  1.000000e-02 0.1121795 0.04011293
## 6  3.162278e-02 0.1019231 0.05087176
## 7  1.000000e-01 0.1096154 0.05246238
## 8  3.162278e-01 0.1044872 0.05154934
## 9  1.000000e+00 0.1044872 0.05154934
## 10 3.162278e+00 0.1044872 0.05154934
## 11 1.000000e+01 0.1019231 0.05501131
## 12 3.162278e+01 0.1019231 0.05501131
## 13 1.000000e+02 0.1019231 0.05501131
## 14 3.162278e+02 0.1019231 0.05501131
## 15 1.000000e+03 0.1019231 0.05501131

Using a linear kernel, you performed cross-validation over a range of cost values. The cross-validation errors indicate:

Lower cost values may lead to underfitting, resulting in higher error rates.
Higher cost values can cause overfitting, also increasing error rates.
An optimal cost value minimizes the cross-validation error, balancing the trade-off between margin width and misclassification.

Now repeat (b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.

results$polynomial <- tune(svm, f,
  data = data, kernel = "polynomial",
  ranges = list(cost = costs, degree = 1:3)
)
summary(results$polynomial)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost degree
##   0.1      1
## 
## - best performance: 0.101859 
## 
## - Detailed performance results:
##            cost degree     error dispersion
## 1  1.000000e-04      1 0.5842949 0.04703306
## 2  3.162278e-04      1 0.5842949 0.04703306
## 3  1.000000e-03      1 0.5842949 0.04703306
## 4  3.162278e-03      1 0.2167949 0.07891173
## 5  1.000000e-02      1 0.1275641 0.04806885
## 6  3.162278e-02      1 0.1147436 0.05661708
## 7  1.000000e-01      1 0.1018590 0.05732429
## 8  3.162278e-01      1 0.1069231 0.05949679
## 9  1.000000e+00      1 0.1069231 0.06307278
## 10 3.162278e+00      1 0.1069231 0.06307278
## 11 1.000000e+01      1 0.1043590 0.06603760
## 12 3.162278e+01      1 0.1043590 0.06603760
## 13 1.000000e+02      1 0.1043590 0.06603760
## 14 3.162278e+02      1 0.1043590 0.06603760
## 15 1.000000e+03      1 0.1043590 0.06603760
## 16 1.000000e-04      2 0.5842949 0.04703306
## 17 3.162278e-04      2 0.5842949 0.04703306
## 18 1.000000e-03      2 0.5842949 0.04703306
## 19 3.162278e-03      2 0.5255128 0.08090636
## 20 1.000000e-02      2 0.3980769 0.08172400
## 21 3.162278e-02      2 0.3674359 0.07974741
## 22 1.000000e-01      2 0.3597436 0.08336609
## 23 3.162278e-01      2 0.3597436 0.09010398
## 24 1.000000e+00      2 0.3444872 0.08767258
## 25 3.162278e+00      2 0.3545513 0.10865903
## 26 1.000000e+01      2 0.3239103 0.09593710
## 27 3.162278e+01      2 0.3035256 0.08184137
## 28 1.000000e+02      2 0.3061538 0.08953945
## 29 3.162278e+02      2 0.3060897 0.08919821
## 30 1.000000e+03      2 0.3035897 0.09305216
## 31 1.000000e-04      3 0.5842949 0.04703306
## 32 3.162278e-04      3 0.4955128 0.10081350
## 33 1.000000e-03      3 0.3750641 0.08043982
## 34 3.162278e-03      3 0.3036538 0.09096445
## 35 1.000000e-02      3 0.2601282 0.07774595
## 36 3.162278e-02      3 0.2499359 0.08407106
## 37 1.000000e-01      3 0.2017949 0.07547413
## 38 3.162278e-01      3 0.1937179 0.08427411
## 39 1.000000e+00      3 0.1478205 0.04579654
## 40 3.162278e+00      3 0.1451923 0.05169638
## 41 1.000000e+01      3 0.1451282 0.04698931
## 42 3.162278e+01      3 0.1500000 0.07549058
## 43 1.000000e+02      3 0.1373718 0.05772558
## 44 3.162278e+02      3 0.1271795 0.06484766
## 45 1.000000e+03      3 0.1322436 0.06764841

results$radial <- tune(svm, f,
  data = data, kernel = "radial",
  ranges = list(cost = costs, gamma = 10^(-2:1))
)
summary(results$radial)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost gamma
##  1000   0.1
## 
## - best performance: 0.08179487 
## 
## - Detailed performance results:
##            cost gamma      error dispersion
## 1  1.000000e-04  0.01 0.58410256 0.05435320
## 2  3.162278e-04  0.01 0.58410256 0.05435320
## 3  1.000000e-03  0.01 0.58410256 0.05435320
## 4  3.162278e-03  0.01 0.58410256 0.05435320
## 5  1.000000e-02  0.01 0.58410256 0.05435320
## 6  3.162278e-02  0.01 0.26557692 0.10963269
## 7  1.000000e-01  0.01 0.15038462 0.05783237
## 8  3.162278e-01  0.01 0.11224359 0.04337812
## 9  1.000000e+00  0.01 0.10730769 0.04512161
## 10 3.162278e+00  0.01 0.10730769 0.04512161
## 11 1.000000e+01  0.01 0.10737179 0.05526490
## 12 3.162278e+01  0.01 0.10480769 0.05610124
## 13 1.000000e+02  0.01 0.10480769 0.05610124
## 14 3.162278e+02  0.01 0.10737179 0.05526490
## 15 1.000000e+03  0.01 0.10993590 0.05690926
## 16 1.000000e-04  0.10 0.58410256 0.05435320
## 17 3.162278e-04  0.10 0.58410256 0.05435320
## 18 1.000000e-03  0.10 0.58410256 0.05435320
## 19 3.162278e-03  0.10 0.58410256 0.05435320
## 20 1.000000e-02  0.10 0.15301282 0.06026554
## 21 3.162278e-02  0.10 0.11480769 0.04514816
## 22 1.000000e-01  0.10 0.10730769 0.04512161
## 23 3.162278e-01  0.10 0.10730769 0.04512161
## 24 1.000000e+00  0.10 0.10737179 0.05526490
## 25 3.162278e+00  0.10 0.10737179 0.05526490
## 26 1.000000e+01  0.10 0.10737179 0.05526490
## 27 3.162278e+01  0.10 0.10737179 0.05526490
## 28 1.000000e+02  0.10 0.09967949 0.04761387
## 29 3.162278e+02  0.10 0.08429487 0.03207585
## 30 1.000000e+03  0.10 0.08179487 0.03600437
## 31 1.000000e-04  1.00 0.58410256 0.05435320
## 32 3.162278e-04  1.00 0.58410256 0.05435320
## 33 1.000000e-03  1.00 0.58410256 0.05435320
## 34 3.162278e-03  1.00 0.58410256 0.05435320
## 35 1.000000e-02  1.00 0.12506410 0.05342773
## 36 3.162278e-02  1.00 0.10730769 0.06255920
## 37 1.000000e-01  1.00 0.10993590 0.05561080
## 38 3.162278e-01  1.00 0.10737179 0.05526490
## 39 1.000000e+00  1.00 0.09711538 0.05107441
## 40 3.162278e+00  1.00 0.08429487 0.03634646
## 41 1.000000e+01  1.00 0.08692308 0.03877861
## 42 3.162278e+01  1.00 0.08948718 0.03503648
## 43 1.000000e+02  1.00 0.09198718 0.03272127
## 44 3.162278e+02  1.00 0.10217949 0.04214031
## 45 1.000000e+03  1.00 0.09692308 0.04645046
## 46 1.000000e-04 10.00 0.58410256 0.05435320
## 47 3.162278e-04 10.00 0.58410256 0.05435320
## 48 1.000000e-03 10.00 0.58410256 0.05435320
## 49 3.162278e-03 10.00 0.58410256 0.05435320
## 50 1.000000e-02 10.00 0.58410256 0.05435320
## 51 3.162278e-02 10.00 0.22205128 0.12710181
## 52 1.000000e-01 10.00 0.11237179 0.03888895
## 53 3.162278e-01 10.00 0.10217949 0.04375722
## 54 1.000000e+00 10.00 0.09717949 0.03809440
## 55 3.162278e+00 10.00 0.09717949 0.03809440
## 56 1.000000e+01 10.00 0.09711538 0.04161705
## 57 3.162278e+01 10.00 0.11487179 0.04240664
## 58 1.000000e+02 10.00 0.13019231 0.03541140
## 59 3.162278e+02 10.00 0.13532051 0.03865626
## 60 1.000000e+03 10.00 0.14044872 0.04251917

sapply(results, function(x) x$best.performance)

##     linear polynomial     radial 
## 0.10192308 0.10185897 0.08179487

sapply(results, function(x) x$best.parameters)

## $linear
##         cost
## 6 0.03162278
## 
## $polynomial
##   cost degree
## 7  0.1      1
## 
## $radial
##    cost gamma
## 30 1000   0.1

Make some plots to back up your assertions in (b) and (c).

Hint: In the lab, we used the plot() function for svm objects only in cases with \(p = 2\). When \(p > 2\), you can use the plot() function to create plots displaying pairs of variables at a time. Essentially, instead of typing
> plot(svmfit, dat)
where svmfit contains your fitted model and dat is a data frame containing your data, you can type
> plot(svmfit, dat, x1 ∼ x4)
in order to plot just the first and fourth variables. However, you must replace x1 and x4 with the correct variable names. To find out more, type ?plot.svm.

table(predict(results$radial$best.model, data), data$high_mpg)

##    
##       0   1
##   0 176   5
##   1  20 191

plot(results$radial$best.model, data, horsepower ~ displacement)

plot(results$radial$best.model, data, horsepower ~ weight)

plot(results$radial$best.model, data, displacement ~ weight)

Plotted the decision boundaries of the best-performing radial SVM model across pairs of features:

Horsepower vs. Displacement: Shows a clear separation between high and low mpg classes.
Horsepower vs. Weight: Highlights the influence of weight on fuel efficiency.
Displacement vs. Weight: Further emphasizes the relationship between engine size, vehicle weight, and mpg.

Question 8

This problem involves the OJ data set which is part of the ISLR2 package.

Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

set.seed(42)
train <- sample(seq_len(nrow(OJ)), 800)
test <- setdiff(seq_len(nrow(OJ)), train)

Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.

fit <- svm(Purchase ~ ., data = OJ[train, ], kernel = "linear", cost = 0.01)
summary(fit)

## 
## Call:
## svm(formula = Purchase ~ ., data = OJ[train, ], kernel = "linear", 
##     cost = 0.01)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.01 
## 
## Number of Support Vectors:  432
## 
##  ( 215 217 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM

Training a linear SVM with a low cost value resulted in a model with a wider margin but potentially more misclassifications. The summary statistics indicate the number of support vectors and the model’s coefficients.

What are the training and test error rates?

err <- function(model, data) {
  t <- table(predict(model, data), data[["Purchase"]])
  1 - sum(diag(t)) / sum(t)
}
errs <- function(model) {
  c(train = err(model, OJ[train, ]), test = err(model, OJ[test, ]))
}
errs(fit)

##    train     test 
## 0.171250 0.162963

Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.

tuned <- tune(svm, Purchase ~ .,
  data = OJ[train, ], kernel = "linear",
  ranges = list(cost = 10^seq(-2, 1, length.out = 10))
)
tuned$best.parameters

##   cost
## 7    1

summary(tuned)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##     1
## 
## - best performance: 0.1775 
## 
## - Detailed performance results:
##           cost   error dispersion
## 1   0.01000000 0.18250 0.04133199
## 2   0.02154435 0.18000 0.04005205
## 3   0.04641589 0.18000 0.05041494
## 4   0.10000000 0.18000 0.04901814
## 5   0.21544347 0.18250 0.04377975
## 6   0.46415888 0.18250 0.04090979
## 7   1.00000000 0.17750 0.04031129
## 8   2.15443469 0.18000 0.03961621
## 9   4.64158883 0.17875 0.03821086
## 10 10.00000000 0.18375 0.03438447

Compute the training and test error rates using this new value for cost.

errs(tuned$best.model)

##    train     test 
## 0.167500 0.162963

Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.

tuned2 <- tune(svm, Purchase ~ .,
  data = OJ[train, ], kernel = "radial",
  ranges = list(cost = 10^seq(-2, 1, length.out = 10))
)
tuned2$best.parameters

##        cost
## 6 0.4641589

errs(tuned2$best.model)

##     train      test 
## 0.1525000 0.1666667

Training an SVM with a radial kernel and tuning the cost parameter resulted in a model with lower test error compared to the linear kernel. This suggests that the radial kernel captures nonlinear relationships in the data more effectively.

Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.

tuned3 <- tune(svm, Purchase ~ .,
  data = OJ[train, ], kernel = "polynomial",
  ranges = list(cost = 10^seq(-2, 1, length.out = 10)), degree = 2
)
tuned3$best.parameters

##       cost
## 9 4.641589

errs(tuned3$best.model)

##     train      test 
## 0.1487500 0.1703704

Using a polynomial kernel of degree 2 and tuning the cost parameter yielded a model with performance between that of the linear and radial kernels. It indicates that while the polynomial kernel can model some nonlinearity, it may not be as flexible as the radial kernel.

Overall, which approach seems to give the best results on this data?

Among all the models tested, the SVM with the radial kernel achieved the lowest test error, indicating it is the most effective for this classification task. It balances the trade-off between bias and variance and captures the underlying patterns in the data more accurately.

SLlab10

Sai Pranam

2025-04-24

Question 4

Question 7

Question 8