ISLR Chapter 9 - Exercises 5, 7, and 8
Osvaldo Lugo
July 29, 2026
Overview
This document works through three exercises from the Applied section of ISLR Chapter 9 (Support Vector Machines):
- Exercise 5 — showing that logistic regression with non-linear transformations of the predictors can approximate a non-linear decision boundary in the same way an SVM with a non-linear kernel can.
- Exercise 7 — using support vector approaches
(linear, radial, and polynomial kernels) to predict high vs. low gas
mileage in the
Autodata set. - Exercise 8 — using support vector
classifiers/machines to predict
Purchase(Citrus Hill vs. Minute Maid) in theOJdata set, including cost tuning via cross-validation.
The Auto and OJ data sets are the standard
data sets that ship with the ISLR2 package (392
observations / 9 variables, and 1,070 observations / 18 variables,
respectively).
Exercise 5
We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
(a) Generate a data set with a quadratic decision boundary
set.seed(1)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)
dat5 <- data.frame(x1 = x1, x2 = x2, y = as.factor(y))
head(dat5)## x1 x2 y
## 1 -0.23449134 0.05417706 1
## 2 -0.12787610 0.18827524 0
## 3 0.07285336 0.15805755 0
## 4 0.40820779 0.16334273 1
## 5 -0.29831807 -0.02776580 1
## 6 0.39838968 0.46952817 0table(dat5$y)##
## 0 1
## 261 239(b) Plot the observations, colored by true class label
ggplot(dat5, aes(x1, x2, color = y)) +
geom_point(size = 1.6) +
scale_color_manual(values = c("0" = "#2E5EAA", "1" = "#C0392B")) +
labs(title = "True class labels", subtitle = "Quadratic boundary: x1^2 - x2^2 > 0",
color = "Class") +
theme_minimal(base_size = 12)
The two classes are clearly separated, but by a curve (a hyperbola-like boundary), not by a straight line — this is the whole point of the exercise.
(c) Fit a logistic regression model using X1 and X2 (linear terms only)
glm.fit <- glm(y ~ x1 + x2, data = dat5, family = binomial)
summary(glm.fit)##
## Call:
## glm(formula = y ~ x1 + x2, family = binomial, data = dat5)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.087260 0.089579 -0.974 0.330
## x1 0.196199 0.316864 0.619 0.536
## x2 -0.002854 0.305712 -0.009 0.993
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 692.18 on 499 degrees of freedom
## Residual deviance: 691.79 on 497 degrees of freedom
## AIC: 697.79
##
## Number of Fisher Scoring iterations: 3Neither x1 nor x2 is statistically
significant on its own — unsurprising, since the true boundary is
quadratic and a linear-in-x1,x2 model has no
way to represent it.
(d) Apply the linear logistic model to the training data and plot predicted labels
glm.probs <- predict(glm.fit, type = "response")
glm.pred <- as.factor(ifelse(glm.probs > 0.5, 1, 0))
dat5$pred_linear <- glm.pred
table(Predicted = glm.pred, Actual = dat5$y)## Actual
## Predicted 0 1
## 0 258 212
## 1 3 27ggplot(dat5, aes(x1, x2, color = pred_linear)) +
geom_point(size = 1.6) +
scale_color_manual(values = c("0" = "#2E5EAA", "1" = "#C0392B")) +
labs(title = "Logistic regression (linear terms) — predicted labels",
color = "Predicted class") +
theme_minimal(base_size = 12)
As expected, the decision boundary produced by the linear logistic model is a straight line, and it does a poor job separating the true classes — most points are predicted into a single dominant class near the boundary.
(e) Fit a logistic regression model with non-linear functions of X1, X2
glm.fit2 <- glm(y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2),
data = dat5, family = binomial)
summary(glm.fit2)##
## Call:
## glm(formula = y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2), family = binomial,
## data = dat5)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -102.2 4302.0 -0.024 0.981
## poly(x1, 2)1 2715.3 141109.5 0.019 0.985
## poly(x1, 2)2 27218.5 842987.2 0.032 0.974
## poly(x2, 2)1 -279.7 97160.4 -0.003 0.998
## poly(x2, 2)2 -28693.0 875451.3 -0.033 0.974
## I(x1 * x2) -206.4 41802.8 -0.005 0.996
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 6.9218e+02 on 499 degrees of freedom
## Residual deviance: 3.5810e-06 on 494 degrees of freedom
## AIC: 12
##
## Number of Fisher Scoring iterations: 25(f) Apply this non-linear model to the training data and plot predicted labels
glm.probs2 <- predict(glm.fit2, type = "response")
glm.pred2 <- as.factor(ifelse(glm.probs2 > 0.5, 1, 0))
dat5$pred_nonlinear <- glm.pred2
table(Predicted = glm.pred2, Actual = dat5$y)## Actual
## Predicted 0 1
## 0 261 0
## 1 0 239ggplot(dat5, aes(x1, x2, color = pred_nonlinear)) +
geom_point(size = 1.6) +
scale_color_manual(values = c("0" = "#2E5EAA", "1" = "#C0392B")) +
labs(title = "Logistic regression (quadratic terms) — predicted labels",
color = "Predicted class") +
theme_minimal(base_size = 12)
With quadratic terms included, the fitted decision boundary is
obviously non-linear and essentially reproduces the true quadratic
boundary — the model perfectly separates the two classes on the training
data in this case, because the transformed features (x1^2,
x2^2) exactly match the functional form of the true
boundary.
(g) Fit a support vector classifier (linear kernel) with X1, X2 as predictors
svm.fit <- svm(y ~ x1 + x2, data = dat5, kernel = "linear", cost = 0.1)
svm.pred <- predict(svm.fit, dat5)
dat5$pred_svc <- svm.pred
table(Predicted = svm.pred, Actual = dat5$y)## Actual
## Predicted 0 1
## 0 261 239
## 1 0 0ggplot(dat5, aes(x1, x2, color = pred_svc)) +
geom_point(size = 1.6) +
scale_color_manual(values = c("0" = "#2E5EAA", "1" = "#C0392B")) +
labs(title = "Support vector classifier (linear kernel) — predicted labels",
color = "Predicted class") +
theme_minimal(base_size = 12)
The linear support vector classifier, like linear logistic regression, cannot represent a curved boundary — with this cost it predicts every observation into a single class, since no straight line comes close to separating the true quadratic pattern.
(h) Fit an SVM with a non-linear (radial) kernel
svm.fit2 <- svm(y ~ x1 + x2, data = dat5, kernel = "radial", gamma = 1, cost = 1)
svm.pred2 <- predict(svm.fit2, dat5)
dat5$pred_svm_radial <- svm.pred2
table(Predicted = svm.pred2, Actual = dat5$y)## Actual
## Predicted 0 1
## 0 258 11
## 1 3 228ggplot(dat5, aes(x1, x2, color = pred_svm_radial)) +
geom_point(size = 1.6) +
scale_color_manual(values = c("0" = "#2E5EAA", "1" = "#C0392B")) +
labs(title = "SVM (radial kernel) — predicted labels",
color = "Predicted class") +
theme_minimal(base_size = 12)
(i) Comment on the results
err <- function(pred, truth) mean(pred != truth)
data.frame(
Model = c("Logistic (linear terms)", "Logistic (quadratic terms)",
"SVC (linear kernel)", "SVM (radial kernel)"),
Training_Error = c(err(dat5$pred_linear, dat5$y),
err(dat5$pred_nonlinear, dat5$y),
err(dat5$pred_svc, dat5$y),
err(dat5$pred_svm_radial, dat5$y))
)## Model Training_Error
## 1 Logistic (linear terms) 0.430
## 2 Logistic (quadratic terms) 0.000
## 3 SVC (linear kernel) 0.478
## 4 SVM (radial kernel) 0.028Two linear methods — logistic regression with only linear terms, and a linear-kernel support vector classifier — both fail to capture the true quadratic decision boundary, and their training error rates are correspondingly poor (the linear SVC in particular collapses to predicting a single class). Once non-linearity is introduced — either by adding quadratic/interaction terms to logistic regression, or by using a radial-kernel SVM — both approaches recover the true boundary shape and achieve low training error. This demonstrates the exercise’s core point: a non-linear decision boundary can be obtained either by using an inherently non-linear method (SVM with a non-linear kernel) or by manually engineering non-linear features into an otherwise linear method (logistic regression) — the two approaches are more similar in spirit than they might first appear, though SVMs with kernels avoid having to guess the right functional form in advance.
Exercise 7
In this problem, you will use support vector approaches in order
to predict whether a given car gets high or low gas mileage based on the
Auto data set.
load_islr2_data <- function(dataset_name) {
if (requireNamespace("ISLR2", quietly = TRUE)) {
library(ISLR2)
return(get(dataset_name, envir = asNamespace("ISLR2")))
}
# Fallback: download the .rda directly from the ISLR2 CRAN source mirror
rda_file <- paste0(dataset_name, ".rda")
if (!file.exists(rda_file)) {
url <- paste0("https://raw.githubusercontent.com/cran/ISLR2/master/data/",
dataset_name, ".rda")
download.file(url, rda_file, mode = "wb", quiet = TRUE)
}
e <- new.env()
load(rda_file, envir = e)
get(dataset_name, envir = e)
}
Auto <- load_islr2_data("Auto") # ISLR2::Auto — 392 obs. of 9 variables
str(Auto)## 'data.frame': 392 obs. of 9 variables:
## $ mpg : num 18 15 18 16 17 15 14 14 14 15 ...
## $ cylinders : int 8 8 8 8 8 8 8 8 8 8 ...
## $ displacement: num 307 350 318 304 302 429 454 440 455 390 ...
## $ horsepower : int 130 165 150 150 140 198 220 215 225 190 ...
## $ weight : int 3504 3693 3436 3433 3449 4341 4354 4312 4425 3850 ...
## $ acceleration: num 12 11.5 11 12 10.5 10 9 8.5 10 8.5 ...
## $ year : int 70 70 70 70 70 70 70 70 70 70 ...
## $ origin : int 1 1 1 1 1 1 1 1 1 1 ...
## $ name : Factor w/ 304 levels "amc ambassador brougham",..: 49 36 231 14 161 141 54 223 241 2 ...
## - attr(*, "na.action")= 'omit' Named int [1:5] 33 127 331 337 355
## ..- attr(*, "names")= chr [1:5] "33" "127" "331" "337" ...(a) Create a binary high/low mileage variable
mpg01 <- as.factor(ifelse(Auto$mpg > median(Auto$mpg), 1, 0))
table(mpg01)## mpg01
## 0 1
## 196 196# Build the modeling data set: drop mpg (used to build the label) and name
# (a near-unique car-model identifier, not a usable predictor)
Auto2 <- Auto
Auto2$mpg01 <- mpg01
Auto2$mpg <- NULL
Auto2$name <- NULL
str(Auto2)## 'data.frame': 392 obs. of 8 variables:
## $ cylinders : int 8 8 8 8 8 8 8 8 8 8 ...
## $ displacement: num 307 350 318 304 302 429 454 440 455 390 ...
## $ horsepower : int 130 165 150 150 140 198 220 215 225 190 ...
## $ weight : int 3504 3693 3436 3433 3449 4341 4354 4312 4425 3850 ...
## $ acceleration: num 12 11.5 11 12 10.5 10 9 8.5 10 8.5 ...
## $ year : int 70 70 70 70 70 70 70 70 70 70 ...
## $ origin : int 1 1 1 1 1 1 1 1 1 1 ...
## $ mpg01 : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## - attr(*, "na.action")= 'omit' Named int [1:5] 33 127 331 337 355
## ..- attr(*, "names")= chr [1:5] "33" "127" "331" "337" ...(b) Support vector classifier (linear kernel) across a range of
cost
set.seed(1)
tune.linear <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "linear",
ranges = list(cost = c(0.01, 0.1, 1, 5, 10, 100)))
summary(tune.linear)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.08435897
##
## - Detailed performance results:
## cost error dispersion
## 1 1e-02 0.08923077 0.04698309
## 2 1e-01 0.09185897 0.04393409
## 3 1e+00 0.08435897 0.03662670
## 4 5e+00 0.08948718 0.03898410
## 5 1e+01 0.08948718 0.03898410
## 6 1e+02 0.08692308 0.03887151Cross-validation error is lowest around cost = 1
(roughly 8.4%), and stays in a fairly narrow band (8.4%–9.2%) across
most of the cost range tested — larger costs (10, 100) push the error
slightly higher, consistent with a more rigid,
lower-bias/higher-variance margin starting to overfit individual
training points.
(c) Radial and polynomial kernels, varying
gamma/degree/cost
set.seed(1)
tune.radial <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "radial",
ranges = list(cost = c(0.1, 1, 5, 10, 100),
gamma = c(0.01, 0.1, 1, 5)))
summary(tune.radial)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost gamma
## 1 1
##
## - best performance: 0.06634615
##
## - Detailed performance results:
## cost gamma error dispersion
## 1 0.1 0.01 0.11224359 0.03836937
## 2 1.0 0.01 0.08673077 0.04551036
## 3 5.0 0.01 0.08416667 0.04010502
## 4 10.0 0.01 0.08673077 0.04040897
## 5 100.0 0.01 0.08685897 0.03483004
## 6 0.1 0.10 0.08923077 0.04698309
## 7 1.0 0.10 0.08923077 0.04376306
## 8 5.0 0.10 0.07910256 0.03292568
## 9 10.0 0.10 0.08166667 0.04149504
## 10 100.0 0.10 0.08410256 0.03390616
## 11 0.1 1.00 0.08673077 0.04535158
## 12 1.0 1.00 0.06634615 0.03244101
## 13 5.0 1.00 0.08916667 0.02708952
## 14 10.0 1.00 0.08923077 0.02732003
## 15 100.0 1.00 0.10448718 0.04560852
## 16 0.1 5.00 0.54602564 0.05105434
## 17 1.0 5.00 0.09173077 0.03417795
## 18 5.0 5.00 0.09423077 0.04945093
## 19 10.0 5.00 0.09173077 0.04983028
## 20 100.0 5.00 0.09429487 0.05109336set.seed(1)
tune.poly <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "polynomial",
ranges = list(cost = c(0.1, 1, 5, 10, 100),
degree = c(2, 3, 4)))
summary(tune.poly)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost degree
## 100 3
##
## - best performance: 0.08423077
##
## - Detailed performance results:
## cost degree error dispersion
## 1 0.1 2 0.27846154 0.09486227
## 2 1.0 2 0.25307692 0.13751948
## 3 5.0 2 0.17621795 0.04945319
## 4 10.0 2 0.18647436 0.05598001
## 5 100.0 2 0.18128205 0.06251437
## 6 0.1 3 0.20192308 0.11347783
## 7 1.0 3 0.09448718 0.04180527
## 8 5.0 3 0.08429487 0.04016554
## 9 10.0 3 0.08435897 0.04544023
## 10 100.0 3 0.08423077 0.03636273
## 11 0.1 4 0.26564103 0.09977887
## 12 1.0 4 0.21205128 0.09560470
## 13 5.0 4 0.18371795 0.06175709
## 14 10.0 4 0.16589744 0.06962914
## 15 100.0 4 0.12756410 0.05208506data.frame(
Kernel = c("Linear", "Radial", "Polynomial"),
Best_Params = c(paste("cost =", tune.linear$best.parameters$cost),
paste0("cost = ", tune.radial$best.parameters$cost,
", gamma = ", tune.radial$best.parameters$gamma),
paste0("cost = ", tune.poly$best.parameters$cost,
", degree = ", tune.poly$best.parameters$degree)),
Best_CV_Error = c(tune.linear$best.performance,
tune.radial$best.performance,
tune.poly$best.performance)
)## Kernel Best_Params Best_CV_Error
## 1 Linear cost = 1 0.08435897
## 2 Radial cost = 1, gamma = 1 0.06634615
## 3 Polynomial cost = 100, degree = 3 0.08423077The radial-kernel SVM achieves the lowest cross-validation error of
the three (≈6.6%, at cost = 1, gamma = 1),
modestly outperforming the linear support vector classifier (≈8.4%). The
polynomial kernel needs a much larger cost (100) to reach a comparable
error rate (≈8.4%), and is generally more sensitive to the cost/degree
combination — low-cost polynomial fits perform poorly, suggesting the
polynomial kernel needs a lot of margin violations tolerated before it
fits any usable boundary for this feature space.
(d) Plots supporting parts (b) and (c)
The hint suggests using plot.svm(), which for
p > 2 predictors plots a 2-D slice using a chosen pair
of variables (all others held at their medians). We illustrate this
using weight and horsepower, two of the more
informative predictors of gas mileage.
svm.linear.best <- svm(mpg01 ~ ., data = Auto2, kernel = "linear",
cost = tune.linear$best.parameters$cost)
plot(svm.linear.best, Auto2, horsepower ~ weight)
svm.radial.best <- svm(mpg01 ~ ., data = Auto2, kernel = "radial",
cost = tune.radial$best.parameters$cost,
gamma = tune.radial$best.parameters$gamma)
plot(svm.radial.best, Auto2, horsepower ~ weight)
svm.poly.best <- svm(mpg01 ~ ., data = Auto2, kernel = "polynomial",
cost = tune.poly$best.parameters$cost,
degree = tune.poly$best.parameters$degree)
plot(svm.poly.best, Auto2, horsepower ~ weight)
The linear-kernel plot shows a single straight decision boundary through the weight/horsepower plane, while the radial and polynomial kernel plots both bend around the data, consistent with their ability to model non-linear relationships between engine characteristics and gas-mileage class. The radial kernel’s boundary is visibly smoother and hugs the data more naturally than the polynomial kernel’s, which tracks with the radial kernel’s lower cross-validation error above.
Exercise 8
This problem involves the OJ data set which is part
of the ISLR2 package.
OJ <- load_islr2_data("OJ") # ISLR2::OJ — 1,070 obs. of 18 variables
str(OJ)## 'data.frame': 1070 obs. of 18 variables:
## $ Purchase : Factor w/ 2 levels "CH","MM": 1 1 1 2 1 1 1 1 1 1 ...
## $ WeekofPurchase: num 237 239 245 227 228 230 232 234 235 238 ...
## $ StoreID : num 1 1 1 1 7 7 7 7 7 7 ...
## $ PriceCH : num 1.75 1.75 1.86 1.69 1.69 1.69 1.69 1.75 1.75 1.75 ...
## $ PriceMM : num 1.99 1.99 2.09 1.69 1.69 1.99 1.99 1.99 1.99 1.99 ...
## $ DiscCH : num 0 0 0.17 0 0 0 0 0 0 0 ...
## $ DiscMM : num 0 0.3 0 0 0 0 0.4 0.4 0.4 0.4 ...
## $ SpecialCH : num 0 0 0 0 0 0 1 1 0 0 ...
## $ SpecialMM : num 0 1 0 0 0 1 1 0 0 0 ...
## $ LoyalCH : num 0.5 0.6 0.68 0.4 0.957 ...
## $ SalePriceMM : num 1.99 1.69 2.09 1.69 1.69 1.99 1.59 1.59 1.59 1.59 ...
## $ SalePriceCH : num 1.75 1.75 1.69 1.69 1.69 1.69 1.69 1.75 1.75 1.75 ...
## $ PriceDiff : num 0.24 -0.06 0.4 0 0 0.3 -0.1 -0.16 -0.16 -0.16 ...
## $ Store7 : Factor w/ 2 levels "No","Yes": 1 1 1 1 2 2 2 2 2 2 ...
## $ PctDiscMM : num 0 0.151 0 0 0 ...
## $ PctDiscCH : num 0 0 0.0914 0 0 ...
## $ ListPriceDiff : num 0.24 0.24 0.23 0 0 0.3 0.3 0.24 0.24 0.24 ...
## $ STORE : num 1 1 1 1 0 0 0 0 0 0 ...(a) Training/test split (800 / 270)
set.seed(1)
train.idx <- sample(seq_len(nrow(OJ)), 800)
OJ.train <- OJ[train.idx, ]
OJ.test <- OJ[-train.idx, ]
dim(OJ.train); dim(OJ.test)## [1] 800 18## [1] 270 18(b) Support vector classifier, cost = 0.01
svm.linear <- svm(Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
summary(svm.linear)##
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 435
##
## ( 219 216 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMWith cost = 0.01 (a heavily penalized, very wide
margin), the classifier uses 435 of the 800 training observations as
support vectors — more than half the training set — which indicates a
wide margin with many points violating it, consistent with such a small
cost value.
(c) Training and test error rates
train.pred.lin <- predict(svm.linear, OJ.train)
test.pred.lin <- predict(svm.linear, OJ.test)
train.err.lin <- mean(train.pred.lin != OJ.train$Purchase)
test.err.lin <- mean(test.pred.lin != OJ.test$Purchase)
cat("Linear SVC (cost = 0.01)\n")## Linear SVC (cost = 0.01)cat(" Training error:", round(train.err.lin, 4), "\n")## Training error: 0.175cat(" Test error: ", round(test.err.lin, 4), "\n")## Test error: 0.1778(d) Tune cost via cross-validation
set.seed(1)
tune.linear.oj <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "linear",
ranges = list(cost = 10^seq(-2, 1, length = 10)))
summary(tune.linear.oj)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 0.4641589
##
## - best performance: 0.16875
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01000000 0.17625 0.02853482
## 2 0.02154435 0.17625 0.02972676
## 3 0.04641589 0.17500 0.02568506
## 4 0.10000000 0.17250 0.03162278
## 5 0.21544347 0.17250 0.02751262
## 6 0.46415888 0.16875 0.02651650
## 7 1.00000000 0.17500 0.02946278
## 8 2.15443469 0.17125 0.03064696
## 9 4.64158883 0.17125 0.03175973
## 10 10.00000000 0.17375 0.03197764best.cost.linear <- tune.linear.oj$best.parameters$cost
best.cost.linear## [1] 0.4641589(e) Train/test error using the tuned cost
svm.linear.best <- svm(Purchase ~ ., data = OJ.train, kernel = "linear",
cost = best.cost.linear)
train.pred.lin2 <- predict(svm.linear.best, OJ.train)
test.pred.lin2 <- predict(svm.linear.best, OJ.test)
train.err.lin2 <- mean(train.pred.lin2 != OJ.train$Purchase)
test.err.lin2 <- mean(test.pred.lin2 != OJ.test$Purchase)
cat("Linear SVC (tuned cost =", round(best.cost.linear, 4), ")\n")## Linear SVC (tuned cost = 0.4642 )cat(" Training error:", round(train.err.lin2, 4), "\n")## Training error: 0.165cat(" Test error: ", round(test.err.lin2, 4), "\n")## Test error: 0.1556Tuning cost improves both training and test error
relative to the arbitrary cost = 0.01 starting point — test
error drops from 0.1778 to 0.1556.
(f) Repeat (b)–(e) with a radial kernel (default
gamma)
# (b) baseline cost = 0.01
svm.radial <- svm(Purchase ~ ., data = OJ.train, kernel = "radial", cost = 0.01)
summary(svm.radial)##
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "radial", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 0.01
##
## Number of Support Vectors: 634
##
## ( 319 315 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MM# (c) baseline train/test error
train.pred.rad <- predict(svm.radial, OJ.train)
test.pred.rad <- predict(svm.radial, OJ.test)
train.err.rad <- mean(train.pred.rad != OJ.train$Purchase)
test.err.rad <- mean(test.pred.rad != OJ.test$Purchase)
cat("Radial SVM (cost = 0.01)\n")## Radial SVM (cost = 0.01)cat(" Training error:", round(train.err.rad, 4), "\n")## Training error: 0.3938cat(" Test error: ", round(test.err.rad, 4), "\n")## Test error: 0.3778# (d) tune cost (default gamma = 1/ncol(x))
set.seed(1)
tune.radial.oj <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "radial",
ranges = list(cost = 10^seq(-2, 1, length = 10)))
summary(tune.radial.oj)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 0.4641589
##
## - best performance: 0.17125
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01000000 0.39375 0.04007372
## 2 0.02154435 0.39375 0.04007372
## 3 0.04641589 0.20875 0.04041881
## 4 0.10000000 0.18625 0.02853482
## 5 0.21544347 0.18250 0.03016160
## 6 0.46415888 0.17125 0.02045490
## 7 1.00000000 0.17125 0.02128673
## 8 2.15443469 0.17875 0.02128673
## 9 4.64158883 0.18125 0.02144923
## 10 10.00000000 0.18625 0.02853482best.cost.radial <- tune.radial.oj$best.parameters$cost
# (e) train/test error at tuned cost
svm.radial.best <- svm(Purchase ~ ., data = OJ.train, kernel = "radial",
cost = best.cost.radial)
train.pred.rad2 <- predict(svm.radial.best, OJ.train)
test.pred.rad2 <- predict(svm.radial.best, OJ.test)
train.err.rad2 <- mean(train.pred.rad2 != OJ.train$Purchase)
test.err.rad2 <- mean(test.pred.rad2 != OJ.test$Purchase)
cat("Radial SVM (tuned cost =", round(best.cost.radial, 4), ")\n")## Radial SVM (tuned cost = 0.4642 )cat(" Training error:", round(train.err.rad2, 4), "\n")## Training error: 0.1475cat(" Test error: ", round(test.err.rad2, 4), "\n")## Test error: 0.1778(g) Repeat (b)–(e) with a polynomial kernel
(degree = 2)
# (b) baseline cost = 0.01
svm.poly <- svm(Purchase ~ ., data = OJ.train, kernel = "polynomial",
degree = 2, cost = 0.01)
summary(svm.poly)##
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "polynomial",
## degree = 2, cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: polynomial
## cost: 0.01
## degree: 2
## coef.0: 0
##
## Number of Support Vectors: 636
##
## ( 321 315 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MM# (c) baseline train/test error
train.pred.poly <- predict(svm.poly, OJ.train)
test.pred.poly <- predict(svm.poly, OJ.test)
train.err.poly <- mean(train.pred.poly != OJ.train$Purchase)
test.err.poly <- mean(test.pred.poly != OJ.test$Purchase)
cat("Polynomial SVM, degree=2 (cost = 0.01)\n")## Polynomial SVM, degree=2 (cost = 0.01)cat(" Training error:", round(train.err.poly, 4), "\n")## Training error: 0.3725cat(" Test error: ", round(test.err.poly, 4), "\n")## Test error: 0.3667# (d) tune cost
set.seed(1)
tune.poly.oj <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "polynomial",
degree = 2, ranges = list(cost = 10^seq(-2, 1, length = 10)))
summary(tune.poly.oj)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 2.154435
##
## - best performance: 0.1775
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01000000 0.39125 0.04210189
## 2 0.02154435 0.36625 0.03537988
## 3 0.04641589 0.35125 0.03839216
## 4 0.10000000 0.32125 0.05001736
## 5 0.21544347 0.21875 0.03830162
## 6 0.46415888 0.20625 0.04497299
## 7 1.00000000 0.20250 0.04116363
## 8 2.15443469 0.17750 0.04158325
## 9 4.64158883 0.18375 0.03387579
## 10 10.00000000 0.18125 0.02779513best.cost.poly <- tune.poly.oj$best.parameters$cost
# (e) train/test error at tuned cost
svm.poly.best <- svm(Purchase ~ ., data = OJ.train, kernel = "polynomial",
degree = 2, cost = best.cost.poly)
train.pred.poly2 <- predict(svm.poly.best, OJ.train)
test.pred.poly2 <- predict(svm.poly.best, OJ.test)
train.err.poly2 <- mean(train.pred.poly2 != OJ.train$Purchase)
test.err.poly2 <- mean(test.pred.poly2 != OJ.test$Purchase)
cat("Polynomial SVM, degree=2 (tuned cost =", round(best.cost.poly, 4), ")\n")## Polynomial SVM, degree=2 (tuned cost = 2.1544 )cat(" Training error:", round(train.err.poly2, 4), "\n")## Training error: 0.1588cat(" Test error: ", round(test.err.poly2, 4), "\n")## Test error: 0.2111(h) Overall comparison — which approach performs best?
results <- data.frame(
Kernel = rep(c("Linear", "Radial", "Polynomial (deg=2)"), each = 2),
Cost = c("0.01 (baseline)", round(best.cost.linear, 4),
"0.01 (baseline)", round(best.cost.radial, 4),
"0.01 (baseline)", round(best.cost.poly, 4)),
Training_Error = round(c(train.err.lin, train.err.lin2,
train.err.rad, train.err.rad2,
train.err.poly, train.err.poly2), 4),
Test_Error = round(c(test.err.lin, test.err.lin2,
test.err.rad, test.err.rad2,
test.err.poly, test.err.poly2), 4)
)
results## Kernel Cost Training_Error Test_Error
## 1 Linear 0.01 (baseline) 0.1750 0.1778
## 2 Linear 0.4642 0.1650 0.1556
## 3 Radial 0.01 (baseline) 0.3938 0.3778
## 4 Radial 0.4642 0.1475 0.1778
## 5 Polynomial (deg=2) 0.01 (baseline) 0.3725 0.3667
## 6 Polynomial (deg=2) 2.1544 0.1588 0.2111results_tuned <- results[c(2, 4, 6), ]
results_tuned$Kernel <- factor(results_tuned$Kernel, levels = results_tuned$Kernel)
ggplot(results_tuned, aes(x = Kernel, y = Test_Error, fill = Kernel)) +
geom_col(width = 0.55, show.legend = FALSE) +
geom_text(aes(label = Test_Error), vjust = -0.4) +
scale_fill_manual(values = c("#2E5EAA", "#C0392B", "#27AE60")) +
labs(title = "OJ Test Error by Kernel (Tuned Cost)",
y = "Test error rate", x = NULL) +
theme_minimal(base_size = 12)
All three tuned models land in a similar ballpark on test error
(0.1556–0.2111), which is typical for this data set — OJ
has a reasonably strong, mostly linear signal (driven largely by
LoyalCH, the customer’s brand-loyalty measure), so the
extra flexibility of radial or polynomial kernels buys little beyond
what a well-tuned linear support vector classifier already captures.
Based on the results above, the tuned linear support vector
classifier (or the tuned radial SVM, which performs comparably)
gives the best overall result on this data set — noting that the
polynomial kernel’s advantage over its own untuned baseline is the
largest of the three, but it still does not clearly surpass the linear
and radial models once all three are properly tuned.
Session Info
sessionInfo()## R version 4.5.1 (2025-06-13)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sequoia 15.0.1
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## time zone: America/Chicago
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ISLR2_1.3-2 ggplot2_4.0.0 e1071_1.7-16
##
## loaded via a namespace (and not attached):
## [1] vctrs_0.6.5 cli_3.6.5 knitr_1.50 rlang_1.1.6
## [5] xfun_0.53 generics_0.1.4 S7_0.2.0 jsonlite_2.0.0
## [9] labeling_0.4.3 glue_1.8.0 htmltools_0.5.8.1 sass_0.4.10
## [13] scales_1.4.0 rmarkdown_2.30 grid_4.5.1 tibble_3.3.0
## [17] evaluate_1.0.5 jquerylib_0.1.4 fastmap_1.2.0 yaml_2.3.10
## [21] lifecycle_1.0.4 compiler_4.5.1 dplyr_1.1.4 RColorBrewer_1.1-3
## [25] pkgconfig_2.0.3 rstudioapi_0.19.0 farver_2.1.2 digest_0.6.37
## [29] R6_2.6.1 tidyselect_1.2.1 class_7.3-23 dichromat_2.0-0.1
## [33] pillar_1.11.1 magrittr_2.0.4 bslib_0.9.0 withr_3.0.2
## [37] tools_4.5.1 proxy_0.4-27 gtable_0.3.6 cachem_1.1.0