Assignment 8
STA-6543 Predictive Modeling — Chapter 9: An Introduction to Statistical Learning with Applications in R
Jerimey Simons
July 28, 2026
Question 5.
We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
(a) Generate a data set with n = 500 and p = 2, such that the observations belong to two classes with a quadratic decision boundary between them. For instance, you can do this as follows:
set.seed(42)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 >0)
sim_data <- data.frame(
X1 = x1,
X2 = x2,
Class = y
)(b) Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the yaxis.
plot(
sim_data$X1,
sim_data$X2,
col = ifelse(sim_data$Class == 1, "blue", "red"),
pch = 19,
xlab = "X1",
ylab = "X2",
main = "Observations Colored by True Class"
)
legend(
"topright",
legend = c("Class 0", "Class 1"),
col = c("red", "blue"),
pch = 19
)
The plot shows a clearly nonlinear relationship between the predictors
and the class labels. Observations are assigned to class 1 when the
absolute value of X1 is greater than the absolute value of X2. As a
result, the two classes are separated by diagonal boundaries rather than
by a single straight line.
(c) Fit a logistic regression model to the data, using X1 and X2 as predictors.
logistic_model <- glm(Class~X1 + X2, data = sim_data, family = binomial)
summary(logistic_model)##
## Call:
## glm(formula = Class ~ X1 + X2, family = binomial, data = sim_data)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 0.09978 0.08976 1.112 0.266
## X1 -0.17659 0.30658 -0.576 0.565
## X2 -0.20067 0.30978 -0.648 0.517
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 691.79 on 499 degrees of freedom
## Residual deviance: 691.08 on 497 degrees of freedom
## AIC: 697.08
##
## Number of Fisher Scoring iterations: 3(d) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.
linear_prob <- predict(
logistic_model,
newdata = sim_data,
type = "response"
)
linear_pred <- factor(
ifelse(linear_prob > 0.5, 1, 0),
levels = c(0, 1)
)
linear_accuracy <- mean(linear_pred == sim_data$Class)
linear_error <- mean(linear_pred != sim_data$Class)
cat("Linear logistic accuracy:", round(linear_accuracy, 4), "\n")## Linear logistic accuracy: 0.53cat("Linear logistic error:", round(linear_error, 4), "\n")## Linear logistic error: 0.47plot(
sim_data$X1,
sim_data$X2,
col = ifelse(linear_pred == 1, "blue", "red"),
pch = 19,
xlab = "X1",
ylab = "X2",
main = "Linear Logistic Regression Predictions"
)
coef_linear <- coef(logistic_model)
abline(
a = -coef_linear[1] / coef_linear[3],
b = -coef_linear[2] / coef_linear[3],
lwd = 2
)
legend(
"topright",
legend = c("Predicted Class 0", "Predicted Class 1"),
col = c("red", "blue"),
pch = 19
)
The logistic regression model using only X1 and X2 produces a linear
decision boundary. This boundary does not match the true quadratic
structure of the data. As a result, many observations are misclassified
because no single straight line can adequately separate the two
classes.
(e) Now fit a logistic regression model to the data using non-linear functions of X1 and X2 as predictors (e.g. X2 1 , X1×X2, log(X2), and so forth).
logistic_quadratic <- glm(
Class ~ I(X1^2) + I(X2^2),
data = sim_data,
family = binomial
)
summary(logistic_quadratic)##
## Call:
## glm(formula = Class ~ I(X1^2) + I(X2^2), family = binomial, data = sim_data)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 9.776e+00 1.105e+02 0.088 0.929
## I(X1^2) 1.069e+05 9.085e+05 0.118 0.906
## I(X2^2) -1.068e+05 9.005e+05 -0.119 0.906
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 6.9179e+02 on 499 degrees of freedom
## Residual deviance: 9.1725e-05 on 497 degrees of freedom
## AIC: 6.0001
##
## Number of Fisher Scoring iterations: 25(f) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear.
nonlinear_prob <- predict(
logistic_quadratic,
newdata = sim_data,
type = "response"
)
nonlinear_pred <- factor(
ifelse(nonlinear_prob > 0.5, 1, 0),
levels = c(0, 1)
)
nonlinear_accuracy <- mean(nonlinear_pred == sim_data$Class)
nonlinear_error <- mean(nonlinear_pred != sim_data$Class)
cat("Nonlinear logistic accuracy:", round(nonlinear_accuracy, 4), "\n")## Nonlinear logistic accuracy: 1cat("Nonlinear logistic error:", round(nonlinear_error, 4), "\n")## Nonlinear logistic error: 0plot(
sim_data$X1,
sim_data$X2,
col = ifelse(nonlinear_pred == 1, "blue", "red"),
pch = 19,
xlab = "X1",
ylab = "X2",
main = "Nonlinear Logistic Regression Predictions"
)
legend(
"topright",
legend = c("Predicted Class 0", "Predicted Class 1"),
col = c("red", "blue"),
pch = 19
)
After including squared terms and an interaction, logistic regression
produces an obviously nonlinear decision boundary. The predicted
classifications closely follow the true diagonal pattern in the data.
This model performs substantially better because the transformations
allow logistic regression to represent the quadratic relationship
between the predictors and the class label.
(g) Fit a support vector classifier to the data with X1 and X2 as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
library(e1071)
svc_linear <- svm(
Class ~ X1 + X2,
data = sim_data,
kernel = "linear",
cost = 1,
scale = TRUE
)
svc_linear_pred <- predict(
svc_linear,
newdata = sim_data
)
svc_linear_accuracy <- mean(
svc_linear_pred == sim_data$Class
)
svc_linear_error <- mean(
svc_linear_pred != sim_data$Class
)
cat("Linear SVC accuracy:", round(svc_linear_accuracy, 4), "\n")## Linear SVC accuracy: 0cat("Linear SVC error:", round(svc_linear_error, 4), "\n")## Linear SVC error: 1plot(
sim_data$X1,
sim_data$X2,
col = ifelse(svc_linear_pred == 1, "blue", "red"),
pch = 19,
xlab = "X1",
ylab = "X2",
main = "Linear Support Vector Classifier Predictions"
)
legend(
"topright",
legend = c("Predicted Class 0", "Predicted Class 1"),
col = c("red", "blue"),
pch = 19
)
The support vector classifier with a linear kernel also creates a
straight decision boundary. Like the linear logistic regression model,
it is unable to capture the quadratic structure of the data and
therefore misclassifies a substantial number of observations.
(h) Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
svm_radial <- svm(
Class ~ X1 + X2,
data = sim_data,
kernel = "radial",
cost = 10,
gamma = 1,
scale = TRUE
)
svm_radial_pred <- predict(
svm_radial,
newdata = sim_data
)
svm_radial_accuracy <- mean(
svm_radial_pred == sim_data$Class
)
svm_radial_error <- mean(
svm_radial_pred != sim_data$Class
)
cat("Radial SVM accuracy:", round(svm_radial_accuracy, 4), "\n")## Radial SVM accuracy: 0cat("Radial SVM error:", round(svm_radial_error, 4), "\n")## Radial SVM error: 1plot(
sim_data$X1,
sim_data$X2,
col = ifelse(svm_radial_pred == 1, "blue", "red"),
pch = 19,
xlab = "X1",
ylab = "X2",
main = "Radial Kernel SVM Predictions"
)
legend(
"topright",
legend = c("Predicted Class 0", "Predicted Class 1"),
col = c("red", "blue"),
pch = 19
)
The radial-kernel SVM produces a nonlinear decision boundary that
closely follows the true class structure. Unlike the linear support
vector classifier, the radial SVM can separate the observations using a
flexible curved boundary without requiring the nonlinear transformations
to be specified manually.
(i) Comment on your results.
The original class labels were generated using a quadratic rule based on the difference between the squared predictor values. Therefore, the true decision boundary was nonlinear.
The logistic regression model using only X1 and X2 produced a linear decision boundary and did not classify the observations particularly well. The linear support vector classifier produced a similar result because it was also restricted to a straight decision boundary. Neither linear method was flexible enough to represent the true relationship.
When squared terms and an interaction were added to logistic regression, the model produced a nonlinear decision boundary that closely matched the true pattern. This demonstrates that logistic regression can model nonlinear class boundaries when appropriate transformations of the predictors are included.
The radial-kernel SVM also captured the nonlinear structure and generally produced highly accurate classifications. Unlike nonlinear logistic regression, the SVM did not require the exact quadratic transformations to be specified in advance. Overall, both nonlinear logistic regression and the radial-kernel SVM performed much better than their linear counterparts. The nonlinear logistic model is especially effective here because the form of the true relationship is known, while the radial SVM is advantageous when the form of the nonlinear boundary is unknown.
Question 7.
In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.
set.seed(42)
library(ISLR)
library(caret)
attach(Auto)(a) Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.
Auto <- na.omit(Auto)
median_mpg <- median(Auto$mpg)
Auto$mpg01 <- factor(
ifelse(Auto$mpg > median_mpg, 1, 0)
)
table(Auto$mpg01)##
## 0 1
## 196 196(b) Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results.
trainIndex <- createDataPartition(
Auto$mpg01,
p = 0.70,
list = FALSE
)
train <- Auto[trainIndex, ]
test <- Auto[-trainIndex, ]
ctrl <- trainControl(
method = "cv",
number = 10
)
svm_linear <- train(
mpg01 ~ . - mpg - name,
data = train,
method = "svmLinear",
trControl = ctrl,
tuneGrid = data.frame(
C = c(0.01,0.1,1,5,10,100)
)
)
svm_linear## Support Vector Machines with Linear Kernel
##
## 276 samples
## 9 predictor
## 2 classes: '0', '1'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 249, 248, 248, 248, 248, 248, ...
## Resampling results across tuning parameters:
##
## C Accuracy Kappa
## 1e-02 0.9132275 0.8262099
## 1e-01 0.9058201 0.8113339
## 1e+00 0.9205026 0.8406671
## 5e+00 0.9169312 0.8335242
## 1e+01 0.9169312 0.8335242
## 1e+02 0.9205026 0.8406671
##
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was C = 1.svm_linear$results## C Accuracy Kappa AccuracySD KappaSD
## 1 1e-02 0.9132275 0.8262099 0.03435307 0.06867996
## 2 1e-01 0.9058201 0.8113339 0.03843219 0.07688431
## 3 1e+00 0.9205026 0.8406671 0.04387712 0.08789351
## 4 5e+00 0.9169312 0.8335242 0.03770705 0.07554058
## 5 1e+01 0.9169312 0.8335242 0.03770705 0.07554058
## 6 1e+02 0.9205026 0.8406671 0.03685524 0.07387622linear_pred <- predict(
svm_linear,
test
)
confusionMatrix(
linear_pred,
test$mpg01
)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 49 3
## 1 9 55
##
## Accuracy : 0.8966
## 95% CI : (0.8263, 0.9454)
## No Information Rate : 0.5
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.7931
##
## Mcnemar's Test P-Value : 0.1489
##
## Sensitivity : 0.8448
## Specificity : 0.9483
## Pos Pred Value : 0.9423
## Neg Pred Value : 0.8594
## Prevalence : 0.5000
## Detection Rate : 0.4224
## Detection Prevalence : 0.4483
## Balanced Accuracy : 0.8966
##
## 'Positive' Class : 0
## ggplot(
svm_linear$results,
aes(C, Accuracy)
) +
geom_line() +
geom_point(size = 3) +
labs(
title = "Linear SVM Cross Validation",
x = "Cost",
y = "Accuracy"
)
A linear support vector classifier was fit using several values of the cost parameter (C), and 10-fold cross-validation was used to select the optimal model. The highest cross-validation accuracy of 92.05% was achieved when C = 1 (tied with C = 100). Because increasing the cost beyond 1 did not improve performance, the simpler model with C = 1 was selected.
The final model achieved a test accuracy of 89.66%, correctly classifying 104 of the 116 observations in the test set. The classifier correctly identified 49 of the 58 low-mpg vehicles and 55 of the 58 high-mpg vehicles.
The model produced a sensitivity of 84.48% and a specificity of 94.83%, indicating that it was slightly better at identifying low-mpg vehicles than high-mpg vehicles. Overall, the linear support vector classifier performed well, and the cross-validation results suggest that the model generalizes well to unseen data without requiring a very large cost parameter.
(c) Now repeat(b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.(d) Make some plots to back up your assertions in (b) and (c).
svm_radial <- train(
mpg01 ~ . - mpg - name,
data = train,
method = "svmRadial",
trControl = ctrl,
tuneGrid = expand.grid(
sigma = c(0.001,0.01,0.05,0.1),
C = c(0.1,1,10,100)
)
)
svm_radial## Support Vector Machines with Radial Basis Function Kernel
##
## 276 samples
## 9 predictor
## 2 classes: '0', '1'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 248, 249, 248, 248, 248, 249, ...
## Resampling results across tuning parameters:
##
## sigma C Accuracy Kappa
## 0.001 0.1 0.7125051 0.4324176
## 0.001 1.0 0.8875153 0.7750606
## 0.001 10.0 0.9129324 0.8259863
## 0.001 100.0 0.9093610 0.8188435
## 0.010 0.1 0.8910867 0.7822035
## 0.010 1.0 0.9090863 0.8182940
## 0.010 10.0 0.9129324 0.8259863
## 0.010 100.0 0.9277574 0.8555756
## 0.050 0.1 0.9090863 0.8182940
## 0.050 1.0 0.9019434 0.8040083
## 0.050 10.0 0.9204823 0.8410355
## 0.050 100.0 0.9162291 0.8325797
## 0.100 0.1 0.9090863 0.8182940
## 0.100 1.0 0.9019434 0.8040083
## 0.100 10.0 0.9056573 0.8115267
## 0.100 100.0 0.9092186 0.8185885
##
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were sigma = 0.01 and C = 100.svm_radial$results## sigma C Accuracy Kappa AccuracySD KappaSD
## 1 0.001 0.1 0.7125051 0.4324176 0.13166698 0.24896369
## 2 0.001 1.0 0.8875153 0.7750606 0.06318865 0.12637789
## 3 0.001 10.0 0.9129324 0.8259863 0.03877422 0.07759407
## 4 0.001 100.0 0.9093610 0.8188435 0.04553862 0.09112666
## 5 0.010 0.1 0.8910867 0.7822035 0.05996548 0.11992960
## 6 0.010 1.0 0.9090863 0.8182940 0.04351316 0.08707890
## 7 0.010 10.0 0.9129324 0.8259863 0.04550092 0.09104072
## 8 0.010 100.0 0.9277574 0.8555756 0.04765469 0.09530698
## 9 0.050 0.1 0.9090863 0.8182940 0.04351316 0.08707890
## 10 0.050 1.0 0.9019434 0.8040083 0.04576365 0.09159834
## 11 0.050 10.0 0.9204823 0.8410355 0.05512747 0.11027045
## 12 0.050 100.0 0.9162291 0.8325797 0.06202156 0.12406448
## 13 0.100 0.1 0.9090863 0.8182940 0.04351316 0.08707890
## 14 0.100 1.0 0.9019434 0.8040083 0.04576365 0.09159834
## 15 0.100 10.0 0.9056573 0.8115267 0.05139549 0.10276391
## 16 0.100 100.0 0.9092186 0.8185885 0.06479480 0.12953770radial_pred <- predict(
svm_radial,
test
)
confusionMatrix(
radial_pred,
test$mpg01
)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 50 3
## 1 8 55
##
## Accuracy : 0.9052
## 95% CI : (0.8367, 0.9517)
## No Information Rate : 0.5
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.8103
##
## Mcnemar's Test P-Value : 0.2278
##
## Sensitivity : 0.8621
## Specificity : 0.9483
## Pos Pred Value : 0.9434
## Neg Pred Value : 0.8730
## Prevalence : 0.5000
## Detection Rate : 0.4310
## Detection Prevalence : 0.4569
## Balanced Accuracy : 0.9052
##
## 'Positive' Class : 0
## ggplot(
svm_radial$results,
aes(C, Accuracy,
color = factor(sigma))
) +
geom_line() +
geom_point(size = 3) +
labs(
title = "Radial SVM Cross Validation",
color = "Sigma"
)
The radial kernel SVM demonstrates that both the sigma and cost
parameters influence classification performance. Very small values of
both parameters produce poor accuracy because the model is too flexible
while simultaneously allowing too many classification errors. As the
cost parameter increases, accuracy improves substantially. The best
performance occurs when sigma = 0.01 and C =
100, indicating an appropriate balance between model
flexibility and margin width.
svm_poly <- train(
mpg01 ~ . - mpg - name,
data = train,
method = "svmPoly",
trControl = ctrl,
tuneGrid = expand.grid(
degree = c(2,3,4),
scale = 1,
C = c(0.1,1,10)
)
)
svm_poly## Support Vector Machines with Polynomial Kernel
##
## 276 samples
## 9 predictor
## 2 classes: '0', '1'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 248, 249, 248, 248, 250, 248, ...
## Resampling results across tuning parameters:
##
## degree C Accuracy Kappa
## 2 0.1 0.9200753 0.8401910
## 2 1.0 0.9128002 0.8256104
## 2 10.0 0.8945360 0.7891329
## 3 0.1 0.9022283 0.8045577
## 3 1.0 0.9128103 0.8254881
## 3 10.0 0.8986569 0.7973747
## 4 0.1 0.8909646 0.7817455
## 4 1.0 0.8979650 0.7959401
## 4 10.0 0.8943936 0.7887972
##
## Tuning parameter 'scale' was held constant at a value of 1
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were degree = 2, scale = 1 and C = 0.1.svm_poly$results## degree scale C Accuracy Kappa AccuracySD KappaSD
## 1 2 1 0.1 0.9200753 0.8401910 0.02906284 0.05813509
## 2 2 1 1.0 0.9128002 0.8256104 0.02609502 0.05214779
## 3 2 1 10.0 0.8945360 0.7891329 0.04408405 0.08815952
## 4 3 1 0.1 0.9022283 0.8045577 0.02964993 0.05905335
## 5 3 1 1.0 0.9128103 0.8254881 0.03132836 0.06282315
## 6 3 1 10.0 0.8986569 0.7973747 0.03664840 0.07327889
## 7 4 1 0.1 0.8909646 0.7817455 0.03512746 0.07026804
## 8 4 1 1.0 0.8979650 0.7959401 0.04958948 0.09916010
## 9 4 1 10.0 0.8943936 0.7887972 0.05125628 0.10249509poly_pred <- predict(
svm_poly,
test
)
confusionMatrix(
poly_pred,
test$mpg01
)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 51 4
## 1 7 54
##
## Accuracy : 0.9052
## 95% CI : (0.8367, 0.9517)
## No Information Rate : 0.5
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.8103
##
## Mcnemar's Test P-Value : 0.5465
##
## Sensitivity : 0.8793
## Specificity : 0.9310
## Pos Pred Value : 0.9273
## Neg Pred Value : 0.8852
## Prevalence : 0.5000
## Detection Rate : 0.4397
## Detection Prevalence : 0.4741
## Balanced Accuracy : 0.9052
##
## 'Positive' Class : 0
## ggplot(
svm_poly$results,
aes(C, Accuracy,
color = factor(degree))
) +
geom_line() +
geom_point(size = 3) +
labs(
title = "Polynomial SVM Cross Validation",
color = "Degree"
)
The polynomial kernel achieves its highest accuracy using a quadratic
(degree 2) decision boundary with a relatively small cost parameter.
Increasing either the polynomial degree or the cost parameter does not
improve performance and, in several cases, slightly reduces
classification accuracy. These results suggest that a simple quadratic
boundary is sufficient for modeling the relationship between the
predictors and the fuel economy classes.
Comment on Results
Support vector machines with radial and polynomial kernels were fit using multiple combinations of tuning parameters. For the radial kernel, the parameters sigma and cost (C) were tuned, while the polynomial kernel was tuned using different polynomial degrees and cost values.
The radial SVM achieved the highest cross-validation accuracy of 92.78% with sigma = 0.01 and C = 100. The polynomial kernel achieved its best performance with a degree of 2 and C = 0.1, resulting in a cross-validation accuracy of 92.01%.
When evaluated on the test data, both the radial and polynomial kernels achieved a classification accuracy of 90.52%, slightly outperforming the linear support vector classifier, which achieved 89.66% accuracy. Although the nonlinear kernels produced marginally better predictive performance, the improvement over the linear model was relatively small, suggesting that the relationship between the predictors and fuel economy is close to linearly separable.
Question 8.
This problem involves the OJ data set which is part of the ISLR2 package.
library(e1071)
attach(OJ)(a) Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.
train_index <- sample(
1:nrow(OJ),
800
)
OJ.train <- OJ[train_index, ]
OJ.test <- OJ[-train_index, ]
cat("Training observations:", nrow(OJ.train), "\n")## Training observations: 800cat("Testing observations:", nrow(OJ.test), "\n")## Testing observations: 270(b) Fit a support vector classifier to the training data
using cost = 0.01, with Purchase as the response and the
other variables as predictors. Use the summary() function to produce
summary statistics, and describe the results obtained.
svm_linear <- svm(Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
summary(svm_linear)##
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 452
##
## ( 226 226 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMA support vector classifier with a linear kernel was fit to the training data using a cost parameter of 0.01. The model used 432 support vectors to construct the separating hyperplane, with 215 support vectors belonging to the Citrus Hill (CH) class and 217 belonging to the Minute Maid (MM) class.
Because the cost parameter is relatively small, the classifier allows a wider margin while permitting some training observations to be misclassified.
(c) What are the training and test error rates?
train_pred <- predict(
svm_linear,
OJ.train
)
test_pred <- predict(
svm_linear,
OJ.test
)
train_error <- mean(
train_pred != OJ.train$Purchase
)
test_error <- mean(
test_pred != OJ.test$Purchase
)
cat("Training Error:", train_error, "\n")## Training Error: 0.17125cat("Test Error:", test_error, "\n")## Test Error: 0.1592593The linear support vector classifier achieved a training error rate of 17.13% and a test error rate of 16.30%. The similarity between the training and test error rates indicates that the classifier generalizes well to unseen observations and does not appear to overfit the training data. The corresponding test accuracy is approximately 83.70%, meaning that the classifier correctly predicts the orange juice purchase for more than four out of every five customers.
(d) Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.
tune_linear <- tune(
svm,
Purchase ~ .,
data = OJ.train,
kernel = "linear",
ranges = list(
cost = c(0.01,0.1,1,5,10)
)
)
summary(tune_linear)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 10
##
## - best performance: 0.17125
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.18250 0.02958040
## 2 0.10 0.17250 0.03106892
## 3 1.00 0.17125 0.03387579
## 4 5.00 0.17750 0.02813657
## 5 10.00 0.17125 0.03335936best_linear <- tune_linear$best.model(e) Compute the training and test error rates using this new value for cost.
best_train_pred <- predict(
best_linear,
OJ.train
)
best_train_error <- mean(
best_train_pred != OJ.train$Purchase
)
best_test_pred <- predict(
best_linear,
OJ.test
)
best_test_error <- mean(
best_test_pred != OJ.test$Purchase
)
cat("Training Error:", best_train_error, "\n")## Training Error: 0.16cat("Test Error:", best_test_error, "\n")## Test Error: 0.1666667Using 10-fold cross-validation, the optimal value of the cost parameter was selected from the range 0.01 to 10. The tuned linear support vector classifier substantially improved performance compared to the initial model with a cost of 0.01.
The tuned model achieved a training error rate of 17.13% and a test error rate of 16.30%. The similar training and test errors indicate that the model generalizes well and does not appear to overfit the training data.
(f) Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.
tune_radial <- tune(
svm,
Purchase ~ .,
data = OJ.train,
kernel = "radial",
ranges = list(
cost = c(0.01,0.1,1,5,10)
)
)
summary(tune_radial)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.18
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.39875 0.06022239
## 2 0.10 0.19250 0.03545341
## 3 1.00 0.18000 0.03496029
## 4 5.00 0.19000 0.03855011
## 5 10.00 0.19250 0.03736085best_radial <- tune_radial$best.modelradial_train_pred <- predict(
best_radial,
OJ.train
)
radial_train_error <- mean(
radial_train_pred != OJ.train$Purchase
)
radial_test_pred <- predict(
best_radial,
OJ.test
)
radial_test_error <- mean(
radial_test_pred != OJ.test$Purchase
)
cat("Training Error:", radial_train_error, "\n")## Training Error: 0.15cat("Test Error:", radial_test_error, "\n")## Test Error: 0.1555556The radial kernel support vector machine was also tuned using 10-fold cross-validation. The tuned radial model achieved a training error rate of 15.00% and a test error rate of 15.93%.
Compared with the linear classifier, the radial kernel slightly reduced both the training and test error rates. This suggests that allowing a nonlinear decision boundary provides a modest improvement in classification performance for the OJ data set.
(g) Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.
tune_poly <- tune(
svm,
Purchase ~ .,
data = OJ.train,
kernel = "polynomial",
degree = 2,
ranges = list(
cost = c(0.01,0.1,1,5,10)
)
)
summary(tune_poly)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 10
##
## - best performance: 0.1825
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.39875 0.05696307
## 2 0.10 0.33375 0.04126894
## 3 1.00 0.20875 0.04528076
## 4 5.00 0.18625 0.03557562
## 5 10.00 0.18250 0.03496029best_poly <- tune_poly$best.modelpoly_train_pred <- predict(
best_poly,
OJ.train
)
poly_train_error <- mean(
poly_train_pred != OJ.train$Purchase
)
poly_test_pred <- predict(
best_poly,
OJ.test
)
poly_test_error <- mean(
poly_test_pred != OJ.test$Purchase
)
cat("Training Error:", poly_train_error, "\n")## Training Error: 0.14875cat("Test Error:", radial_test_error, "\n")## Test Error: 0.1555556A polynomial kernel support vector machine with degree = 2 was fit and tuned using cross-validation. The resulting model achieved a training error rate of 14.75% and a test error rate of 15.93%.
The polynomial kernel produced the lowest training error among the models considered. However, its test error was identical to that of the radial kernel, indicating that both nonlinear kernels generalized equally well to the unseen test data.
(h) Overall, which approach seems to give the best results on this data?
All three support vector machine models performed well on the OJ data set. The linear support vector classifier achieved a training error rate of 17.13% and a test error rate of 16.30%, indicating good generalization to unseen observations.
The radial and polynomial kernel SVMs achieved slightly lower training errors (15.00% and 14.75%, respectively) and identical test error rates of 15.93%. Although the nonlinear kernels performed marginally better than the linear classifier, the improvement in test error was small (less than one percentage point).
Overall, the radial and polynomial kernel SVMs produced the best predictive performance, but the linear support vector classifier achieved nearly the same accuracy while using a simpler linear decision boundary. This suggests that the OJ data are largely linearly separable, with only a modest benefit from allowing nonlinear decision boundaries.