Assignment8

Problem 5

We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.

(a) Generate a data set with n = 500 and p = 2, such that the observations belong to two classes with a quadratic decision boundary between them.

set.seed(1)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y  <- 1 * (x1^2 - x2^2 > 0)
data <- data.frame(x1 = x1, x2 = x2, y = as.factor(y))

(b) Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the y-axis.

plot(x1, x2, col = ifelse(y == 1, "red", "blue"), pch = 19,
     xlab = "X1", ylab = "X2", main = "True Class Labels")

(c) Fit a logistic regression model to the data, using X1 and X2 as predictors.

glm_fit <- glm(y ~ x1 + x2, data = data, family = binomial)
summary(glm_fit)

Call:
glm(formula = y ~ x1 + x2, family = binomial, data = data)

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.087260   0.089579  -0.974    0.330
x1           0.196199   0.316864   0.619    0.536
x2          -0.002854   0.305712  -0.009    0.993

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 692.18  on 499  degrees of freedom
Residual deviance: 691.79  on 497  degrees of freedom
AIC: 697.79

Number of Fisher Scoring iterations: 3

(d) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.

glm_probs <- predict(glm_fit, data, type = "response")
glm_pred  <- ifelse(glm_probs > 0.5, 1, 0)

plot(x1, x2, col = ifelse(glm_pred == 1, "red", "blue"), pch = 19,
     xlab = "X1", ylab = "X2", main = "Logistic Regression (Linear Terms)")

(e) Now fit a logistic regression model to the data using non-linear functions of X1 and X2 as predictors (e.g. X1², X1×X2, log(X2), and so forth).

glm_fit2 <- glm(y ~ poly(x1, 2) + poly(x2, 2) + x1:x2, data = data, family = binomial)
summary(glm_fit2)

Call:
glm(formula = y ~ poly(x1, 2) + poly(x2, 2) + x1:x2, family = binomial, 
    data = data)

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept)    -102.2     4302.0  -0.024    0.981
poly(x1, 2)1   2715.3   141109.5   0.019    0.985
poly(x1, 2)2  27218.5   842987.2   0.032    0.974
poly(x2, 2)1   -279.7    97160.4  -0.003    0.998
poly(x2, 2)2 -28693.0   875451.3  -0.033    0.974
x1:x2          -206.4    41802.8  -0.005    0.996

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 6.9218e+02  on 499  degrees of freedom
Residual deviance: 3.5810e-06  on 494  degrees of freedom
AIC: 12

Number of Fisher Scoring iterations: 25

(f) Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear.

glm_probs2 <- predict(glm_fit2, data, type = "response")
glm_pred2  <- ifelse(glm_probs2 > 0.5, 1, 0)

plot(x1, x2, col = ifelse(glm_pred2 == 1, "red", "blue"), pch = 19,
     xlab = "X1", ylab = "X2", main = "Logistic Regression (Quadratic Terms)")

(g) Fit a support vector classifier to the data with X1 and X2 as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

library(caret)

train_control <- trainControl(method = "cv", number = 10)
svm_linear <- train(y~x1+x2, data = data, method = "svmLinear", trControl = train_control, 
                    preProcess =   c("center","scale"))
svm_linear_pred <- predict(svm_linear, data)

plot(x1, x2, col = ifelse(svm_linear_pred == 1, "red", "blue"), pch = 19,
     xlab = "X1", ylab = "X2", main = "Support Vector Classifier (Linear)")

• The linear support vector classifier predicted every observation as the same class. This suggests that a linear decision boundary is not flexible enough to capture the non-linear relationship in the data.

(h) Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

train_control <- trainControl(method = "cv", number = 10)
svm_radial <- train(y~x1+x2, data = data, method = "svmRadial", trControl = train_control, 
                    preProcess = c("center", "scale"))
svm_radial_pred <- predict(svm_radial, data)

plot(x1, x2, col = ifelse(svm_radial_pred == 1, "red", "blue"), pch = 19,
     xlab = "X1", ylab = "X2", main = "Support Vector Machine (Radial Kernel)")

• The SVM with a radial kernel correctly classified nearly all of the observations, indicating that the non-linear kernel was able to model the underlying relationship between x1, x2, and y.

(i) Comment on your results.

• This exercise shows that both logistic regression and SVMs can model non-linear decision boundaries when the appropriate transformations or kernels are used. The linear models were unable to separate the classes, while the non-linear logistic regression and radial SVM provided a much better fit to the data.

---

Problem 7

In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.

(a) Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.

library(ISLR2)

Auto$mpg01 <- as.factor(ifelse(Auto$mpg > median(Auto$mpg), 1, 0))

(b) Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results.

library(e1071)

set.seed(1)
Auto_svm <- Auto[, !(names(Auto) %in% c("mpg", "name"))]

train_control = trainControl(method = "cv", number = 10)
tune_linear <- train(mpg01~.,data=Auto_svm,method='svmLinear',trControl=train_control
                     ,preProcess=c('center','scale'),tuneGrid=expand.grid(C=c(0.01, 0.1, 1, 5, 10, 100)))
tune_linear$results

• The cross-validation error ranged from approximately 8.7% to 9.5% across the tested cost values. The lowest error occurred when C = 10, but the differences between the models were minimal, indicating that changing the cost parameter had little effect on classification performance.

(c) Now repeat (b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.

set.seed(1)
train_control = trainControl(method = "cv", number = 10)
tune_radial <- train(mpg01~.,data=Auto_svm,method='svmRadial',trControl=train_control
                     ,preProcess=c('center','scale'),tuneGrid=
                       expand.grid(C = c(0.1, 1, 5, 10, 100),sigma = c(0.01, 0.1, 1, 5)))

tune_radial$results

• The radial SVM produced cross-validation accuracies between approximately 89.0% and 91.3%. The best performance occurred when sigma = 0.10 and C = 5 or 10, with an accuracy of about 91.3%. Compared with the linear SVM, the radial kernel showed only a slight improvement, suggesting that both models performed similarly on this dataset.

set.seed(1)
train_control = trainControl(method = "cv", number = 10)
tune_poly <- train(mpg01~.,data=Auto_svm,method='svmPoly',trControl=train_control
                     ,preProcess=c('center','scale'),tuneGrid=
                      expand.grid(C = c(0.1, 1, 5, 10, 100), degree = c(2, 3, 4), scale = c(0.1,.1)))
tune_poly$results

• The polynomial SVM achieved a maximum cross-validation accuracy of approximately 91.1% with a degree of 3 and C = 5. Its performance was very similar to the linear and radial SVMs, indicating that the choice of kernel had only a modest impact on prediction accuracy for this dataset.

(d) Make some plots to back up your assertions in (b) and (c).

svm_best_linear <- svm(mpg01 ~ ., data = Auto_svm, kernel = "linear",
                       cost = tune_linear$bestTune)
svm_best_radial <- svm(mpg01 ~ ., data = Auto_svm, kernel = "radial",
                       cost = tune_radial$bestTune,
                       sigma = tune_radial$bestTune)

plot(svm_best_linear, Auto_svm, horsepower ~ weight)

plot(svm_best_radial, Auto_svm, horsepower ~ weight)

plot(svm_best_linear, Auto_svm, displacement ~ acceleration)

• These pairwise plots show the classification results from the SVM model using horsepower vs. weight and displacement vs. acceleration. The variables provide a reasonable separation between the two mpg classes, as heavier and more powerful vehicles tend to fall into the low-mpg category, while lighter vehicles tend to fall into the high-mpg category.

---

Problem 8

This problem involves the OJ data set which is part of the ISLR2 package.

(a) Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

set.seed(1)
train_oj <- sample(1:nrow(OJ), 800)
oj_train <- OJ[train_oj, ]
oj_test  <- OJ[-train_oj, ]

(b) Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.

svm_oj_linear <- train(Purchase~.,data = oj_train, method = "svmLinear", 
                       trControl = train_control, preProcess = c("center", "scale"), tuneGrid = expand.grid(C=0.01))
svm_oj_linear$results

• With cost = 0.01, the SVM uses a large number of support vectors. This is expected because a small cost value creates a wider margin and allows more observations to contribute to defining the decision boundary.

(c) What are the training and test error rates?

train_pred_linear <- predict(svm_oj_linear, oj_train)
test_pred_linear   <- predict(svm_oj_linear, oj_test)

mean(train_pred_linear != oj_train$Purchase)

[1] 0.17625

mean(test_pred_linear != oj_test$Purchase)

[1] 0.1851852

• Both the training and test error rates are about 17-18%, showing us the classifier generalizes well and isn’t overfitting despite the low cost value.

(d) Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.

set.seed(1)
tune_oj_linear <- train(Purchase~., data = oj_train, method = "svmLinear", trControl = train_control,
                        preProcess = c("center", "scale"), 
                        tuneGrid = expand.grid(C = seq(0.01, 10, length.out = 10)))
tune_oj_linear$results

tune_oj_linear$bestTune

(e) Compute the training and test error rates using this new value for cost.

train_pred_linear <- predict(tune_oj_linear, oj_train)
test_pred_linear <- predict(tune_oj_linear, oj_test)

mean(train_pred != oj_train$Purchase)

[1] 0.16625

mean(test_pred != oj_test$Purchase)

[1] 0.1518519

• Using the tuned cost value produces a very slight improvement in both training and test error compared to cost = 0.01, though the difference is small since the original error rates were already pretty low.

(f) Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.

#Ploy results and best tune before tune
svm_oj_radial <- train(Purchase~.,data = oj_train, method = "svmRadial", 
                       trControl = train_control, preProcess = c("center", "scale"), 
                       tuneGrid = expand.grid(C=0.01,sigma = 0.01))
svm_oj_radial$results

NA

#radial test train errors before tune
train_pred_radial <- predict(svm_oj_radial, oj_train)
test_pred_radial <- predict(svm_oj_radial, oj_test)

mean(train_pred_radial != oj_train$Purchase)

[1] 0.39375

mean(test_pred_radial != oj_test$Purchase)

[1] 0.3777778

#Radial results and best tune
set.seed(1)
tune_oj_radial <- train(Purchase~., data = oj_train, method = "svmRadial", trControl = train_control,
                        preProcess = c("center", "scale"), 
                        tuneLength = 10)
tune_oj_radial$results

tune_oj_radial$bestTune

NA

#radial test train errors
train_pred_radial <- predict(tune_oj_radial, oj_train)
test_pred_radial <- predict(tune_oj_radial, oj_test)

mean(train_pred_radial != oj_train$Purchase)

[1] 0.14875

mean(test_pred_radial != oj_test$Purchase)

[1] 0.1814815

• With cost = 0.01, the radial kernel essentially predicts the majority class for every observation, since the default gamma combined with such a low cost produces an overly simple boundary. After tuning cost, the radial kernel’s performance improves substantially and becomes competitive with the linear kernel, achieving a test error rate in a similar range (roughly 17-19%).

(g) Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.

#Ploy results and best tune
svm_oj_poly <- train(Purchase~.,data = oj_train, method = "svmPoly", 
                       trControl = train_control, preProcess = c("center", "scale"), 
                       tuneGrid = expand.grid(C=0.01,scale = 0.01, degree = 2))
svm_oj_poly$results

#Poly test train errors before tune
train_pred_poly <- predict(svm_oj_poly, oj_train)
test_pred_poly <- predict(svm_oj_poly, oj_test)

mean(train_pred_poly != oj_train$Purchase)

[1] 0.39375

mean(test_pred_poly != oj_test$Purchase)

[1] 0.3777778

#Ploy results and best tune
set.seed(1)
tune_oj_poly <- train(Purchase~., data = oj_train, method = "svmPoly", trControl = train_control,
                        preProcess = c("center", "scale"), 
                        tuneGrid = expand.grid(degree = c(2, 3, 4),scale = 0.01,C = c(0.01, 0.1, 1, 10)))
tune_oj_poly$results

tune_oj_poly$bestTune

#Poly test train errors
train_pred_poly <- predict(tune_oj_poly, oj_train)
test_pred_poly <- predict(tune_oj_poly, oj_test)

mean(train_pred_poly != oj_train$Purchase)

[1] 0.155

mean(test_pred_poly != oj_test$Purchase)

[1] 0.1777778

• The polynomial kernel with a small cost value (C = 0.01) performs poorly compared with the tuned model. After tuning, the polynomial SVM improves its classification performance, but it does not outperform the linear or radial kernels. This suggests that the additional complexity from the polynomial boundary does not significantly improve prediction for this dataset.

(h) Overall, which approach seems to give the best results on this data?

results_oj <- data.frame(
  Kernel = c("Linear", "Radial", "Polynomial"),
  Test_Error = c(
    mean(test_pred_linear != oj_test$Purchase),
    mean(test_pred_radial != oj_test$Purchase),
    mean(test_pred_poly != oj_test$Purchase)
  ))
results_oj

• Overall, the tuned linear SVM produced the lowest test error rate, outperforming both the radial and polynomial kernels. The radial and polynomial kernels had similar performance, with slightly higher error rates around 18%. This suggests that the additional complexity provided by the non-linear kernels did not improve classification accuracy for the OJ dataset, and a simpler linear decision boundary was sufficient.