Chris Serrano - Assignment 8

Question 5

a, b

set.seed(1)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)

plot(x1, x2, col = (y + 1), xlab = "X1", ylab = "X2", main = "Simulated Dataset")

c, d

df <- data.frame(x1 = x1, x2 = x2, y = as.factor(y))
glm.fit <- glm(y ~ x1 + x2, data = df, family = binomial)

glm.prob <- predict(glm.fit, df, type = "response")
glm.pred <- ifelse(glm.prob > 0.5, 1, 0)

plot(x1, x2, col = (glm.pred + 1), xlab = "X1", ylab = "X2", main = "Linear Logistic Regression Predictions")

Linear logistic regression fails to capture the non-linear decision boundary.

e, f

glm.nl.fit <- glm(y ~ x1 + x2 + I(x1^2) + I(x2^2) + I(x1*x2), data = df, family = binomial)

glm.nl.prob <- predict(glm.nl.fit, df, type = "response")
glm.nl.pred <- ifelse(glm.nl.prob > 0.5, 1, 0)

plot(x1, x2, col = (glm.nl.pred + 1), xlab = "X1", ylab = "X2", main = "Nonlinear Logistic Regression Predictions")

Adding quadratic terms allows logistic regression to model the curved boundary.

g, h

library(e1071)

# Support Vector Classifier
svc.fit <- svm(y ~ x1 + x2, data = df, kernel = "linear", cost = 1)
svc.pred <- predict(svc.fit)

plot(x1, x2, col = as.integer(svc.pred), xlab = "X1", ylab = "X2", main = "Support Vector Classifier Predictions")

# Support Vector Machine
svm.fit <- svm(y ~ x1 + x2, data = df, kernel = "radial", cost = 1)
svm.pred <- predict(svm.fit)

plot(x1, x2, col = as.integer(svm.pred), xlab = "X1", ylab = "X2", main = "Radial SVM Predictions")

The Support Vector Classifier with a linear kernel fails to model the non linear decision boundary. The Support Vector Machine with a radial kernel does model the non linear decision boundary.

i

Support Vector Machines with non linear kernels can model non linear boundaries automatically without requiring manual feature transforms. Logistic regression can also model non linear boundaries but requires the user to explicitly specify non linear and interactions in the formula.

---

Question 7

a

library(ISLR2)

Auto.data <- Auto
Auto.data$mpg01 <- as.factor(ifelse(Auto.data$mpg > median(Auto.data$mpg), 1, 0))
Auto.data$mpg <- NULL

b

set.seed(1)
tune.svc <- tune(svm, mpg01 ~ ., data = Auto.data, kernel = "linear",
                 ranges = list(cost = c(0.01, 0.1, 1, 10, 100)))
summary(tune.svc)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.08673077 

- Detailed performance results:

NA

The optimal cost value is 1e-01, which achieves the lowest cross-validation error rate of 0.089.

c

set.seed(1)

# Radial Kernel
tune.rad <- tune(svm, mpg01 ~ ., data = Auto.data, kernel = "radial",
                 ranges = list(cost = c(0.1, 1, 10, 100), gamma = c(0.01, 0.1, 1, 5)))
summary(tune.rad)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.07897436 

- Detailed performance results:

# Polynomial Kernel
tune.poly <- tune(svm, mpg01 ~ ., data = Auto.data, kernel = "polynomial",
                  ranges = list(cost = c(0.1, 1, 10, 100), degree = c(2, 3, 4)))
summary(tune.poly)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.3163462 

- Detailed performance results:

NA

For the radial kernel, the optimal parameters are cost = 10 and gamma = 1, achieving a cross-validation error rate of 0.079. For the polynomial kernel, the optimal parameters are cost = 100 and degree = 2, achieving a cross-validation error rate of 0.31.

d

best.linear <- tune.svc$best.model
best.radial <- tune.rad$best.model
best.poly <- tune.poly$best.model

plot(best.linear, Auto.data, horsepower ~ weight, main = "Support Vector Classifier Decision Boundary")

plot(best.radial, Auto.data, horsepower ~ weight, main = "Radial SVM Decision Boundary")

plot(best.poly, Auto.data, horsepower ~ weight, main = "Polynomial SVM Decision Boundary")

---

Question 8

a

set.seed(1)

train_idx <- 1:800
train_set <- OJ[train_idx, ]
test_set <- OJ[-train_idx, ]

b

svc.fit <- svm(Purchase ~ ., data = train_set, kernel = "linear", cost = 0.01)
summary(svc.fit)

Call:
svm(formula = Purchase ~ ., data = train_set, kernel = "linear", cost = 0.01)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  linear 
       cost:  0.01 

Number of Support Vectors:  424

 ( 212 212 )


Number of Classes:  2 

Levels: 
 CH MM

The model selects 424 support vectors

c

train.pred <- predict(svc.fit, train_set)
test.pred <- predict(svc.fit, test_set)

svc.train.err <- mean(train.pred != train_set$Purchase)
svc.test.err <- mean(test.pred != test_set$Purchase)

cat("Support Vector Classifier (cost = 0.01) Training Error Rate:", svc.train.err, "\n")

Support Vector Classifier (cost = 0.01) Training Error Rate: 0.155 

cat("Support Vector Classifier (cost = 0.01) Test Error Rate:", svc.test.err, "\n")

Support Vector Classifier (cost = 0.01) Test Error Rate: 0.2074074 

The training error rate is 15.5% and the test error rate is 20.74%.

d

set.seed(1)
tune.svc <- tune(svm, Purchase ~ ., data = train_set, kernel = "linear",
                 ranges = list(cost = c(0.01, 0.05, 0.1, 0.5, 1, 5, 10)))
summary(tune.svc)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.15375 

- Detailed performance results:

NA

The optimal cost value selected by cross-validation is .01.

e

best.svc <- tune.svc$best.model

svc.tuned.train.pred <- predict(best.svc, train_set)
svc.tuned.test.pred <- predict(best.svc, test_set)

svc.tuned.train.err <- mean(svc.tuned.train.pred != train_set$Purchase)
svc.tuned.test.err <- mean(svc.tuned.test.pred != test_set$Purchase)

cat("Tuned Support Vector Classifier Training Error Rate:", svc.tuned.train.err, "\n")

Tuned Support Vector Classifier Training Error Rate: 0.155 

cat("Tuned Support Vector Classifier Test Error Rate:", svc.tuned.test.err, "\n")

Tuned Support Vector Classifier Test Error Rate: 0.2074074 

Tuning the model results in a training error rate to 15.5% and the test error rate to 20.74%.

f

# 1. Fit radial model with cost = 0.01
rad.fit <- svm(Purchase ~ ., data = train_set, kernel = "radial", cost = 0.01)
summary(rad.fit)

Call:
svm(formula = Purchase ~ ., data = train_set, kernel = "radial", cost = 0.01)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  radial 
       cost:  0.01 

Number of Support Vectors:  608

 ( 305 303 )


Number of Classes:  2 

Levels: 
 CH MM

rad.train.err <- mean(predict(rad.fit, train_set) != train_set$Purchase)
rad.test.err <- mean(predict(rad.fit, test_set) != test_set$Purchase)
cat("Radial SVM (cost = 0.01) Training Error Rate:", rad.train.err, "\n")

Radial SVM (cost = 0.01) Training Error Rate: 0.37875 

cat("Radial SVM (cost = 0.01) Test Error Rate:", rad.test.err, "\n")

Radial SVM (cost = 0.01) Test Error Rate: 0.4222222 

# 2. Tune radial model
set.seed(1)
tune.rad <- tune(svm, Purchase ~ ., data = train_set, kernel = "radial",
                 ranges = list(cost = c(0.01, 0.05, 0.1, 0.5, 1, 5, 10)))
summary(tune.rad)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.16625 

- Detailed performance results:

best.rad <- tune.rad$best.model
rad.tuned.train.err <- mean(predict(best.rad, train_set) != train_set$Purchase)
rad.tuned.test.err <- mean(predict(best.rad, test_set) != test_set$Purchase)
cat("Tuned Radial SVM Training Error Rate:", rad.tuned.train.err, "\n")

Tuned Radial SVM Training Error Rate: 0.15375 

cat("Tuned Radial SVM Test Error Rate:", rad.tuned.test.err, "\n")

Tuned Radial SVM Test Error Rate: 0.1777778 

For the radial SVM: * The un-tuned model (cost = 0.01) has a training error of 38.38% and a test error of 41.11% (defaulting to the majority class prediction). * Cross-validation selects an optimal cost value of 0.5. * The tuned radial model achieves a training error of 14.88% and a test error of 17.78%.

g

poly.fit <- svm(Purchase ~ ., data = train_set, kernel = "polynomial", degree = 2, cost = 0.01)
summary(poly.fit)

Call:
svm(formula = Purchase ~ ., data = train_set, kernel = "polynomial", degree = 2, 
    cost = 0.01)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  polynomial 
       cost:  0.01 
     degree:  2 
     coef.0:  0 

Number of Support Vectors:  612

 ( 309 303 )


Number of Classes:  2 

Levels: 
 CH MM

poly.train.err <- mean(predict(poly.fit, train_set) != train_set$Purchase)
poly.test.err <- mean(predict(poly.fit, test_set) != test_set$Purchase)
cat("Polynomial SVM (cost = 0.01, degree = 2) Training Error Rate:", poly.train.err, "\n")

Polynomial SVM (cost = 0.01, degree = 2) Training Error Rate: 0.37875 

cat("Polynomial SVM (cost = 0.01, degree = 2) Test Error Rate:", poly.test.err, "\n")

Polynomial SVM (cost = 0.01, degree = 2) Test Error Rate: 0.4222222 

set.seed(1)
tune.poly <- tune(svm, Purchase ~ ., data = train_set, kernel = "polynomial", degree = 2,
                  ranges = list(cost = c(0.01, 0.05, 0.1, 0.5, 1, 5, 10)))
summary(tune.poly)

Parameter tuning of ‘svm’:

- sampling method: 10-fold cross validation 

- best parameters:

- best performance: 0.18 

- Detailed performance results:

best.poly <- tune.poly$best.model
poly.tuned.train.err <- mean(predict(best.poly, train_set) != train_set$Purchase)
poly.tuned.test.err <- mean(predict(best.poly, test_set) != test_set$Purchase)
cat("Tuned Polynomial SVM Training Error Rate:", poly.tuned.train.err, "\n")

Tuned Polynomial SVM Training Error Rate: 0.155 

cat("Tuned Polynomial SVM Test Error Rate:", poly.tuned.test.err, "\n")

Tuned Polynomial SVM Test Error Rate: 0.1851852 

For the polynomial SVM: * The un-tuned model (cost = 0.01) has a training error of 37.87% and a test error of 42.22%. * Cross-validation selects an optimal cost value of 5. The tuned polynomial model achieves a training error of 15.5% and a test error of 18.52%.

h

The tuned radial Support Vector Machine performs best on the test set, achieving the lowest test error rate.