STA-6543: Assignment 8
Austin Thrash - FFK221
2024-04-26
Table of Contents:
ISLR Chapter 9
Question 5
We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
5.(a) Generate Data Set
Generate a data set with n = 500 and p = 2, such that the observations belong to two classes with a quadratic decision boundary between them.
set.seed(12345)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)5.(b) Plot Data
Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the y- axis.
plot(x1,x2,col=ifelse(y,'red','blue'))
5.(c) Logistic Regression Model
Fit a logistic regression model to the data, using X1 and X2 as predictors.
logModel = glm(y ~ . , family='binomial', data=data.frame(x1,x2,y))
logModel##
## Call: glm(formula = y ~ ., family = "binomial", data = data.frame(x1,
## x2, y))
##
## Coefficients:
## (Intercept) x1 x2
## 0.07563 0.46494 0.41377
##
## Degrees of Freedom: 499 Total (i.e. Null); 497 Residual
## Null Deviance: 692.2
## Residual Deviance: 688 AIC: 6945.(d) Plot Predicted Class Label
Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.
logModel.pred = predict(logModel, data.frame(x1,x2))
plot(x1, x2, col=ifelse(logModel.pred > 0, 'red', 'blue'), pch = ifelse(as.integer(logModel.pred > 0) == y, 1, 3))
5.(e) Logistic Regression using Non-Linear Functions
Now fit a logistic regression model to the data using non-linear functions of X1 and X2 as predictors (e.g. X12, X1 ×X2, log(X2), and so forth).
logModel = glm(y ~ poly(x1, 2) + poly(x2, 2) , family='binomial', data=data.frame(x1, x2, y))## Warning: glm.fit: algorithm did not converge## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred5.(f) Plot Predicted Class Label for New Model
Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear.
logModel.pred = predict(logModel, data.frame(x1,x2))
plot(x1, x2, col=ifelse(logModel.pred > 0, 'red', 'blue'), pch = ifelse(as.integer(logModel.pred > 0) == y, 1, 3))
5.(g) Fitting and Plotting SVM
Fit a support vector classifier to the data with X1 and X2 as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
svmModel = svm(y ~ ., data = data.frame(x1, x2, y=as.factor(y)), kernel='linear')
svmModel.pred = predict(svmModel, data.frame(x1,x2), type='response')
plot(x1, x2, col = ifelse(svmModel.pred != 0, 'red', 'blue'), pch = ifelse(svmModel.pred == y, 1, 3))
5.(h) Fitting and Plotting Non-Linear SVM
Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
svmModel = svm(y ~ ., data = data.frame(x1, x2, y=as.factor(y)), kernel='radial')
svmModel.pred = predict(svmModel, data.frame(x1,x2), type='response')
plot(x1, x2, col = ifelse(svmModel.pred != 0, 'red', 'blue'), pch = ifelse(svmModel.pred == y, 1, 3))
5.(i) Results
Comment on your results.
Looking at our results, we can see that logistic regression with interactions and SVM’s with non-linear kernels can be equally as powerful when attempting to identify non-linear decision boundaries. Like expected, when attempting to use SVM’s with linear kernels and logistic regression without interactions they do not perform as well when attempting to identify the same non-linear decisions boundaries. From that, we can conclude that SVM’s can be useful in both linear and non-linear situations, however further tuning is necessary to achieve the best performance.
Question 7
In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.
7.(a) Binary Variable for Gas Mileage
Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.
Auto$gas_med <- ifelse(Auto$mpg > median(Auto$mpg), 1, 0)7.(b) Linear SVM
Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results.
svmTuned0 = tune(svm, gas_med ~ . - mpg, data = Auto, ranges = list(cost = c(0.001, 0.01, 0.1, 1, 5, 10, 100)), kernel = 'linear')
svmTuned0$performances## cost error dispersion
## 1 1e-03 0.10845689 0.02413298
## 2 1e-02 0.10376012 0.03443415
## 3 1e-01 0.10020081 0.03789915
## 4 1e+00 0.09412209 0.03266744
## 5 5e+00 0.09934967 0.03193312
## 6 1e+01 0.10536253 0.03197532
## 7 1e+02 0.11756741 0.03342222plot(svmTuned0$performance[, c(1, 2)], type = 'l')
We can see that in the beginning, as cost increased, the error decreased significantly. However after a certain point, the error stops decreasing and begins to increase, this valley in the graph with the lowest error indicates the best cost value. In our case, a cost of 1 performs the best.
7.(c) Radial and Polynomial SVM
Now repeat (b), this time using SVMs with radial and polynomial basis kernels, with different values of gamma and degree and cost. Comment on your results.
svmTuned1 = tune(svm, gas_med ~ . - mpg, data = Auto, ranges = list(cost = c(0.001, 0.01, 0.1, 1, 5, 10, 100), degree = 2:10), kernel = 'polynomial')
svmTuned1$performances## cost degree error dispersion
## 1 1e-03 2 0.4984479 0.04871973
## 2 1e-02 2 0.4982038 0.04870642
## 3 1e-01 2 0.4957401 0.04855367
## 4 1e+00 2 0.4715967 0.04846382
## 5 5e+00 2 0.3822803 0.05960291
## 6 1e+01 2 0.3115269 0.06261205
## 7 1e+02 2 0.2527266 0.03343528
## 8 1e-03 3 0.4984684 0.04872129
## 9 1e-02 3 0.4984089 0.04872194
## 10 1e-01 3 0.4978145 0.04872887
## 11 1e+00 3 0.4918093 0.04882159
## 12 5e+00 3 0.4659938 0.05016607
## 13 1e+01 3 0.4356513 0.05324926
## 14 1e+02 3 0.2457496 0.03933104
## 15 1e-03 4 0.4984749 0.04872121
## 16 1e-02 4 0.4984735 0.04872115
## 17 1e-01 4 0.4984600 0.04872057
## 18 1e+00 4 0.4983246 0.04871479
## 19 5e+00 4 0.4977238 0.04869147
## 20 1e+01 4 0.4969557 0.04866300
## 21 1e+02 4 0.4834570 0.04914540
## 22 1e-03 5 0.4984750 0.04872122
## 23 1e-02 5 0.4984749 0.04872123
## 24 1e-01 5 0.4984734 0.04872127
## 25 1e+00 5 0.4984583 0.04872177
## 26 5e+00 5 0.4983912 0.04872400
## 27 1e+01 5 0.4983075 0.04872682
## 28 1e+02 5 0.4967803 0.04878020
## 29 1e-03 6 0.4984750 0.04872122
## 30 1e-02 6 0.4984750 0.04872122
## 31 1e-01 6 0.4984750 0.04872122
## 32 1e+00 6 0.4984743 0.04872120
## 33 5e+00 6 0.4984715 0.04872114
## 34 1e+01 6 0.4984680 0.04872105
## 35 1e+02 6 0.4984046 0.04871960
## 36 1e-03 7 0.4984750 0.04872122
## 37 1e-02 7 0.4984750 0.04872122
## 38 1e-01 7 0.4984750 0.04872122
## 39 1e+00 7 0.4984750 0.04872122
## 40 5e+00 7 0.4984747 0.04872123
## 41 1e+01 7 0.4984744 0.04872124
## 42 1e+02 7 0.4984688 0.04872142
## 43 1e-03 8 0.4984750 0.04872122
## 44 1e-02 8 0.4984750 0.04872122
## 45 1e-01 8 0.4984750 0.04872122
## 46 1e+00 8 0.4984750 0.04872122
## 47 5e+00 8 0.4984750 0.04872122
## 48 1e+01 8 0.4984750 0.04872122
## 49 1e+02 8 0.4984747 0.04872121
## 50 1e-03 9 0.4984750 0.04872122
## 51 1e-02 9 0.4984750 0.04872122
## 52 1e-01 9 0.4984750 0.04872122
## 53 1e+00 9 0.4984750 0.04872122
## 54 5e+00 9 0.4984750 0.04872122
## 55 1e+01 9 0.4984750 0.04872122
## 56 1e+02 9 0.4984750 0.04872122
## 57 1e-03 10 0.4984750 0.04872122
## 58 1e-02 10 0.4984750 0.04872122
## 59 1e-01 10 0.4984750 0.04872122
## 60 1e+00 10 0.4984750 0.04872122
## 61 5e+00 10 0.4984750 0.04872122
## 62 1e+01 10 0.4984750 0.04872122
## 63 1e+02 10 0.4984750 0.04872122The best results for the model with a polynomial kernel occurred when using a cost of 100 and a degree of 3, this model yielded an error of 0.2439237
svmTuned2 = tune(svm, gas_med ~ . - mpg, data = Auto, ranges = list(cost = c(0.001, 0.01, 0.1, 1, 5, 10, 100), gamma = c(0.001, 0.01, 0.1, 1, 10, 100, 1000)), kernel = 'radial')
svmTuned2$performances## cost gamma error dispersion
## 1 1e-03 1e-03 0.49987970 0.033043741
## 2 1e-02 1e-03 0.47450477 0.030401893
## 3 1e-01 1e-03 0.26742465 0.024866900
## 4 1e+00 1e-03 0.10374379 0.026033188
## 5 5e+00 1e-03 0.10082409 0.031233736
## 6 1e+01 1e-03 0.09914611 0.032816234
## 7 1e+02 1e-03 0.08857806 0.034985013
## 8 1e-03 1e-02 0.47999750 0.031033907
## 9 1e-02 1e-02 0.30529702 0.023258644
## 10 1e-01 1e-02 0.09907724 0.025818379
## 11 1e+00 1e-02 0.08646731 0.030627393
## 12 5e+00 1e-02 0.08264774 0.034260036
## 13 1e+01 1e-02 0.08168629 0.033894211
## 14 1e+02 1e-02 0.07324683 0.026329553
## 15 1e-03 1e-01 0.45425520 0.029338571
## 16 1e-02 1e-01 0.15296418 0.023143738
## 17 1e-01 1e-01 0.07217802 0.029517882
## 18 1e+00 1e-01 0.07065579 0.032907684
## 19 5e+00 1e-01 0.06139390 0.025255693
## 20 1e+01 1e-01 0.06329588 0.024254859
## 21 1e+02 1e-01 0.08234786 0.029041124
## 22 1e-03 1e+00 0.50010684 0.033371894
## 23 1e-02 1e+00 0.47661249 0.033418131
## 24 1e-01 1e+00 0.28231852 0.031571397
## 25 1e+00 1e+00 0.09706563 0.013597495
## 26 5e+00 1e+00 0.10035315 0.014152582
## 27 1e+01 1e+00 0.10046556 0.014389877
## 28 1e+02 1e+00 0.10046399 0.014391592
## 29 1e-03 1e+01 0.50238280 0.033413848
## 30 1e-02 1e+01 0.49754685 0.033494420
## 31 1e-01 1e+01 0.45170575 0.033576071
## 32 1e+00 1e+01 0.24266165 0.003994669
## 33 5e+00 1e+01 0.24265728 0.003991108
## 34 1e+01 1e+01 0.24265728 0.003991108
## 35 1e+02 1e+01 0.24265728 0.003991108
## 36 1e-03 1e+02 0.50248854 0.033333373
## 37 1e-02 1e+02 0.49801525 0.033004631
## 38 1e-01 1e+02 0.45580106 0.030467064
## 39 1e+00 1e+02 0.25091922 0.001020318
## 40 5e+00 1e+02 0.25091922 0.001020318
## 41 1e+01 1e+02 0.25091922 0.001020318
## 42 1e+02 1e+02 0.25091922 0.001020318
## 43 1e-03 1e+03 0.50248859 0.033333315
## 44 1e-02 1e+03 0.49801545 0.033004493
## 45 1e-01 1e+03 0.45580182 0.030466880
## 46 1e+00 1e+03 0.25092340 0.001019278
## 47 5e+00 1e+03 0.25092340 0.001019278
## 48 1e+01 1e+03 0.25092340 0.001019278
## 49 1e+02 1e+03 0.25092340 0.001019278The best results for the radial model occurred when using a cost of 5 and a gamma of 0.1, this model yielded an error of 0.06322822.
7.(d) SVM Plots
Make some plots to back up your assertions in (b) and (c).
# Function to plot performance metrics
plot_performance <- function(tuned_model, parameter_name) {
# Extract performance metrics
performances <- tuned_model$performances
# Plot performance metrics
plot(performances[[parameter_name]], performances$error, type = "b",
xlab = parameter_name, ylab = "Cross-validation error",
main = paste("Cross-validation error vs", parameter_name))
}
plot_performance(svmTuned0, "cost")
plot_performance(svmTuned1, "degree")
plot_performance(svmTuned2, "gamma")
I tried using the following function “plot(svmTuned0, Auto, weight ~ horsepower)” to try and create classifications plots, however for some reason my R studio does nothing after running the R chunk. No errors occur when running the chunk so I am having trouble debugging the issue. In light of this issue, I decided to at least attempt to graph the cross validation error against cost (Linear Model - svmTuned0), degree (Radial Model - svmTuned1), and gamma (Polynomial Model - svmTuned2).
Question 8
This problem involves the OJ data set which is part of the ISLR2 package.
8.(a) Create Training Set
Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.
set.seed(12345)
train <- sample(nrow(OJ), 800)
OJ_train <- OJ[train, ]
OJ_test <- OJ[-train, ]8.(b) SVM using Cost = 0.01
Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.
svm_Linear <- svm(Purchase ~ ., data = OJ_train, kernel = "linear", cost = 0.01)
summary(svm_Linear)##
## Call:
## svm(formula = Purchase ~ ., data = OJ_train, kernel = "linear", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 440
##
## ( 220 220 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMSVM-Type:
- C-classification: Indicates that the SVM model is being used or classification
SVM-Kernel:
- linear: The kernel being used in the SVM is linear and the decision boundary is a linear function
Cost:
- 0.01: determines the trade off between maximizing the margin and minimizing the classification error
Number of support vectors:
- 440: indicates the number of data points closest to the decision boundary.
Number of Classes:
- 2: SVM model is designed to classify instances into two classes
Level:
- CH MM: these are the levels of the target variable
8.(c) Training and Test Error Rates
What are the training and test error rates?
predTrain <- predict(svm_Linear, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 439 61
## MM 77 223predTest <- predict(svm_Linear, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 144 9
## MM 35 82train_Error <- (77 + 61) / (439 + 223 + 77 + 61)
test_Error <- (35 + 9) / (144 + 82 + 35 + 9)
paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.1725"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.162962962962963"8.(d) Tune() Function
Use the tune() function to select an optimal cost. Consider values in the range 0.01 to 10.
set.seed(12345)
SVMTuned <- tune(svm, Purchase ~ ., data = OJ_train, kernel = "linear", ranges = list(cost = 10^seq(-2, 1, by = 0.25)))
summary(SVMTuned)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1.778279
##
## - best performance: 0.17125
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01000000 0.18000 0.04174992
## 2 0.01778279 0.18125 0.04177070
## 3 0.03162278 0.18125 0.04093101
## 4 0.05623413 0.18000 0.04297932
## 5 0.10000000 0.17875 0.03866254
## 6 0.17782794 0.17625 0.03884174
## 7 0.31622777 0.17500 0.03584302
## 8 0.56234133 0.17375 0.03458584
## 9 1.00000000 0.17250 0.03622844
## 10 1.77827941 0.17125 0.03634805
## 11 3.16227766 0.17125 0.03120831
## 12 5.62341325 0.17500 0.03584302
## 13 10.00000000 0.17500 0.037267808.(e) Compute new Training and Test Error Rates
Compute the training and test error rates using this new value for cost.
SVM_cost <- svm(Purchase ~ ., kernel = "linear", data = OJ_train, cost = SVMTuned$best.parameter$cost)
predTrain <- predict(SVM_cost, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 438 62
## MM 72 228predTest <- predict(SVM_cost, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 143 10
## MM 33 84train_Error <- (72 + 62) / (438 + 228 + 72 + 62)
test_Error <- (33 + 10) / (143 + 84 + 33 + 10)
print("- Using Best Cost - ")## [1] "- Using Best Cost - "paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.1675"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.159259259259259"8.(f) Radial: Repeat (b) through (e)
Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.
SVM_Radial <- svm(Purchase ~ ., kernel = "radial", data = OJ_train)
summary(SVM_Radial)##
## Call:
## svm(formula = Purchase ~ ., data = OJ_train, kernel = "radial")
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 1
##
## Number of Support Vectors: 375
##
## ( 188 187 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMpredTrain <- predict(SVM_Radial, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 457 43
## MM 79 221predTest <- predict(SVM_Radial, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 145 8
## MM 37 80train_Error <- (79 + 43) / (457 + 221 + 79 + 43)
test_Error <- (37 + 8) / (145 + 80 + 37 + 8)
print("- Using Radial - ")## [1] "- Using Radial - "paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.1525"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.166666666666667"set.seed(12345)
Radial_tuned <- tune(svm, Purchase ~ ., data = OJ_train, kernel = "radial", ranges = list(cost = 10^seq(-2,
1, by = 0.25)))
SVM_cost <- svm(Purchase ~ ., kernel = "radial", data = OJ_train, cost = Radial_tuned$best.parameter$cost)predTrain <- predict(SVM_cost, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 457 43
## MM 79 221predTest <- predict(SVM_cost, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 145 8
## MM 37 80train_Error <- (79 + 43) / (457 + 221 + 79 + 43)
test_Error <- (37 + 8) / (145 + 80 + 37 + 8)
print("- Using Radial and Best Cost - ")## [1] "- Using Radial and Best Cost - "paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.1525"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.166666666666667"8.(g) Polynomial: Repeat (b) through (e)
Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.
SVM_Poly <- svm(Purchase ~ ., kernel = "polynomial", data = OJ_train, degree = 2)
summary(SVM_Poly)##
## Call:
## svm(formula = Purchase ~ ., data = OJ_train, kernel = "polynomial",
## degree = 2)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: polynomial
## cost: 1
## degree: 2
## coef.0: 0
##
## Number of Support Vectors: 455
##
## ( 233 222 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMpredTrain <- predict(SVM_Poly, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 462 38
## MM 108 192predTest <- predict(SVM_Poly, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 148 5
## MM 47 70train_Error <- (108 + 38) / (462 + 192 + 108 + 38)
test_Error <- (47 + 5) / (148 + 70 + 47 + 5)
print("- Using Polynomial - ")## [1] "- Using Polynomial - "paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.1825"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.192592592592593"set.seed(12345)
Poly_tuned <- tune(svm, Purchase ~ ., data = OJ_train, kernel = "polynomial", degree = 2, ranges = list(cost = 10^seq(-2,
1, by = 0.25)))
SVM_cost <- svm(Purchase ~ ., kernel = "polynomial", data = OJ_train, cost = Poly_tuned$best.parameter$cost)predTrain <- predict(SVM_cost, OJ_train)
table(OJ_train$Purchase, predTrain)## predTrain
## CH MM
## CH 462 38
## MM 79 221predTest <- predict(SVM_cost, OJ_test)
table(OJ_test$Purchase, predTest)## predTest
## CH MM
## CH 146 7
## MM 44 73train_Error <- (79 + 38) / (462 + 221 + 79 + 38)
test_Error <- (44 + 7) / (146 + 73 + 44 + 7)
print("- Using Polynomial and Best Cost - ")## [1] "- Using Polynomial and Best Cost - "paste("Train Error Rate: ", train_Error)## [1] "Train Error Rate: 0.14625"paste("Test Error Rate: ", test_Error)## [1] "Test Error Rate: 0.188888888888889"8.(h) Results
Overall, which approach seems to give the best results on this data?
The SVM model with a linear kernel using a cost of 1.778279 seems to perform the best on the data as it has the smallest test error rate. While this model does have a better test error rate, the SVM model with a polynomial kernel and a cost of 10 had a train error rate of 0.14625 which could possibly suggest that further tuning could yield better test results.