Q5
We have seen that we can fit an SVM with a non-linear kernel in
order to perform classification using a non-linear decision boundary. We
will now see that we can also obtain a non-linear decision boundary by
performing logistic regression using non-linear transformations of the
features.
(a) Generate a data set with n = 500 and p = 2, such that the obser-
vations belong to two classes with a quadratic decision boundary between
them.For instance, you can do this as follows: > x1 <- runif (500)
- 0.5 > x2 <- runif (500) - 0.5 > y <- 1 * (x1^2 - x2^2 >
0)
set.seed(42)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- as.factor(ifelse(x1^2 - x2^2 > 0, 1, 0))
(b) Plot the observations, colored according to their class labels.
Your plot should display X1 on the x-axis, and X2 on the y- axis.
plot(x1, x2, col = (3 - as.numeric(y)), xlab = "X1", ylab = "X2",
main = "True Quadratic Boundary")
(c) Fit a logistic regression model to the data, using X1 and X2 as
predictors.
glm.linear <- glm(y ~ x1 + x2, family = "binomial")
(d) Apply this model to the training data in order to obtain a
predicted class label for each training observation. Plot the
observations,colored according to the predicted class labels. The
decision boundary should be linear.
prob.linear <- predict(glm.linear, type = "response")
pred.linear <- ifelse(prob.linear > 0.5, 1, 0)
plot(x1, x2, col = (3 - pred.linear), main = "Linear Logistic Regression Predictions")
(e) Now fit a logistic regression model to the data using non-linear
functions of X1 and X2 as predictors (e.g. X2 1 , X1 ×X2, log(X2), and
so forth).
glm.nonlinear <- glm(y ~ poly(x1, 2) + poly(x2, 2), family = "binomial")
## Warning: glm.fit: algorithm did not converge
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
summary(glm.nonlinear)
##
## Call:
## glm(formula = y ~ poly(x1, 2) + poly(x2, 2), family = "binomial")
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 241.2 2464.3 0.098 0.922
## poly(x1, 2)1 -1560.4 43432.8 -0.036 0.971
## poly(x1, 2)2 150754.9 1452847.9 0.104 0.917
## poly(x2, 2)1 3829.4 54613.8 0.070 0.944
## poly(x2, 2)2 -145721.8 1403130.1 -0.104 0.917
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 6.9179e+02 on 499 degrees of freedom
## Residual deviance: 6.5043e-05 on 495 degrees of freedom
## AIC: 10
##
## Number of Fisher Scoring iterations: 25
(f) Apply this model to the training data in order to obtain a
predicted class label for each training observation. Plot the
observations, colored according to the predicted class labels. The
decision boundary should be obviously non-linear. If it is not, then
repeat (a)-(e) until you come up with an example in which the predicted
class labels are obviously non-linear.
prob.nonlinear <- predict(glm.nonlinear, type = "response")
pred.nonlinear <- ifelse(prob.nonlinear > 0.5, 1, 0)
plot(x1, x2, col = (3 - pred.nonlinear), xlab = "X1", ylab = "X2",
main = "Non-Linear Logistic Regression Predictions")
(g) Fit a support vector classifier to the data with X1 and X2 as
predictors. Obtain a class prediction for each training observation.
Plot the observations, colored according to the predicted class
labels.
library(e1071)
svm.linear <- svm(y ~ x1 + x2, kernel = "linear", cost = 1)
pred.svm_lin <- predict(svm.linear)
plot(x1, x2, col = (3 - as.numeric(pred.svm_lin)), xlab = "X1", ylab = "X2",
main = "Linear Support Vector Classifier Predictions")
(h) Fit a SVM using a non-linear kernel to the data. Obtain a class
prediction for each training observation. Plot the observations, colored
according to the predicted class labels.
svm.radial <- svm(y ~ x1 + x2, kernel = "radial", cost = 1, gamma = 1)
pred.svm_rad <- predict(svm.radial)
plot(x1, x2, col = (3 - as.numeric(pred.svm_rad)), xlab = "X1", ylab = "X2",
main = "Radial SVM Predictions")
A: What I see from this assignment, is basically, when our real data
is shaped like a circle or a curve, trying to use regular logistic
regression or a standard linear SVM is completely useless. These models
are stubborn and they are strictly forced to draw a straight line, which
means they end up chopping right through the middle of our curve and
getting a ton of predictions wrong. The fix would be to manually force
logistic regression to look at curves by adding math variables ourselves
or just use Radial SVM. The Radial SVM does alot of the work for
us.
Q7.
In this problem, you will use support vector approaches in order to
predict whether a given car gets high or low gas mileage based on the
Auto data set. Including Plots
(d) Make some plots to back up your assertions in (b) and (c).
# overall best linear and radial models
best.linear <- tune.linear$best.model
best.radial <- tune.radial$best.model
# zero errors
plot(best.linear, data = Auto_clean, horsepower ~ weight)
plot(best.radial, data = Auto_clean, horsepower ~ weight)
Q8.
This problem involves the OJ data set which is part of the ISLR2
package.
(a) Create a training set containing a random sample of 800
observations, and a test set containing the remaining observations.
set.seed(123)
# Pick random numbers
train_indices <- sample(1:nrow(OJ), 800)
# Spliting subsets
train_data <- OJ[train_indices, ]
test_data <- OJ[-train_indices, ]
(b) Fit a support vector classifier to the training data using cost
= 0.01, with Purchase as the response and the other vari- ables as
predictors. Use the summary() function to produce sum- mary statistics,
and describe the results obtained.
library(e1071)
# Fit Support Vector Classifier
svm.oj_linear <- svm(Purchase ~ ., data = train_data, kernel = "linear", cost = 0.01)
# model details
summary(svm.oj_linear)
##
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "linear",
## cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 442
##
## ( 220 222 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MM
(c) What are the training and test error rates?
# Training Error Rate
train.pred <- predict(svm.oj_linear, train_data)
train.table <- table(Actual = train_data$Purchase, Predicted = train.pred)
print(train.table)
## Predicted
## Actual CH MM
## CH 426 61
## MM 71 242
train_error <- 1 - sum(diag(train.table)) / sum(train.table)
print(train_error)
## [1] 0.165
# Test Error Rate
test.pred <- predict(svm.oj_linear, test_data)
test.table <- table(Actual = test_data$Purchase, Predicted = test.pred)
print(test.table)
## Predicted
## Actual CH MM
## CH 145 21
## MM 27 77
test_error <- 1 - sum(diag(test.table)) / sum(test.table)
print(test_error)
## [1] 0.1777778
(d) Use the tune() function to select an optimal cost. Consider val-
ues in the range 0.01 to 10.
set.seed(123)
# Tuneing cost
tune.oj_linear <- tune(svm, Purchase ~ ., data = train_data, kernel = "linear",
ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
# cross-validation error matrix
summary(tune.oj_linear)
##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.16875
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.17375 0.04910660
## 2 0.10 0.17500 0.04823265
## 3 1.00 0.16875 0.03963812
## 4 5.00 0.17250 0.04241004
## 5 10.00 0.17000 0.04005205
(e) Compute the training and test error rates using this new value
for cost.
# Extract the best tuned model from part d
best.oj_linear <- tune.oj_linear$best.model
# New Error Rate
best_train.pred <- predict(best.oj_linear, train_data)
best_train.error <- 1 - mean(best_train.pred == train_data$Purchase)
print(best_train.error)
## [1] 0.16
best_test.pred <- predict(best.oj_linear, test_data)
best_test.error <- 1 - mean(best_test.pred == test_data$Purchase)
print(best_test.error)
## [1] 0.1555556
(f) Repeat parts (b) through (e) using a support vector machine with
a radial kernel. Use the default value for gamma.
# Fit Model
svm.oj_radial <- svm(Purchase ~ ., data = train_data, kernel = "radial", cost = 0.01)
summary(svm.oj_radial)
##
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "radial",
## cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 0.01
##
## Number of Support Vectors: 629
##
## ( 313 316 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MM
# Errors
rad_train.error <- 1 - mean(predict(svm.oj_radial, train_data) == train_data$Purchase)
rad_test.error <- 1 - mean(predict(svm.oj_radial, test_data) == test_data$Purchase)
# Model cost
set.seed(123)
tune.oj_radial <- tune(svm, Purchase ~ ., data = train_data, kernel = "radial",
ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
best.oj_radial <- tune.oj_radial$best.model
# Radial Errors
opt_rad_train.error <- 1 - mean(predict(best.oj_radial, train_data) == train_data$Purchase)
opt_rad_test.error <- 1 - mean(predict(best.oj_radial, test_data) == test_data$Purchase)
(g) Repeat parts (b) through (e) using a support vector machine with
a polynomial kernel. Set degree = 2.
# Fit Model
svm.oj_poly <- svm(Purchase ~ ., data = train_data, kernel = "polynomial", degree = 2, cost = 0.01)
summary(svm.oj_poly)
##
## Call:
## svm(formula = Purchase ~ ., data = train_data, kernel = "polynomial",
## degree = 2, cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: polynomial
## cost: 0.01
## degree: 2
## coef.0: 0
##
## Number of Support Vectors: 631
##
## ( 313 318 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MM
# Errors
poly_train.error <- 1 - mean(predict(svm.oj_poly, train_data) == train_data$Purchase)
poly_poly_test.error <- 1 - mean(predict(svm.oj_poly, test_data) == test_data$Purchase)
# Tune Model some
set.seed(123)
tune.oj_poly <- tune(svm, Purchase ~ ., data = train_data, kernel = "polynomial", degree = 2,
ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
best.oj_poly <- tune.oj_poly$best.model
# Errors
opt_poly_train.error <- 1 - mean(predict(best.oj_poly, train_data) == train_data$Purchase)
opt_poly_test.error <- 1 - mean(predict(best.oj_poly, test_data) == test_data$Purchase)
(h) Overall, which approach seems to give the best results on this
data?
A: To find our best model, we look past how good the models
memorized the training data and check to see how they performed on our
other Test Set. When we compare our final optimized models, we are
looking specifically for the one with the lowest Test Error Rate
(best_test.error, opt_rad_test.error, or opt_poly_test.error). The
Linear SVM and Radial SVM usually end up performing very similiar, with
the final test error hovering around 15% to 18% depending on computer’s
random train/test split. The Polynomial kernel often trails slightly
behind with a higher error rate. Because the straight-line linear model
performs just as well as (or slightly better than) the complex curved
models here, it wins! In data science, if a simple straight line gives
you the same accuracy as a complicated curve, we always pick the simpler
Linear SVM because it is faster to compute and much easier to explain to
a regular boss or client.