Assignment 8
Mario Martinez
ISLR2 book starting on PDF Page 410.
# Load dependencies
pacman::p_load(ISLR2, e1071, caret)Question 5
We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
Part a)
Generate a data set with \(n = 500\) and \(p = 2\), such that the observations belong to two classes with a quadratic decision boundary between them. For instance, you can do this as follows:
> \(x_1\) = runif(500) -
0.5
> \(x_2\) =
runif(500) - 0.5
> \(y\) = 1 x (\(x_1^2 - x_2^2\) > 0)
set.seed(42)
x1 = runif(500) - 0.5
x2 = runif(500) - 0.5
y = 1 * (x1^2 - x2^2 > 0)Part b)
Plot the observations, colored according to their class labels. Your plot should display \(X_1\) on the \(x\)-axis, and \(X_2\) on the \(y\)-axis.
plot(x1, x2, col= c('blue', 'red')[y+1],
main= 'Observations Colored by Class Label')
Part c)
Fit a logistic regression model to the data, using \(X_1\) and \(X_2\) as predictors.
fit_glm = glm(y ~ x1 + x2, family= 'binomial')
summary(fit_glm)##
## Call:
## glm(formula = y ~ x1 + x2, family = "binomial")
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 0.09978 0.08976 1.112 0.266
## x1 -0.17659 0.30658 -0.576 0.565
## x2 -0.20067 0.30978 -0.648 0.517
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 691.79 on 499 degrees of freedom
## Residual deviance: 691.08 on 497 degrees of freedom
## AIC: 697.08
##
## Number of Fisher Scoring iterations: 3Part d)
Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.
# Predict
preds = predict(fit_glm, type= 'response')
# Classifications
class_labels = ifelse(preds > 0.5, 1, 0)
# Plot
plot(x1, x2, col= c('blue', 'red')[class_labels+1])
Part e)
Now fit a logistic regression model to the data using non-linear functions of \(X_1\) and \(X_2\) as predictors (e.g. \(X^2_1\) , \(X_1×X_2\), \(log(X_2)\), and so forth).
# Fit a logistic regression with transformations
fit_glm_poly = glm(y ~ poly(x1, 2) + poly(x2, 2), family= 'binomial')## Warning: glm.fit: algorithm did not converge## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurredsummary(fit_glm_poly)##
## Call:
## glm(formula = y ~ poly(x1, 2) + poly(x2, 2), family = "binomial")
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 241.2 2464.3 0.098 0.922
## poly(x1, 2)1 -1560.4 43432.8 -0.036 0.971
## poly(x1, 2)2 150754.9 1452847.9 0.104 0.917
## poly(x2, 2)1 3829.4 54613.8 0.070 0.944
## poly(x2, 2)2 -145721.8 1403130.1 -0.104 0.917
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 6.9179e+02 on 499 degrees of freedom
## Residual deviance: 6.5043e-05 on 495 degrees of freedom
## AIC: 10
##
## Number of Fisher Scoring iterations: 25Part f)
Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear.
preds = predict(fit_glm_poly, type= 'response')
class_labels = ifelse(preds > 0.5, 1, 0)
plot(x1, x2, col= c('blue', 'red')[class_labels+1])
Part g)
Fit a support vector classifier to the data with \(X_1\) and \(X_2\) as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
# Create a dataframe of the data
df = data.frame(x1 = x1, x2 = x2, y = as.factor(y))
# Fit a SVC model
fit_svc = svm(y ~ x1 + x2, data= df, kernel= 'linear', cost= 0.1)
# Predict
preds = predict(fit_svc)
df.pos = df[preds == 1, ]
df.neg = df[preds == 0, ]
# Plot
plot(df.pos$x1, df.pos$x2, col= 'blue', xlab= 'X1', ylab= 'X2')
points(df.neg$x1, df.neg$x2, col = "red")
Part h)
Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
# Fit a SVM model
fit_svm = svm(y ~ x1 + x2, data= df, kernel= 'radial', gamma= 5)
# Predict
preds = predict(fit_svm, df)
df.pos = df[preds == 1, ]
df.neg = df[preds == 0, ]
# Plot
plot(df.pos$x1, df.pos$x2, col= 'blue', xlab= 'X1', ylab= 'X2')
points(df.neg$x1, df.neg$x2, col = "red")
Part i)
Comment on your results.
The SVC model did not identify any relationship while the SVM model
with radial kernels clearly identify a non-linear relationship.
Question 7
In this problem, you will use support vector approaches in order
to predict whether a given car gets high or low gas mileage based on the
Auto data set.
Part a)
Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.
# Read in the data
auto_df = Auto
head(auto_df)## mpg cylinders displacement horsepower weight acceleration year origin
## 1 18 8 307 130 3504 12.0 70 1
## 2 15 8 350 165 3693 11.5 70 1
## 3 18 8 318 150 3436 11.0 70 1
## 4 16 8 304 150 3433 12.0 70 1
## 5 17 8 302 140 3449 10.5 70 1
## 6 15 8 429 198 4341 10.0 70 1
## name
## 1 chevrolet chevelle malibu
## 2 buick skylark 320
## 3 plymouth satellite
## 4 amc rebel sst
## 5 ford torino
## 6 ford galaxie 500# Calculate the median mpg
mpg_median = median(auto_df$mpg, na.rm= TRUE)
# Create binary variable
auto_df$high_mpg = ifelse(auto_df$mpg > mpg_median, 1, 0)
auto_df$high_mpg = as.factor(auto_df$high_mpg)Part b)
Fit a support vector classifier to the data with various values
of cost, in order to predict whether a car gets high or low
gas mileage. Report the cross-validation errors associated with
different values of this parameter. Comment on your results. Note you
will need to fit the classifier without the gas mileage variable to
produce sensible results.
The SVC model with 10-fold cross validation identified the best cost to be 0.01 with the lowest error rate of 0.08916667 .
set.seed(42)
# Tune a SVC model with cross validation
cv_svc = tune(svm, high_mpg ~ . -mpg, data= auto_df, kernel= 'linear', ranges = list(cost= c(0.001, 0.01, 0.1, 1, 5, 10, 100)))
summary(cv_svc)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 0.01
##
## - best performance: 0.08916667
##
## - Detailed performance results:
## cost error dispersion
## 1 1e-03 0.12775641 0.06746999
## 2 1e-02 0.08916667 0.05258186
## 3 1e-01 0.09160256 0.05869690
## 4 1e+00 0.09173077 0.04357345
## 5 5e+00 0.10942308 0.04734731
## 6 1e+01 0.11705128 0.05314992
## 7 1e+02 0.12993590 0.05797340Part c)
Now repeat (b), this time using SVMs with radial and polynomial
basis kernels, with different values of gamma and
degree and cost. Comment on your
results.
The SVM model with a radial kernel and 10-fold cross validation identified the best cost to be 1 and gamma as 1 with the lowest error rate of 0.07891026.
The SVM model with a polynomial kernel and 10-fold cross validation identified the best cost to be 1, gamma as 0.1, and degree as 3 with the lowest error rate of 0.08641026. Achieving a similar error rate as a radial kernel.
set.seed(42)
# Tune a SVM model with cross validation
cv_svm_rad = tune(svm, high_mpg ~ . -mpg, data= auto_df, kernel= 'radial',
ranges = list(cost= c(0.001, 0.01, 0.1, 1, 5, 10, 100),
gamma= c(0.01, 0.1, 1)))
summary(cv_svm_rad)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost gamma
## 1 1
##
## - best performance: 0.07891026
##
## - Detailed performance results:
## cost gamma error dispersion
## 1 1e-03 0.01 0.59679487 0.05312225
## 2 1e-02 0.01 0.59679487 0.05312225
## 3 1e-01 0.01 0.11493590 0.05837054
## 4 1e+00 0.01 0.08660256 0.05519479
## 5 5e+00 0.01 0.09166667 0.05754492
## 6 1e+01 0.01 0.08660256 0.06001684
## 7 1e+02 0.01 0.10442308 0.05805399
## 8 1e-03 0.10 0.59679487 0.05312225
## 9 1e-02 0.10 0.29326923 0.08606908
## 10 1e-01 0.10 0.08666667 0.05390785
## 11 1e+00 0.10 0.08660256 0.05519479
## 12 5e+00 0.10 0.08647436 0.03940028
## 13 1e+01 0.10 0.08647436 0.04445518
## 14 1e+02 0.10 0.10173077 0.04696584
## 15 1e-03 1.00 0.59679487 0.05312225
## 16 1e-02 1.00 0.59679487 0.05312225
## 17 1e-01 1.00 0.59679487 0.05312225
## 18 1e+00 1.00 0.07891026 0.03633038
## 19 5e+00 1.00 0.08910256 0.04132724
## 20 1e+01 1.00 0.08910256 0.04132724
## 21 1e+02 1.00 0.08910256 0.04132724set.seed(42)
# Tune a SVM model with cross validation
cv_svm_poly = tune(svm, high_mpg ~ . -mpg, data= auto_df,
kernel= 'polynomial',
ranges = list(cost= c(0.001, 0.01, 0.1, 1, 5, 10, 100),
gamma= c(0.01, 0.1, 1),
degree= c(2, 3, 4)))
summary(cv_svm_poly)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost gamma degree
## 100 0.1 3
##
## - best performance: 0.08641026
##
## - Detailed performance results:
## cost gamma degree error dispersion
## 1 1e-03 0.01 2 0.59679487 0.05312225
## 2 1e-02 0.01 2 0.59679487 0.05312225
## 3 1e-01 0.01 2 0.59679487 0.05312225
## 4 1e+00 0.01 2 0.58653846 0.05488085
## 5 5e+00 0.01 2 0.39064103 0.10386602
## 6 1e+01 0.01 2 0.31698718 0.09209637
## 7 1e+02 0.01 2 0.28083333 0.06885413
## 8 1e-03 0.10 2 0.59679487 0.05312225
## 9 1e-02 0.10 2 0.58653846 0.05488085
## 10 1e-01 0.10 2 0.31698718 0.09209637
## 11 1e+00 0.10 2 0.28083333 0.06885413
## 12 5e+00 0.10 2 0.16775641 0.10359086
## 13 1e+01 0.10 2 0.17294872 0.08470264
## 14 1e+02 0.10 2 0.19839744 0.07771274
## 15 1e-03 1.00 2 0.31698718 0.09209637
## 16 1e-02 1.00 2 0.28083333 0.06885413
## 17 1e-01 1.00 2 0.17294872 0.08470264
## 18 1e+00 1.00 2 0.19839744 0.07771274
## 19 5e+00 1.00 2 0.21121795 0.08538310
## 20 1e+01 1.00 2 0.22391026 0.09572557
## 21 1e+02 1.00 2 0.25717949 0.09485533
## 22 1e-03 0.01 3 0.59679487 0.05312225
## 23 1e-02 0.01 3 0.59679487 0.05312225
## 24 1e-01 0.01 3 0.59679487 0.05312225
## 25 1e+00 0.01 3 0.59679487 0.05312225
## 26 5e+00 0.01 3 0.36064103 0.14593226
## 27 1e+01 0.01 3 0.30166667 0.10288776
## 28 1e+02 0.01 3 0.25301282 0.07977961
## 29 1e-03 0.10 3 0.59679487 0.05312225
## 30 1e-02 0.10 3 0.30166667 0.10288776
## 31 1e-01 0.10 3 0.25301282 0.07977961
## 32 1e+00 0.10 3 0.09679487 0.05029680
## 33 5e+00 0.10 3 0.09929487 0.05393973
## 34 1e+01 0.10 3 0.09166667 0.05101515
## 35 1e+02 0.10 3 0.08641026 0.05301971
## 36 1e-03 1.00 3 0.09679487 0.05029680
## 37 1e-02 1.00 3 0.09166667 0.05101515
## 38 1e-01 1.00 3 0.08641026 0.05301971
## 39 1e+00 1.00 3 0.10179487 0.04394941
## 40 5e+00 1.00 3 0.10442308 0.03816415
## 41 1e+01 1.00 3 0.10442308 0.03816415
## 42 1e+02 1.00 3 0.10442308 0.03816415
## 43 1e-03 0.01 4 0.59679487 0.05312225
## 44 1e-02 0.01 4 0.59679487 0.05312225
## 45 1e-01 0.01 4 0.59679487 0.05312225
## 46 1e+00 0.01 4 0.59679487 0.05312225
## 47 5e+00 0.01 4 0.59679487 0.05312225
## 48 1e+01 0.01 4 0.59679487 0.05312225
## 49 1e+02 0.01 4 0.41666667 0.13285341
## 50 1e-03 0.10 4 0.59679487 0.05312225
## 51 1e-02 0.10 4 0.41666667 0.13285341
## 52 1e-01 0.10 4 0.34256410 0.11428217
## 53 1e+00 0.10 4 0.27314103 0.06981031
## 54 5e+00 0.10 4 0.19602564 0.07480024
## 55 1e+01 0.10 4 0.21929487 0.08090842
## 56 1e+02 0.10 4 0.20416667 0.05433005
## 57 1e-03 1.00 4 0.21929487 0.08090842
## 58 1e-02 1.00 4 0.20416667 0.05433005
## 59 1e-01 1.00 4 0.21410256 0.06210638
## 60 1e+00 1.00 4 0.22166667 0.07114891
## 61 5e+00 1.00 4 0.22166667 0.07114891
## 62 1e+01 1.00 4 0.22166667 0.07114891
## 63 1e+02 1.00 4 0.22166667 0.07114891Part d)
Make some plots to back up your assertions in (b) and (c).
Hint: In the lab, we used the plot() function for
svm objects only in cases with \(p = 2\). When \(p
> 2\), you can use the plot() function to create
plots displaying pairs of variables at a time. Essentially, instead of
typing
> plot(svmfit, dat)
where svmfit
contains your fitted model and dat is a data frame
containing your data, you can type
> plot(svmfit, dat, \(x_1 ∼
x_4\))
in order to plot just the first and fourth
variables. However, you must replace \(x_1\) and \(x_4\) with the correct variable names. To
find out more, type ?plot.svm.
# Subset data to only two predictors
auto_df_subset = auto_df[, c("horsepower", "weight", "high_mpg")]
# Re-fit model with best parameters for SVC
best_svc = svm(
high_mpg ~ ., data = auto_df_subset,
kernel = "linear",
cost = cv_svc$best.parameters$cost
)
# Re-fit model with best parameters for SVM radial
best_rad = svm(
high_mpg ~ ., data = auto_df_subset,
kernel = "radial",
cost = cv_svm_rad$best.parameters$cost,
gamma = cv_svm_rad$best.parameters$gamma
)
# Re-fit model with best parameters for SVM polynomial
best_poly = svm(
high_mpg ~ ., data = auto_df_subset,
kernel = "polynomial",
cost = cv_svm_poly$best.parameters$cost,
gamma = cv_svm_poly$best.parameters$gamma,
degree = cv_svm_poly$best.parameters$degree
)
# Plot the models
plot(best_svc, data= auto_df_subset)
plot(best_rad, data= auto_df_subset)
plot(best_poly, data= auto_df_subset)
Question 8
This problem involves the OJ data set which is part
of the ISLR2 package.
Part a)
Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.
set.seed(42)
# Read in the data
oj_df = OJ
head(oj_df)## Purchase WeekofPurchase StoreID PriceCH PriceMM DiscCH DiscMM SpecialCH
## 1 CH 237 1 1.75 1.99 0.00 0.0 0
## 2 CH 239 1 1.75 1.99 0.00 0.3 0
## 3 CH 245 1 1.86 2.09 0.17 0.0 0
## 4 MM 227 1 1.69 1.69 0.00 0.0 0
## 5 CH 228 7 1.69 1.69 0.00 0.0 0
## 6 CH 230 7 1.69 1.99 0.00 0.0 0
## SpecialMM LoyalCH SalePriceMM SalePriceCH PriceDiff Store7 PctDiscMM
## 1 0 0.500000 1.99 1.75 0.24 No 0.000000
## 2 1 0.600000 1.69 1.75 -0.06 No 0.150754
## 3 0 0.680000 2.09 1.69 0.40 No 0.000000
## 4 0 0.400000 1.69 1.69 0.00 No 0.000000
## 5 0 0.956535 1.69 1.69 0.00 Yes 0.000000
## 6 1 0.965228 1.99 1.69 0.30 Yes 0.000000
## PctDiscCH ListPriceDiff STORE
## 1 0.000000 0.24 1
## 2 0.000000 0.24 1
## 3 0.091398 0.23 1
## 4 0.000000 0.00 1
## 5 0.000000 0.00 0
## 6 0.000000 0.30 0# Split data into train and test
train_index = sample(1:nrow(oj_df), 800)
train = oj_df[train_index, ]
test = oj_df[-train_index, ]Part b)
Fit a support vector classifer to the training data using
cost = 0.01, with Purchase as the response and
the other variables as predictors. Use the summary()
function to produce summary statistics, and describe the results
obtained.
The SVC model created 435 vectors. Out of those, 215 vectors belong to level CH and 217 vectors belong to level MM.
# Fit a SVC model
fit_svc = svm(Purchase ~ ., data= train, kernel= 'linear', cost= 0.01)
summary(fit_svc)##
## Call:
## svm(formula = Purchase ~ ., data = train, kernel = "linear", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 432
##
## ( 215 217 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMPart c)
What are the training and test error rates?
Train error: 17.12%
Test error: 16.3%
# Calculate the train error rate
preds = predict(fit_svc, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 432 77
## MM 60 231
##
## Accuracy : 0.8288
## 95% CI : (0.8008, 0.8542)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.6346
##
## Mcnemar's Test P-Value : 0.1716
##
## Sensitivity : 0.7500
## Specificity : 0.8780
## Pos Pred Value : 0.7938
## Neg Pred Value : 0.8487
## Prevalence : 0.3850
## Detection Rate : 0.2888
## Detection Prevalence : 0.3638
## Balanced Accuracy : 0.8140
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(fit_svc, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 142 25
## MM 19 84
##
## Accuracy : 0.837
## 95% CI : (0.7875, 0.879)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.6585
##
## Mcnemar's Test P-Value : 0.451
##
## Sensitivity : 0.7706
## Specificity : 0.8820
## Pos Pred Value : 0.8155
## Neg Pred Value : 0.8503
## Prevalence : 0.4037
## Detection Rate : 0.3111
## Detection Prevalence : 0.3815
## Balanced Accuracy : 0.8263
##
## 'Positive' Class : MM
## Part d)
Use the tune() function to select an optimal
cost. Consider values in the range 0.01 to 10.
set.seed(42)
# Tune a SVC model with cross validation
cv_svc = tune(svm, Purchase ~ ., data= train, kernel= 'linear', ranges = list(cost= c(0.01, 0.1, 1, 5, 10)))
summary(cv_svc)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.175
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.17750 0.02415229
## 2 0.10 0.17625 0.03356689
## 3 1.00 0.17500 0.02886751
## 4 5.00 0.18375 0.02703521
## 5 10.00 0.18625 0.02729087Part e)
Compute the training and test error rates using this new value
for cost.
Train error: 16.75%
Test error: 16.3%
# Re-fit model with best parameters for SVC
best_svc = svm(
Purchase ~ ., data = train,
kernel = "linear",
cost = cv_svc$best.parameters$cost
)
# Calculate the train error rate
preds = predict(best_svc, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 434 76
## MM 58 232
##
## Accuracy : 0.8325
## 95% CI : (0.8048, 0.8577)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.6424
##
## Mcnemar's Test P-Value : 0.1419
##
## Sensitivity : 0.7532
## Specificity : 0.8821
## Pos Pred Value : 0.8000
## Neg Pred Value : 0.8510
## Prevalence : 0.3850
## Detection Rate : 0.2900
## Detection Prevalence : 0.3625
## Balanced Accuracy : 0.8177
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(best_svc, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 140 23
## MM 21 86
##
## Accuracy : 0.837
## 95% CI : (0.7875, 0.879)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 0.6605
##
## Mcnemar's Test P-Value : 0.8802
##
## Sensitivity : 0.7890
## Specificity : 0.8696
## Pos Pred Value : 0.8037
## Neg Pred Value : 0.8589
## Prevalence : 0.4037
## Detection Rate : 0.3185
## Detection Prevalence : 0.3963
## Balanced Accuracy : 0.8293
##
## 'Positive' Class : MM
## Part f)
Repeat parts (b) through (e) using a support vector machine with
a radial kernel. Use the default value for gamma.
The SVC model created 621 vectors. Out of those, 308 vectors belong to level CH and 313 vectors belong to level MM.
# Fit a SVM model
fit_svm_rad = svm(Purchase ~ ., data= train, kernel= 'radial', cost= 0.01)
summary(fit_svm_rad)##
## Call:
## svm(formula = Purchase ~ ., data = train, kernel = "radial", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: radial
## cost: 0.01
##
## Number of Support Vectors: 621
##
## ( 308 313 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMTrain error: 38.5%
Test error: 40.37%
# Calculate the train error rate
preds = predict(fit_svm_rad, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 492 308
## MM 0 0
##
## Accuracy : 0.615
## 95% CI : (0.5803, 0.6489)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : 0.5156
##
## Kappa : 0
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.000
## Specificity : 1.000
## Pos Pred Value : NaN
## Neg Pred Value : 0.615
## Prevalence : 0.385
## Detection Rate : 0.000
## Detection Prevalence : 0.000
## Balanced Accuracy : 0.500
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(fit_svm_rad, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 161 109
## MM 0 0
##
## Accuracy : 0.5963
## 95% CI : (0.5351, 0.6553)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : 0.5263
##
## Kappa : 0
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.0000
## Specificity : 1.0000
## Pos Pred Value : NaN
## Neg Pred Value : 0.5963
## Prevalence : 0.4037
## Detection Rate : 0.0000
## Detection Prevalence : 0.0000
## Balanced Accuracy : 0.5000
##
## 'Positive' Class : MM
## set.seed(42)
# Tune a SVM model with cross validation
cv_svm_rad = tune(svm, Purchase ~ ., data= train, kernel= 'radial', ranges = list(cost= c(0.01, 0.1, 1, 5, 10)))
summary(cv_svm_rad)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.18
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01 0.38500 0.04199868
## 2 0.10 0.18125 0.03784563
## 3 1.00 0.18000 0.03343734
## 4 5.00 0.18625 0.03701070
## 5 10.00 0.19375 0.03738408Train error: 15%
Test error: 15.93%
# Re-fit model with best parameters for SVM
best_svm_rad = svm(
Purchase ~ ., data = train,
kernel = "radial",
cost = cv_svm_rad$best.parameters$cost
)
# Calculate the train error rate
preds = predict(best_svm_rad, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 453 81
## MM 39 227
##
## Accuracy : 0.85
## 95% CI : (0.8233, 0.874)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.675
##
## Mcnemar's Test P-Value : 0.000182
##
## Sensitivity : 0.7370
## Specificity : 0.9207
## Pos Pred Value : 0.8534
## Neg Pred Value : 0.8483
## Prevalence : 0.3850
## Detection Rate : 0.2838
## Detection Prevalence : 0.3325
## Balanced Accuracy : 0.8289
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(best_svm_rad, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 146 28
## MM 15 81
##
## Accuracy : 0.8407
## 95% CI : (0.7915, 0.8823)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : < 2e-16
##
## Kappa : 0.6627
##
## Mcnemar's Test P-Value : 0.06725
##
## Sensitivity : 0.7431
## Specificity : 0.9068
## Pos Pred Value : 0.8438
## Neg Pred Value : 0.8391
## Prevalence : 0.4037
## Detection Rate : 0.3000
## Detection Prevalence : 0.3556
## Balanced Accuracy : 0.8250
##
## 'Positive' Class : MM
## Part g)
Repeat parts (b) through (e) using a support vector machine with
a polynomial kernel. Set degree = 2.
The SVC model created 621 vectors. Out of those, 308 vectors belong to level CH and 313 vectors belong to level MM.
# Fit a SVM model
fit_svm_poly = svm(Purchase ~ ., data= train, kernel= 'polynomial', cost= 0.01, degree= 2)
summary(fit_svm_poly)##
## Call:
## svm(formula = Purchase ~ ., data = train, kernel = "polynomial",
## cost = 0.01, degree = 2)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: polynomial
## cost: 0.01
## degree: 2
## coef.0: 0
##
## Number of Support Vectors: 621
##
## ( 308 313 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMTrain error: 38.5%
Test error: 40.37%
# Calculate the train error rate
preds = predict(fit_svm_poly, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 492 308
## MM 0 0
##
## Accuracy : 0.615
## 95% CI : (0.5803, 0.6489)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : 0.5156
##
## Kappa : 0
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.000
## Specificity : 1.000
## Pos Pred Value : NaN
## Neg Pred Value : 0.615
## Prevalence : 0.385
## Detection Rate : 0.000
## Detection Prevalence : 0.000
## Balanced Accuracy : 0.500
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(fit_svm_poly, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 161 109
## MM 0 0
##
## Accuracy : 0.5963
## 95% CI : (0.5351, 0.6553)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : 0.5263
##
## Kappa : 0
##
## Mcnemar's Test P-Value : <2e-16
##
## Sensitivity : 0.0000
## Specificity : 1.0000
## Pos Pred Value : NaN
## Neg Pred Value : 0.5963
## Prevalence : 0.4037
## Detection Rate : 0.0000
## Detection Prevalence : 0.0000
## Balanced Accuracy : 0.5000
##
## 'Positive' Class : MM
## set.seed(42)
# Tune a SVM model with cross validation
cv_svm_poly = tune(svm, Purchase ~ ., data= train, kernel= 'polynomial', ranges = list(cost= c(0.01, 0.1, 1, 5, 10), degree= 2))
summary(cv_svm_poly)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost degree
## 5 2
##
## - best performance: 0.18375
##
## - Detailed performance results:
## cost degree error dispersion
## 1 0.01 2 0.38625 0.04308019
## 2 0.10 2 0.31625 0.05529278
## 3 1.00 2 0.19250 0.04216370
## 4 5.00 2 0.18375 0.04041881
## 5 10.00 2 0.19000 0.03425801Train error: 14.75%
Test error: 16.67%
# Re-fit model with best parameters for SVM
best_svm_poly = svm(
Purchase ~ ., data = train,
kernel = 'polynomial',
cost = cv_svm_poly$best.parameters$cost,
degree = cv_svm_poly$best.parameters$degree
)
# Calculate the train error rate
preds = predict(best_svm_poly, train)
confusionMatrix(preds, train$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 459 85
## MM 33 223
##
## Accuracy : 0.8525
## 95% CI : (0.826, 0.8764)
## No Information Rate : 0.615
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.6784
##
## Mcnemar's Test P-Value : 2.667e-06
##
## Sensitivity : 0.7240
## Specificity : 0.9329
## Pos Pred Value : 0.8711
## Neg Pred Value : 0.8438
## Prevalence : 0.3850
## Detection Rate : 0.2787
## Detection Prevalence : 0.3200
## Balanced Accuracy : 0.8285
##
## 'Positive' Class : MM
## # Calculate the test error rate
preds = predict(best_svm_poly, test)
confusionMatrix(preds, test$Purchase, positive= 'MM')## Confusion Matrix and Statistics
##
## Reference
## Prediction CH MM
## CH 146 30
## MM 15 79
##
## Accuracy : 0.8333
## 95% CI : (0.7834, 0.8758)
## No Information Rate : 0.5963
## P-Value [Acc > NIR] : < 2e-16
##
## Kappa : 0.646
##
## Mcnemar's Test P-Value : 0.03689
##
## Sensitivity : 0.7248
## Specificity : 0.9068
## Pos Pred Value : 0.8404
## Neg Pred Value : 0.8295
## Prevalence : 0.4037
## Detection Rate : 0.2926
## Detection Prevalence : 0.3481
## Balanced Accuracy : 0.8158
##
## 'Positive' Class : MM
## Part h)
Overall, which approach seems to give the best results on this data?
The best test error produced was 15.93% by the SVM model using a radial kernel.