question 5 Part a and b:

set.seed(5)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- as.numeric(x1^2 - x2^2 > 0)
dat <- data.frame(x1 = x1, x2 = x2, y = as.factor(y))
plot(dat$x1, dat$x2, col = ifelse(dat$y == 1, "red", "blue"),
     pch = 19, xlab = "X1", ylab = "X2",
     main = "True class labels")

I simulated a synthetic data set of n = 500 observations with two predictors, X1 and X2, each drawn from a Uniform(−0.5, 0.5) distribution. Class labels were assigned based on the rule X1² − X2² > 0, creating a quadratic (non-linear) decision boundary between the two classes rather than a linear one. No external data set or R package was required, since the data was generated directly with base R (runif) and plotted with the base plot() function. The scatterplot confirms this: the red and blue classes are separated by two diagonal boundaries (X2 = X1 and X2 = −X1), a shape that cannot be captured by a single straight line, this sets up the comparison later between linear and non-linear classifiers.

part c : Fit logistic regression with X1 and X2 as predictors

glm.fit <- glm(y ~ x1 + x2, data = dat, family = "binomial")
summary(glm.fit)
## 
## Call:
## glm(formula = y ~ x1 + x2, family = "binomial", data = dat)
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.03150    0.08949  -0.352    0.725
## x1          -0.06176    0.30506  -0.202    0.840
## x2          -0.11509    0.31086  -0.370    0.711
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 693.02  on 499  degrees of freedom
## Residual deviance: 692.85  on 497  degrees of freedom
## AIC: 698.85
## 
## Number of Fisher Scoring iterations: 3

A logistic regression model was fit using only the raw predictors X1 and X2. Neither coefficient was statistically significant (p = 0.840 and p = 0.711), and the residual deviance (692.85) was nearly identical to the null deviance (693.02), indicating the model has essentially no predictive power. This is expected: because the true decision boundary is quadratic (X1² − X2² > 0), a linear combination of X1 and X2 alone cannot capture the relationship.

part d : Predict class labels and plot

glm.probs <- predict(glm.fit, type = "response")
glm.pred <- ifelse(glm.probs > 0.5, 1, 0)

plot(dat$x1, dat$x2, col = ifelse(glm.pred == 1, "red", "blue"),
     pch = 19, xlab = "X1", ylab = "X2",
     main = "Logistic Regression (linear terms) - Predicted labels")

Using the fitted logistic regression model, predicted class labels were generated for each training observation and plotted. As expected, the predicted decision boundary is linear the model separates the data along a roughly horizontal line in X2, rather than capturing the true quadratic (bowtie-shaped) boundary. This confirms that logistic regression restricted to linear terms cannot model a non-linear relationship between the predictors and the response.

part e: Fit logistic regression with non-linear terms ˆ

glm.fit2 <- glm(y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2), 
                 data = dat, family = "binomial")
## Warning: glm.fit: algorithm did not converge
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
summary(glm.fit2)
## 
## Call:
## glm(formula = y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2), family = "binomial", 
##     data = dat)
## 
## Coefficients:
##                Estimate Std. Error    z value Pr(>|z|)    
## (Intercept)   1.005e+14  3.016e+06   33331405   <2e-16 ***
## poly(x1, 2)1  2.256e+15  6.748e+07   33427335   <2e-16 ***
## poly(x1, 2)2  3.947e+16  6.823e+07  578468154   <2e-16 ***
## poly(x2, 2)1  5.312e+14  6.750e+07    7870343   <2e-16 ***
## poly(x2, 2)2 -4.563e+16  6.770e+07 -674041375   <2e-16 ***
## I(x1 * x2)   -5.007e+14  3.411e+07  -14676275   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance:  693.02  on 499  degrees of freedom
## Residual deviance: 2378.88  on 494  degrees of freedom
## AIC: 2390.9
## 
## Number of Fisher Scoring iterations: 25

Adding quadratic terms (X1², X2²) and the interaction (X1×X2) allowed the logistic regression model to almost perfectly separate the two classes, since the true boundary is itself quadratic in X1 and X2. This resulted in complete separation, causing the algorithm to fail to converge and produce extremely large coefficient estimates a known artifact of complete separation rather than an error. Despite this, the fitted model can still be used to generate predicted class labels, which we expect to reveal a non-linear decision boundary.

part f:predict and plot using this model:

glm.probs2 <- predict(glm.fit2, type = "response")
glm.pred2 <- ifelse(glm.probs2 > 0.5, 1, 0)

plot(dat$x1, dat$x2, col = ifelse(glm.pred2 == 1, "red", "blue"),
     pch = 19, xlab = "X1", ylab = "X2",
     main = "Logistic Regression (non-linear terms) - Predicted labels")

Using the logistic regression model with quadratic and interaction terms, predicted class labels were generated and plotted. Unlike the linear model in part (d), this plot closely resembles the true class labels from part (b), correctly capturing the diagonal, crossing (bowtie-shaped) decision boundary. This confirms that logistic regression can approximate non-linear decision boundaries when the input features themselves are transformed non-linearly, even though the underlying model is still linear in its (transformed) parameters.

Part g: fit a support vector classifier (linear kernel)

library(e1071)
svm.fit <- svm(y ~ x1 + x2, data = dat, kernel = "linear", cost = 1)
svm.pred <- predict(svm.fit, dat)

plot(dat$x1, dat$x2, col = ifelse(svm.pred == 1, "red", "blue"),
     pch = 19, xlab = "X1", ylab = "X2",
     main = "SVM (linear kernel) - Predicted labels")

A support vector classifier with a linear kernel was fit to the data using X1 and X2 as predictors. The resulting predictions assigned every observation to a single class, illustrating that no linear boundary can meaningfully separate the two classes given the quadratic (bowtie-shaped) true decision boundary. This reinforces the finding from part (d): linear classifiers are fundamentally unable to capture this non-linear relationship, regardless of whether logistic regression or a support vector classifier is used.

part h :SVM with radial kernel

svm.fit2 <- svm(y ~ x1 + x2, data = dat, kernel = "radial", gamma = 1, cost = 1)
svm.pred2 <- predict(svm.fit2, dat)

plot(dat$x1, dat$x2, col = ifelse(svm.pred2 == 1, "red", "blue"),
     pch = 19, xlab = "X1", ylab = "X2",
     main = "SVM (radial kernel) - Predicted labels")

A support vector machine with a radial kernel was fit to the data using X1 and X2 as predictors. The resulting predicted labels closely match the true class labels, correctly capturing the diagonal, crossing decision boundary. Unlike the linear SVM, the radial kernel was able to model the non-linear relationship automatically, without requiring manual feature engineering.

part i :Overall comparison:

This exercise illustrates two different strategies for handling non-linear decision boundaries. When restricted to linear terms, both logistic regression (part d) and the support vector classifier (part g) failed to capture the true quadratic boundary logistic regression settled on a single roughly horizontal split, while the linear SVM classified nearly all points into a single class, since no straight line can meaningfully separate the two classes. In contrast, both non-linear approaches succeeded: logistic regression with manually added quadratic and interaction terms (part e/f) and the SVM with a radial kernel (part h) both closely reproduced the true bowtie-shaped decision boundary. The key difference between these two successful approaches is that logistic regression required us to explicitly engineer the non-linear features (X1², X2², X1×X2) ourselves, whereas the radial-kernel SVM captured the non-linearity automatically through the kernel trick, without any manual feature transformation. This demonstrates the practical advantage of kernel-based methods: they can model complex, non-linear decision boundaries more flexibly and with less manual effort than transforming features by hand for a linear model.

Question 7, part a :

library(ISLR2)
library(e1071)

data(Auto)

# (a) Create binary variable: 1 = above median mpg, 0 = below median mpg
Auto$mpglevel <- as.factor(ifelse(Auto$mpg > median(Auto$mpg), 1, 0))

# quick check
table(Auto$mpglevel)
## 
##   0   1 
## 196 196

A binary response variable, mpglevel, was created by comparing each car’s mpg to the median mpg of the data set. Cars above the median were coded as 1 (high mileage) and those at or below as 0 (low mileage). As expected from a median split, this produced a balanced classification problem with 196 observations in each class.

part b : upport vector classifier (linear kernel) with various costs, using CV

set.seed(1)
tune.out <- tune(svm, mpglevel ~ . - mpg, data = Auto, kernel = "linear",
                  ranges = list(cost = c(0.001, 0.01, 0.1, 1, 5, 10, 100)))
summary(tune.out)
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##   0.1
## 
## - best performance: 0.08673077 
## 
## - Detailed performance results:
##    cost      error dispersion
## 1 1e-03 0.13525641 0.05661708
## 2 1e-02 0.08923077 0.04698309
## 3 1e-01 0.08673077 0.04040897
## 4 1e+00 0.09961538 0.04923181
## 5 5e+00 0.11230769 0.05826857
## 6 1e+01 0.11237179 0.05701890
## 7 1e+02 0.11750000 0.06208951

A support vector classifier with a linear kernel was tuned over a range of cost values (0.001 to 100) using 10-fold cross-validation, with mpglevel as the response and all other variables (excluding mpg itself) as predictors. The lowest cross-validation error rate, approximately 8.67%, was achieved at cost = 0.1. Error rates were higher for very small cost values (under-fitting, wide margin) and increased again for larger cost values (over-fitting, narrow margin), indicating that a moderate cost value provides the best generalization for this data.

part c : SVM with radial kernel - tune over cost and gamma

set.seed(1)
tune.out.radial <- tune(svm, mpglevel ~ . - mpg, data = Auto, kernel = "radial",
                          ranges = list(cost = c(0.1, 1, 5, 10, 100),
                                        gamma = c(0.01, 0.1, 1, 5, 10)))
tune.out.radial$best.parameters
##    cost gamma
## 14   10     1
tune.out.radial$best.performance
## [1] 0.07897436

A support vector machine with a radial kernel was tuned over cost values (0.1 to 100) and gamma values (0.01 to 10) using 10-fold cross-validation. The best combination was cost = 10 and gamma = 1, yielding a cross-validation error rate of approximately 7.90%. This is slightly lower than the best linear-kernel error rate (8.67%), suggesting the radial kernel captures a bit of non-linear structure in the relationship between the predictors and mileage class that the linear kernel misses.

SVM with polynomial kernel - tune over cost and degree

set.seed(1)
tune.out.poly <- tune(svm, mpglevel ~ . - mpg, data = Auto, kernel = "polynomial",
                        ranges = list(cost = c(0.1, 1, 5, 10, 100),
                                      degree = c(2, 3, 4)))
tune.out.poly$best.parameters
##   cost degree
## 5  100      2
tune.out.poly$best.performance
## [1] 0.3167308

A support vector machine with a polynomial kernel was tuned over cost values (0.1 to 100) and degree values (2, 3, 4) using 10-fold cross-validation. The best combination was cost = 100 and degree = 2, but this yielded a cross-validation error rate of approximately 31.7% — substantially worse than both the linear kernel (8.67%) and radial kernel (7.90%). This suggests that, within the range of parameters tested, the polynomial kernel is not well suited to capturing the relationship between the predictors and mileage class for this data set, at least without more extensive tuning.

Comparing all three kernels, the radial kernel achieved the lowest cross-validation error (7.90%), followed closely by the linear kernel (8.67%), while the polynomial kernel performed noticeably worse (31.7%). This suggests that any non-linearity in the true relationship between car characteristics and mileage class is better captured by the radial kernel’s local, flexible boundary than by a fixed-degree polynomial transformation.

part d :Refit best models - Plot using a pair of predictors

Auto3 <- Auto[, !(names(Auto) %in% c("name", "mpg"))]

svm.linear <- svm(mpglevel ~ ., data = Auto3, kernel = "linear", cost = 0.1)
svm.radial <- svm(mpglevel ~ ., data = Auto3, kernel = "radial", cost = 10, gamma = 1)
svm.poly   <- svm(mpglevel ~ ., data = Auto3, kernel = "polynomial", cost = 100, degree = 2)

plot(svm.linear, Auto3, horsepower ~ weight)

plot(svm.radial, Auto3, horsepower ~ weight)

plot(svm.poly, Auto3, horsepower ~ weight)

Classification plots of horsepower vs. weight were produced for the tuned linear, radial, and polynomial kernel SVMs. The linear kernel shows a clean, interpretable boundary consistent with the intuition that heavier, higher-horsepower cars tend to have lower mileage. The radial kernel produces a much tighter, localized decision region reflecting its flexibility, while the polynomial kernel’s plot is nearly uniform across the plotted range, consistent with its poor cross-validation performance. These visualizations support the numerical results: the linear and radial kernels produce meaningful, sensible decision boundaries, while the polynomial kernel struggles to separate the classes effectively.

Overall: Cross-validation identified cost = 0.1 as optimal for the linear kernel (error ≈ 8.67%), cost = 10 and gamma = 1 for the radial kernel (error ≈ 7.90%), and cost = 100 and degree = 2 for the polynomial kernel (error ≈ 31.7%). The radial kernel achieved the best overall performance, narrowly outperforming the linear kernel, while the polynomial kernel performed substantially worse. Visualizing the fitted models on the horsepower/weight plane supports these findings: both the linear and radial kernels produce sensible, well-separated decision regions consistent with the known relationship between vehicle size/power and fuel efficiency, while the polynomial kernel fails to meaningfully distinguish the classes.

question 8 : part a:

library(ISLR2)
library(e1071)

data(OJ)

set.seed(1)
train_idx <- sample(1:nrow(OJ), 800)
OJ.train <- OJ[train_idx, ]
OJ.test <- OJ[-train_idx, ]

dim(OJ.train)
## [1] 800  18
dim(OJ.test)
## [1] 270  18

The OJ data set (1070 observations) was split into a training set of 800 randomly sampled observations and a test set containing the remaining 270 observations.

part b :Support vector classifier with cost = 0.01

svm.linear <- svm(Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
summary(svm.linear)
## 
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.01)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.01 
## 
## Number of Support Vectors:  435
## 
##  ( 219 216 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM

A support vector classifier with a linear kernel and cost = 0.01 was fit to the training data, using Purchase as the response and all other variables as predictors. The model used 435 support vectors (219 for CH, 216 for MM) out of 800 training observations. This relatively large number of support vectors reflects the small cost value, which corresponds to a wide margin and a more heavily regularized (less strict) classifier.

part c : Training and test error rates

train.pred <- predict(svm.linear, OJ.train)
test.pred <- predict(svm.linear, OJ.test)

train.table <- table(Predicted = train.pred, Actual = OJ.train$Purchase)
test.table <- table(Predicted = test.pred, Actual = OJ.test$Purchase)

train.table
##          Actual
## Predicted  CH  MM
##        CH 420  75
##        MM  65 240
test.table
##          Actual
## Predicted  CH  MM
##        CH 153  33
##        MM  15  69
train.err <- mean(train.pred != OJ.train$Purchase)
test.err <- mean(test.pred != OJ.test$Purchase)

train.err
## [1] 0.175
test.err
## [1] 0.1777778

Using the linear-kernel support vector classifier with cost = 0.01, the training error rate was 17.5% (140/800 misclassified) and the test error rate was 17.78% (48/270 misclassified). The similarity between training and test error suggests the model generalizes reasonably well, without substantial overfitting.

part d :Tune cost using cross-validation

set.seed(1)
tune.out <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "linear",
                  ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
summary(tune.out)
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##   0.1
## 
## - best performance: 0.1725 
## 
## - Detailed performance results:
##    cost   error dispersion
## 1  0.01 0.17625 0.02853482
## 2  0.10 0.17250 0.03162278
## 3  1.00 0.17500 0.02946278
## 4  5.00 0.17250 0.03162278
## 5 10.00 0.17375 0.03197764
tune.out$best.parameters
##   cost
## 2  0.1
tune.out$best.performance
## [1] 0.1725

The support vector classifier was tuned over cost values ranging from 0.01 to 10 using 10-fold cross-validation. The lowest cross-validation error (17.25%) was achieved at cost = 0.1 (tied with cost = 5), a modest improvement over the cost = 0.01 model’s CV error of 17.625%.

Part e :Refit with optimal cost and compute training/test errors

svm.linear.best <- svm(Purchase ~ ., data = OJ.train, kernel = "linear", cost = 0.1)

train.pred.best <- predict(svm.linear.best, OJ.train)
test.pred.best <- predict(svm.linear.best, OJ.test)

train.err.best <- mean(train.pred.best != OJ.train$Purchase)
test.err.best <- mean(test.pred.best != OJ.test$Purchase)

train.err.best
## [1] 0.165
test.err.best
## [1] 0.162963

Using the optimal cost value of 0.1 selected via cross-validation, the linear-kernel support vector classifier achieved a training error rate of 16.5% and a test error rate of 16.30%. Both error rates improved compared to the cost = 0.01 model (17.5% train, 17.78% test), confirming that cross-validation successfully identified a better-performing value of cost.

part f :Radial kernel with cost = 0.01

svm.radial <- svm(Purchase ~ ., data = OJ.train, kernel = "radial", cost = 0.01)
summary(svm.radial)
## 
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "radial", cost = 0.01)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  0.01 
## 
## Number of Support Vectors:  634
## 
##  ( 319 315 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
train.pred.r <- predict(svm.radial, OJ.train)
test.pred.r <- predict(svm.radial, OJ.test)

train.err.r <- mean(train.pred.r != OJ.train$Purchase)
test.err.r <- mean(test.pred.r != OJ.test$Purchase)

train.err.r
## [1] 0.39375
test.err.r
## [1] 0.3777778

A radial-kernel SVM with cost = 0.01 was fit to the training data, using 634 support vectors (319 CH, 315 MM) — nearly 80% of the training observations. This model performed poorly, with a training error of 39.4% and test error of 37.8%, indicating that such a small cost value results in severe under-fitting for the radial kernel.

part e :Tune cost for radial kernel

set.seed(1)
tune.out.r <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "radial",
                     ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
summary(tune.out.r)
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##     1
## 
## - best performance: 0.17125 
## 
## - Detailed performance results:
##    cost   error dispersion
## 1  0.01 0.39375 0.04007372
## 2  0.10 0.18625 0.02853482
## 3  1.00 0.17125 0.02128673
## 4  5.00 0.18000 0.02220485
## 5 10.00 0.18625 0.02853482

The radial-kernel SVM was tuned over cost values from 0.01 to 10 using 10-fold cross-validation. The lowest cross-validation error (17.125%) was achieved at cost = 1. Error was substantially higher at very small cost values due to under-fitting, and increased again slightly at larger cost values, indicating mild overfitting.

Refit radial kernel with optimal cost and compute errors

svm.radial.best <- svm(Purchase ~ ., data = OJ.train, kernel = "radial", cost = 1)

train.pred.rb <- predict(svm.radial.best, OJ.train)
test.pred.rb <- predict(svm.radial.best, OJ.test)

train.err.rb <- mean(train.pred.rb != OJ.train$Purchase)
test.err.rb <- mean(test.pred.rb != OJ.test$Purchase)

train.err.rb
## [1] 0.15125
test.err.rb
## [1] 0.1851852

Using the optimal cost value of 1 selected via cross-validation, the radial-kernel SVM achieved a training error rate of 15.125% and a test error rate of 18.52%. While the training error is the lowest seen so far, the test error is slightly higher than that of the tuned linear-kernel model (16.30%), suggesting the radial kernel fits the training data more closely without necessarily improving generalization.

part g :Polynomial kernel

svm.poly <- svm(Purchase ~ ., data = OJ.train, kernel = "polynomial", degree = 2, cost = 0.01)
summary(svm.poly)
## 
## Call:
## svm(formula = Purchase ~ ., data = OJ.train, kernel = "polynomial", 
##     degree = 2, cost = 0.01)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  polynomial 
##        cost:  0.01 
##      degree:  2 
##      coef.0:  0 
## 
## Number of Support Vectors:  636
## 
##  ( 321 315 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
train.pred.p <- predict(svm.poly, OJ.train)
test.pred.p <- predict(svm.poly, OJ.test)

train.err.p <- mean(train.pred.p != OJ.train$Purchase)
test.err.p <- mean(test.pred.p != OJ.test$Purchase)

train.err.p
## [1] 0.3725
test.err.p
## [1] 0.3666667

A polynomial-kernel (degree = 2) SVM with cost = 0.01 was fit to the training data, using 636 support vectors (321 CH, 315 MM). This model performed poorly, with a training error of 37.25% and test error of 36.67%, again reflecting severe under-fitting at this very small cost value

Tune cost for polynomial kernel (degree = 2)

set.seed(1)
tune.out.p <- tune(svm, Purchase ~ ., data = OJ.train, kernel = "polynomial", degree = 2,
                     ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
summary(tune.out.p)
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##    10
## 
## - best performance: 0.18125 
## 
## - Detailed performance results:
##    cost   error dispersion
## 1  0.01 0.39125 0.04210189
## 2  0.10 0.32125 0.05001736
## 3  1.00 0.20250 0.04116363
## 4  5.00 0.18250 0.03496029
## 5 10.00 0.18125 0.02779513

The polynomial-kernel (degree = 2) SVM was tuned over cost values from 0.01 to 10 using 10-fold cross-validation. The lowest cross-validation error (18.125%) was achieved at cost = 10, the largest value tested. Error decreased steadily as cost increased throughout the tested range, indicating this kernel required less regularization (a larger cost) than the linear or radial kernels to fit the data well.

Refit polynomial kernel with optimal cost and compute errors

svm.poly.best <- svm(Purchase ~ ., data = OJ.train, kernel = "polynomial", degree = 2, cost = 10)

train.pred.pb <- predict(svm.poly.best, OJ.train)
test.pred.pb <- predict(svm.poly.best, OJ.test)

train.err.pb <- mean(train.pred.pb != OJ.train$Purchase)
test.err.pb <- mean(test.pred.pb != OJ.test$Purchase)

train.err.pb
## [1] 0.15
test.err.pb
## [1] 0.1888889

Using the optimal cost value of 10 selected via cross-validation, the polynomial-kernel (degree = 2) SVM achieved a training error rate of 15.0% and a test error rate of 18.89%. While this is the lowest training error among all three tuned models, it has the highest test error, suggesting this model fits the training data closely but generalizes slightly less well than the linear or radial kernels.

Overall: Comparing all three tuned SVM approaches on the OJ data set, the linear kernel achieved the best test error (16.30%), narrowly outperforming the radial kernel (18.52%) and the polynomial kernel (18.89%), even though the radial and polynomial kernels achieved marginally lower training errors (15.13% and 15.00%, respectively). This suggests that the relationship between the predictors and Purchase is close to linear, and that the additional flexibility offered by the radial and polynomial kernels does not translate into improved generalization for this data set if anything, it leads to slightly more overfitting. Overall, the linear-kernel support vector classifier appears to be the best choice for this classification problem.