STA 6543 Predictive Modeling
Homework 8
Haniya Farhat
July 26, 2026
Problem 5: Non-linear Decision Boundaries — Logistic Regression vs. SVM
(a) Generate a data set with a quadratic decision boundary
set.seed(1)
x1 <- runif(500) - 0.5
x2 <- runif(500) - 0.5
y <- 1 * (x1^2 - x2^2 > 0)
dat5 <- data.frame(x1 = x1, x2 = x2, y = as.factor(y))
table(y)## y
## 0 1
## 261 239(b) Plot the observations, colored by class
plot(x1, x2, col = ifelse(y == 1, "red", "blue"), pch = 19, cex = 0.6,
main = "True Classes (Quadratic Boundary)")
The two classes are separated by the curve \(x_1^2 = x_2^2\), i.e. the two diagonal lines \(x_2 = \pm x_1\) — a shape no straight line can reproduce.
(c) Logistic regression with linear terms
##
## Call:
## glm(formula = y ~ x1 + x2, family = binomial, data = dat5)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.087260 0.089579 -0.974 0.330
## x1 0.196199 0.316864 0.619 0.536
## x2 -0.002854 0.305712 -0.009 0.993
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 692.18 on 499 degrees of freedom
## Residual deviance: 691.79 on 497 degrees of freedom
## AIC: 697.79
##
## Number of Fisher Scoring iterations: 3(d) Predicted labels from the linear logistic model
prob_lin <- predict(glm_lin, type = "response")
pred_lin <- ifelse(prob_lin > 0.5, 1, 0)
err_lin <- mean(pred_lin != y)
plot(x1, x2, col = ifelse(pred_lin == 1, "red", "blue"), pch = 19, cex = 0.6,
main = "Logistic Regression (Linear Terms): Predicted Classes")
## Training error: 0.43With only x1 and x2 as linear predictors,
neither coefficient is close to significant and the fitted decision
boundary is a straight line, which is the wrong shape for this problem
entirely. The predicted-class plot confirms it: the model essentially
splits the plane in half rather than along the true \(x_2 = \pm x_1\) diagonals, and the training
error (0.43) is barely better than the roughly 50% error of a coin
flip.
(e) Logistic regression with non-linear terms
glm_nl <- glm(y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2), data = dat5, family = binomial)
summary(glm_nl)##
## Call:
## glm(formula = y ~ poly(x1, 2) + poly(x2, 2) + I(x1 * x2), family = binomial,
## data = dat5)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -102.2 4302.0 -0.024 0.981
## poly(x1, 2)1 2715.3 141109.5 0.019 0.985
## poly(x1, 2)2 27218.5 842987.2 0.032 0.974
## poly(x2, 2)1 -279.7 97160.4 -0.003 0.998
## poly(x2, 2)2 -28693.0 875451.3 -0.033 0.974
## I(x1 * x2) -206.4 41802.8 -0.005 0.996
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 6.9218e+02 on 499 degrees of freedom
## Residual deviance: 3.5810e-06 on 494 degrees of freedom
## AIC: 12
##
## Number of Fisher Scoring iterations: 25(f) Predicted labels from the non-linear logistic model
prob_nl <- predict(glm_nl, type = "response")
pred_nl <- ifelse(prob_nl > 0.5, 1, 0)
err_nl <- mean(pred_nl != y)
plot(x1, x2, col = ifelse(pred_nl == 1, "red", "blue"), pch = 19, cex = 0.6,
main = "Logistic Regression (Quadratic Terms): Predicted Classes")
## Training error: 0Once \(x_1^2\) and \(x_2^2\) terms are available, the model can reproduce the true boundary almost exactly — the predicted-class plot is visually indistinguishable from the true-class plot in (b), and the training error drops to 0. That makes sense: the true boundary is a quadratic function of \(x_1\) and \(x_2\), so a logistic regression with quadratic terms is the correct functional form here, not just a flexible approximation.
(g) Support vector classifier (linear kernel)
svm_lin <- svm(y ~ x1 + x2, data = dat5, kernel = "linear", cost = 0.1)
pred_svm_lin <- predict(svm_lin)
err_svm_lin <- mean(pred_svm_lin != y)
plot(x1, x2, col = ifelse(pred_svm_lin == 1, "red", "blue"), pch = 19, cex = 0.6,
main = "Linear SVM: Predicted Classes")
## Training error: 0.478Like the linear logistic model, a linear-kernel SVM can only carve the plane with a straight line, so it fails for the same structural reason: it predicts almost the entire training set into a single class here, giving a training error of 0.478 — no better than the linear logistic fit.
(h) SVM with a non-linear (radial) kernel
svm_rad <- svm(y ~ x1 + x2, data = dat5, kernel = "radial", gamma = 1, cost = 1)
pred_svm_rad <- predict(svm_rad)
err_svm_rad <- mean(pred_svm_rad != y)
plot(x1, x2, col = ifelse(pred_svm_rad == 1, "red", "blue"), pch = 19, cex = 0.6,
main = "Radial-Kernel SVM: Predicted Classes")
## Training error: 0.028The radial kernel lets the SVM bend its decision boundary to match the data, and it recovers the diagonal cross shape almost perfectly, with a training error of only 0.028.
(i) Comment on the results
The four fits split cleanly into two pairs. The linear
logistic regression and the linear-kernel SVM
are both restricted to a straight-line decision boundary, and both fail
badly (training error around 0.43–0.48) because the true boundary here
is quadratic, not linear — no amount of re-fitting a straight line will
fix a mismatched functional form. The logistic regression with
quadratic terms and the radial-kernel SVM, on
the other hand, can both represent curved boundaries and both
essentially recover the true \(x_1^2 =
x_2^2\) boundary, with training errors near 0. The broader point
is exactly the one this exercise is built to make: a linear method can
be made to handle a non-linear problem if you hand-engineer the right
non-linear features (here, adding x1^2, x2^2,
and the interaction term), while a kernel SVM gets non-linearity “for
free” through the kernel trick without needing anyone to guess the right
transformation in advance — which matters a lot once the true boundary’s
shape isn’t already known.
Problem 7: Predicting Gas Mileage in the Auto Data Set
(a) Create a binary high/low mileage variable
data(Auto)
Auto$mpg01 <- as.factor(ifelse(Auto$mpg > median(Auto$mpg), 1, 0))
Auto2 <- Auto[, !(names(Auto) %in% c("mpg", "name"))]
table(Auto$mpg01)##
## 0 1
## 196 196mpg itself and name are dropped from the
predictor set — mpg because it was used to build the
response and would leak the answer directly, and name
because it’s just a car-specific label with no generalizable signal.
(b) Support vector classifier (linear kernel) across a range of cost
set.seed(1)
tune_lin7 <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "linear",
ranges = list(cost = c(0.01, 0.1, 1, 5, 10, 100)))
summary(tune_lin7)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 1
##
## - best performance: 0.08435897
##
## - Detailed performance results:
## cost error dispersion
## 1 1e-02 0.08923077 0.04698309
## 2 1e-01 0.09185897 0.04393409
## 3 1e+00 0.08435897 0.03662670
## 4 5e+00 0.08948718 0.03898410
## 5 1e+01 0.08948718 0.03898410
## 6 1e+02 0.08692308 0.03887151best_cost_lin7 <- tune_lin7$best.parameters$cost
cat("Best cost:", best_cost_lin7, " CV error:", round(min(tune_lin7$performances$error), 4), "\n")## Best cost: 1 CV error: 0.0844Across costs spanning four orders of magnitude, the 10-fold CV error
barely moves — it stays in a tight band of roughly 0.084 to 0.092,
bottoming out at cost = 1. That flatness suggests mpg01 is
close to linearly separable from the other Auto predictors
(mainly displacement, horsepower, and
weight, which are all strongly tied to fuel economy), so
the margin isn’t very sensitive to exactly how much the classifier is
allowed to violate it.
(c) SVMs with radial and polynomial kernels
set.seed(1)
tune_rad7 <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "radial",
ranges = list(cost = c(0.1, 1, 5, 10, 100), gamma = c(0.01, 0.1, 1, 5)))
summary(tune_rad7)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost gamma
## 1 1
##
## - best performance: 0.06634615
##
## - Detailed performance results:
## cost gamma error dispersion
## 1 0.1 0.01 0.11224359 0.03836937
## 2 1.0 0.01 0.08673077 0.04551036
## 3 5.0 0.01 0.08416667 0.04010502
## 4 10.0 0.01 0.08673077 0.04040897
## 5 100.0 0.01 0.08685897 0.03483004
## 6 0.1 0.10 0.08923077 0.04698309
## 7 1.0 0.10 0.08923077 0.04376306
## 8 5.0 0.10 0.07910256 0.03292568
## 9 10.0 0.10 0.08166667 0.04149504
## 10 100.0 0.10 0.08410256 0.03390616
## 11 0.1 1.00 0.08673077 0.04535158
## 12 1.0 1.00 0.06634615 0.03244101
## 13 5.0 1.00 0.08916667 0.02708952
## 14 10.0 1.00 0.08923077 0.02732003
## 15 100.0 1.00 0.10448718 0.04560852
## 16 0.1 5.00 0.54602564 0.05105434
## 17 1.0 5.00 0.09173077 0.03417795
## 18 5.0 5.00 0.09423077 0.04945093
## 19 10.0 5.00 0.09173077 0.04983028
## 20 100.0 5.00 0.09429487 0.05109336## Best radial params:## cost gamma
## 12 1 1## Best radial CV error: 0.0663set.seed(1)
tune_poly7 <- tune(svm, mpg01 ~ ., data = Auto2, kernel = "polynomial",
ranges = list(cost = c(0.1, 1, 5, 10, 100), degree = c(2, 3, 4)))
summary(tune_poly7)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost degree
## 100 3
##
## - best performance: 0.08423077
##
## - Detailed performance results:
## cost degree error dispersion
## 1 0.1 2 0.27846154 0.09486227
## 2 1.0 2 0.25307692 0.13751948
## 3 5.0 2 0.17621795 0.04945319
## 4 10.0 2 0.18647436 0.05598001
## 5 100.0 2 0.18128205 0.06251437
## 6 0.1 3 0.20192308 0.11347783
## 7 1.0 3 0.09448718 0.04180527
## 8 5.0 3 0.08429487 0.04016554
## 9 10.0 3 0.08435897 0.04544023
## 10 100.0 3 0.08423077 0.03636273
## 11 0.1 4 0.26564103 0.09977887
## 12 1.0 4 0.21205128 0.09560470
## 13 5.0 4 0.18371795 0.06175709
## 14 10.0 4 0.16589744 0.06962914
## 15 100.0 4 0.12756410 0.05208506## Best polynomial params:## cost degree
## 10 100 3## Best polynomial CV error: 0.0842results7 <- data.frame(
Kernel = c("Linear", "Radial", "Polynomial"),
CV_Error = round(c(min(tune_lin7$performances$error),
tune_rad7$best.performance,
tune_poly7$best.performance), 4)
)
print(results7)## Kernel CV_Error
## 1 Linear 0.0844
## 2 Radial 0.0663
## 3 Polynomial 0.0842The radial kernel comes out slightly ahead of the other two, with a best CV error of 0.0663 at cost = 1, gamma = 1, versus 0.0844 for the best linear fit and 0.0842 for the best polynomial fit (cost = 100, degree = 3). The polynomial kernel is noticeably more sensitive to its tuning parameters than the other two — low-cost, high-degree combinations perform much worse, since a high-degree polynomial at low cost is either too wiggly or too weakly fit. Still, all three kernels land within about 1.8 percentage points of each other, reinforcing the same read as part (b): the boundary between high- and low-mileage cars in this feature space is close enough to linear that a non-linear kernel only buys a small improvement.
(d) Supporting plots
best_svm7 <- svm(mpg01 ~ ., data = Auto2, kernel = "radial",
cost = tune_rad7$best.parameters$cost,
gamma = tune_rad7$best.parameters$gamma)
par(mfrow = c(2, 2))
plot(best_svm7, Auto2, horsepower ~ weight)
These pairwise slices of the fitted (best) radial-kernel SVM show
fairly smooth, close-to-linear-looking boundaries in each 2-D projection
— weight and displacement in particular
separate the two classes cleanly, which matches the CV results in (b)
and (c): a mostly-linear cut through these variables already does most
of the work, and the radial kernel only bends that boundary a little at
the margins.
Problem 8: Predicting Orange Juice Purchase in the OJ Data Set
(a) Train/test split
data(OJ)
set.seed(1)
train8 <- sample(nrow(OJ), 800)
OJ_tr <- OJ[train8, ]
OJ_te <- OJ[-train8, ]
cat("Training:", nrow(OJ_tr), " Test:", nrow(OJ_te), "\n")## Training: 800 Test: 270(b) Support vector classifier with cost = 0.01
##
## Call:
## svm(formula = Purchase ~ ., data = OJ_tr, kernel = "linear", cost = 0.01)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 0.01
##
## Number of Support Vectors: 435
##
## ( 219 216 )
##
##
## Number of Classes: 2
##
## Levels:
## CH MMWith cost = 0.01 the margin is wide and the classifier
tolerates a lot of misclassification during training in exchange for a
simpler boundary — that shows up as 435 support vectors out of 800
training observations, roughly evenly split between the two classes,
which is a large fraction of the data sitting on or inside the
margin.
(c) Training and test error at cost = 0.01
tr_pred8 <- predict(svm_lin8, OJ_tr)
te_pred8 <- predict(svm_lin8, OJ_te)
err_tr8 <- mean(tr_pred8 != OJ_tr$Purchase)
err_te8 <- mean(te_pred8 != OJ_te$Purchase)
cat("Train error:", round(err_tr8, 4), " Test error:", round(err_te8, 4), "\n")## Train error: 0.175 Test error: 0.1778(d) Tuning cost by cross-validation
set.seed(1)
tune_lin8 <- tune(svm, Purchase ~ ., data = OJ_tr, kernel = "linear",
ranges = list(cost = 10^seq(-2, 1, length.out = 10)))
summary(tune_lin8)##
## Parameter tuning of 'svm':
##
## - sampling method: 10-fold cross validation
##
## - best parameters:
## cost
## 0.4641589
##
## - best performance: 0.16875
##
## - Detailed performance results:
## cost error dispersion
## 1 0.01000000 0.17625 0.02853482
## 2 0.02154435 0.17625 0.02972676
## 3 0.04641589 0.17500 0.02568506
## 4 0.10000000 0.17250 0.03162278
## 5 0.21544347 0.17250 0.02751262
## 6 0.46415888 0.16875 0.02651650
## 7 1.00000000 0.17500 0.02946278
## 8 2.15443469 0.17125 0.03064696
## 9 4.64158883 0.17125 0.03175973
## 10 10.00000000 0.17375 0.03197764## Best cost: 0.4642(e) Training and test error at the tuned cost
svm_lin8_best <- svm(Purchase ~ ., data = OJ_tr, kernel = "linear", cost = best_cost8)
tr_pred8b <- predict(svm_lin8_best, OJ_tr)
te_pred8b <- predict(svm_lin8_best, OJ_te)
err_tr8b <- mean(tr_pred8b != OJ_tr$Purchase)
err_te8b <- mean(te_pred8b != OJ_te$Purchase)
cat("Train error:", round(err_tr8b, 4), " Test error:", round(err_te8b, 4), "\n")## Train error: 0.165 Test error: 0.1556Tuning cost improves both error rates a little: train error goes from
0.175 to 0.165 and test error from 0.1778 to 0.1556. The improvement is
modest because cost = 0.01 was already a reasonable choice
for a roughly linear problem like this one — there just wasn’t much room
for tuning to help.
(f) Radial-kernel SVM
svm_rad8 <- svm(Purchase ~ ., data = OJ_tr, kernel = "radial", cost = 0.01)
tr_predr8 <- predict(svm_rad8, OJ_tr)
te_predr8 <- predict(svm_rad8, OJ_te)
cat("Cost = 0.01 -- Train error:", round(mean(tr_predr8 != OJ_tr$Purchase), 4),
" Test error:", round(mean(te_predr8 != OJ_te$Purchase), 4), "\n")## Cost = 0.01 -- Train error: 0.3938 Test error: 0.3778set.seed(1)
tune_rad8 <- tune(svm, Purchase ~ ., data = OJ_tr, kernel = "radial",
ranges = list(cost = 10^seq(-2, 1, length.out = 10)))
best_cost_rad8 <- tune_rad8$best.parameters$cost
cat("Best cost:", round(best_cost_rad8, 4), "\n")## Best cost: 0.4642svm_rad8_best <- svm(Purchase ~ ., data = OJ_tr, kernel = "radial", cost = best_cost_rad8)
tr_predrb <- predict(svm_rad8_best, OJ_tr)
te_predrb <- predict(svm_rad8_best, OJ_te)
err_tr_rad <- mean(tr_predrb != OJ_tr$Purchase)
err_te_rad <- mean(te_predrb != OJ_te$Purchase)
cat("Tuned -- Train error:", round(err_tr_rad, 4), " Test error:", round(err_te_rad, 4), "\n")## Tuned -- Train error: 0.1475 Test error: 0.1778At cost = 0.01 the default-gamma radial kernel is far
too constrained — nearly every point falls inside the margin and the
classifier does little better than guessing the majority class, giving a
train/test error around 0.39. Tuning fixes most of that: the best cost
(0.464) brings the train error down to 0.1475 and the test error to
0.1778, a large improvement over the untuned fit, though the test error
still ends up close to the tuned linear model’s from part (e).
(g) Polynomial-kernel SVM (degree 2)
svm_poly8 <- svm(Purchase ~ ., data = OJ_tr, kernel = "polynomial", degree = 2, cost = 0.01)
tr_predp8 <- predict(svm_poly8, OJ_tr)
te_predp8 <- predict(svm_poly8, OJ_te)
cat("Cost = 0.01 -- Train error:", round(mean(tr_predp8 != OJ_tr$Purchase), 4),
" Test error:", round(mean(te_predp8 != OJ_te$Purchase), 4), "\n")## Cost = 0.01 -- Train error: 0.3725 Test error: 0.3667set.seed(1)
tune_poly8 <- tune(svm, Purchase ~ ., data = OJ_tr, kernel = "polynomial", degree = 2,
ranges = list(cost = 10^seq(-2, 1, length.out = 10)))
best_cost_poly8 <- tune_poly8$best.parameters$cost
cat("Best cost:", round(best_cost_poly8, 4), "\n")## Best cost: 2.1544svm_poly8_best <- svm(Purchase ~ ., data = OJ_tr, kernel = "polynomial", degree = 2, cost = best_cost_poly8)
tr_predpb <- predict(svm_poly8_best, OJ_tr)
te_predpb <- predict(svm_poly8_best, OJ_te)
err_tr_poly <- mean(tr_predpb != OJ_tr$Purchase)
err_te_poly <- mean(te_predpb != OJ_te$Purchase)
cat("Tuned -- Train error:", round(err_tr_poly, 4), " Test error:", round(err_te_poly, 4), "\n")## Tuned -- Train error: 0.1588 Test error: 0.2111The degree-2 polynomial kernel shows the same pattern as the radial
kernel: at cost = 0.01 it’s badly under-fit (train/test
error around 0.37), and tuning cost (best value 2.154) brings it down
substantially, to a train error of 0.1588 and test error of 0.2111.
(h) Overall comparison
results8 <- data.frame(
Kernel = c("Linear", "Radial", "Polynomial (deg 2)"),
Best_Cost = round(c(best_cost8, best_cost_rad8, best_cost_poly8), 3),
Train_Error = round(c(err_tr8b, err_tr_rad, err_tr_poly), 4),
Test_Error = round(c(err_te8b, err_te_rad, err_te_poly), 4)
)
print(results8)## Kernel Best_Cost Train_Error Test_Error
## 1 Linear 0.464 0.1650 0.1556
## 2 Radial 0.464 0.1475 0.1778
## 3 Polynomial (deg 2) 2.154 0.1588 0.2111Once every kernel is properly tuned, the differences between them are
small, but the tuned linear kernel gives the lowest
test error (0.1556), just ahead of the tuned radial and polynomial
kernels (0.1778 and 0.2111). That’s consistent with problem 7’s finding
on a different data set: once cost is chosen sensibly, letting the
kernel bend the boundary doesn’t necessarily buy anything if the true
boundary between CH and MM purchases is
already close to linear in the OJ predictors (things like
LoyalCH, price, and discount variables). On this data set
I’d go with the linear-kernel SVM — it’s the simplest
model, it’s the cheapest to tune (one parameter instead of two), and it
comes out at least as accurate as the more flexible kernels on the
held-out test set.