Both exercises are from Chapter 7 of Kuhn and Johnson, Applied Predictive Modeling. Exercise 7.2 uses simulated data with a known nonlinear formula, so the true answer is known. Exercise 7.5 returns to the chemical manufacturing data. The models are the four from the chapter: K-nearest neighbors (KNN), a single-hidden-layer neural network (averaged over several random starts), multivariate adaptive regression splines (MARS), and a support vector machine (SVM) with a radial basis kernel.
7.2. Friedman (1991) introduced several benchmark data sets create by simulation. One of these simulations used the following nonlinear equation to create data: \(y = 10 \sin(\pi x_1 x_2) + 20 (x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + N(0, \sigma^2)\), where the \(x\) values are random variables uniformly distributed between \([0, 1]\) (there are also 5 other non-informative variables also created in the simulation). The package
mlbenchcontains a function calledmlbench.friedman1that simulates these data:
library(mlbench)
set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
## We convert the 'x' data from a matrix to a data frame
## One reason is that this will give the columns names.
trainingData$x <- data.frame(trainingData$x)
## Look at the data using
featurePlot(trainingData$x, trainingData$y)
## or other methods.
## This creates a list with a vector 'y' and a matrix
## of predictors 'x'. Also simulate a large test set to
## estimate the true error rate with good precision:
testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- data.frame(testData$x)
Tune several models on these data. For example:
library(caret) knnModel <- train(x = trainingData$x, y = trainingData$y, method = "knn", preProc = c("center", "scale"), tuneLength = 10) knnModel knnPred <- predict(knnModel, newdata = testData$x) ## The function 'postResample' can be used to get the test set ## perforamnce values postResample(pred = knnPred, obs = testData$y)
The book’s printed result for this call is a test set RMSE of 3.2286834 and R-squared of 0.6871735.
Which models appear to give the best performance? Does MARS select the informative predictors (those named X1-X5)?
The training set has 200 samples and 10 predictors; only
X1 to X5 enter the formula, and
X6 to X10 are noise. The test set of 5,000
samples gives a precise estimate of the true error. The error that
cannot be removed is the simulated noise (standard deviation 1), so the
best possible test RMSE is about 1.
All four models are tuned on the same resamples (the seed is set
before each train() call) with 10-fold cross-validation
repeated 5 times. KNN, the neural network and the SVM are sensitive to
predictor scale, so they are centered and scaled. MARS needs no
pre-processing (Section 7.2).
f1_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)
f1_models <- fit_once("f1_models", {
## KNN, as in the exercise but with repeated cross-validation instead of the bootstrap
set.seed(200)
knnModel <- train(x = trainingData$x, y = trainingData$y,
method = "knn",
preProc = c("center", "scale"),
tuneLength = 10,
trControl = f1_ctrl)
## single-hidden-layer neural network, averaged over 5 random starts (avNNet)
## the predictors are independent uniform variables, so no correlation filtering is needed
nnetGrid <- expand.grid(.decay = c(0, 0.01, 0.1),
.size = 1:10,
.bag = FALSE)
set.seed(200)
nnetModel <- train(x = trainingData$x, y = trainingData$y,
method = "avNNet",
preProc = c("center", "scale"),
tuneGrid = nnetGrid,
linout = TRUE, trace = FALSE,
MaxNWts = 10 * (ncol(trainingData$x) + 1) + 10 + 1,
maxit = 2000,
trControl = f1_ctrl)
## MARS: degree of the features and number of retained terms
marsGrid <- expand.grid(.degree = 1:2, .nprune = 2:20)
set.seed(200)
marsModel <- train(x = trainingData$x, y = trainingData$y,
method = "earth",
tuneGrid = marsGrid,
trControl = f1_ctrl)
## SVM, radial basis kernel (sigma estimated analytically, cost tuned)
set.seed(200)
svmModel <- train(x = trainingData$x, y = trainingData$y,
method = "svmRadial",
preProc = c("center", "scale"),
tuneLength = 14,
trControl = f1_ctrl)
list(KNN = knnModel, NNet = nnetModel, MARS = marsModel, SVM = svmModel)
})
knnModel <- f1_models$KNN; nnetModel <- f1_models$NNet
marsModel <- f1_models$MARS; svmModel <- f1_models$SVM
lapply(f1_models, function(m) m$bestTune)
## $KNN
## k
## 5 13
##
## $NNet
## size decay bag
## 4 4 0 FALSE
##
## $MARS
## nprune degree
## 32 14 2
##
## $SVM
## sigma C
## 6 0.06299324 8
# convergence of the five networks averaged in the final model: 0 = converged,
# 1 = stopped at maxit (the iteration limit) before converging
sapply(nnetModel$finalModel$model, function(m) m$convergence)
## [1] 0 0 0 0 0
Why these grids, and whether the optimum sits on an edge.
tuneLength = 10: caret tries ten
odd values of k starting at 5 (5, 7, …, 23), the book’s own call. The
best k is 13, inside the grid.tuneLength = 14: cost from 0.25
to 2048 (powers of two), the chapter’s example. The best cost is 8;
larger costs give nearly identical resampled RMSE, and
train() selected 8 because it had the lowest value.
sigma is estimated from the data rather than tuned (next
section shows what it does).par(mfrow = c(2, 2))
plot(knnModel$results$k, knnModel$results$RMSE, type = "b", pch = 19,
xlab = "Neighbors (k)", ylab = "RMSE (repeated CV)", main = "KNN")
plot(svmModel$results$C, svmModel$results$RMSE, type = "b", pch = 19, log = "x",
xlab = "Cost (log scale)", ylab = "RMSE (repeated CV)", main = "SVM, radial kernel")
mars_res <- marsModel$results
plot(mars_res$nprune[mars_res$degree == 1], mars_res$RMSE[mars_res$degree == 1],
type = "b", pch = 19,
ylim = range(mars_res$RMSE) + c(0, 0.3 * diff(range(mars_res$RMSE))),
xlab = "Terms retained (nprune)", ylab = "RMSE (repeated CV)", main = "MARS")
lines(mars_res$nprune[mars_res$degree == 2], mars_res$RMSE[mars_res$degree == 2],
type = "b", pch = 17, col = "firebrick")
legend("top", c("degree 1", "degree 2"), pch = c(19, 17),
col = c("black", "firebrick"), bty = "n", horiz = TRUE)
nn_res <- nnetModel$results
plot(nn_res$size[nn_res$decay == 0], nn_res$RMSE[nn_res$decay == 0], type = "b", pch = 19,
ylim = range(nn_res$RMSE) + c(0, 0.3 * diff(range(nn_res$RMSE))),
xlab = "Hidden units", ylab = "RMSE (repeated CV)", main = "Neural network")
lines(nn_res$size[nn_res$decay == 0.01], nn_res$RMSE[nn_res$decay == 0.01],
type = "b", pch = 17, col = "steelblue")
lines(nn_res$size[nn_res$decay == 0.1], nn_res$RMSE[nn_res$decay == 0.1],
type = "b", pch = 15, col = "firebrick")
legend("top", c("decay 0", "decay 0.01", "decay 0.1"), pch = c(19, 17, 15),
col = c("black", "steelblue", "firebrick"), bty = "n", horiz = TRUE)
par(mfrow = c(1, 1))
Each profile has a clear minimum or plateau: KNN is lowest at k = 13, the SVM is flat from a cost of about 8, MARS is lowest at 14 terms with degree 2, and the neural network at 4 hidden units with decay 0.
Resampled and test set performance of each model at its best tuning values:
f1_resampled <- t(sapply(f1_models, function(m) {
best <- m$results[which.min(m$results$RMSE), ]
c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))
f1_test <- t(sapply(f1_models, function(m)
postResample(pred = predict(m, newdata = testData$x), obs = testData$y)))
f1_perf <- cbind(f1_resampled, f1_test)
colnames(f1_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(f1_perf, 3)
## CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## KNN 3.065 0.697 2.468 3.148 0.675 2.524
## NNet 1.950 0.844 1.544 2.459 0.775 1.779
## MARS 1.241 0.937 0.992 1.172 0.945 0.932
## SVM 1.868 0.857 1.477 2.054 0.829 1.559
f1_resamp <- resamples(f1_models)
bwplot(f1_resamp, metric = "RMSE")
summary(diff(f1_resamp))
##
## Call:
## summary.diff.resamples(object = diff(f1_resamp))
##
## p-value adjustment: bonferroni
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
##
## MAE
## KNN NNet MARS SVM
## KNN 0.92357 1.47580 0.99066
## NNet < 2.2e-16 0.55223 0.06709
## MARS < 2.2e-16 < 2.2e-16 -0.48514
## SVM < 2.2e-16 0.7452 3.537e-16
##
## RMSE
## KNN NNet MARS SVM
## KNN 1.11505 1.82398 1.19730
## NNet < 2.2e-16 0.70893 0.08225
## MARS < 2.2e-16 4.727e-16 -0.62668
## SVM < 2.2e-16 0.6167 < 2.2e-16
##
## Rsquared
## KNN NNet MARS SVM
## KNN -0.14743 -0.24035 -0.16032
## NNet 7.124e-16 -0.09292 -0.01289
## MARS < 2.2e-16 1.262e-11 0.08003
## SVM < 2.2e-16 0.668 3.369e-13
# the KNN model from the exercise's call, tuned with repeated cross-validation
knnModel
## k-Nearest Neighbors
##
## 200 samples
## 10 predictor
##
## Pre-processing: centered (10), scaled (10)
## Resampling: Cross-Validated (10 fold, repeated 5 times)
## Summary of sample sizes: 180, 180, 180, 180, 180, 180, ...
## Resampling results across tuning parameters:
##
## k RMSE Rsquared MAE
## 5 3.199670 0.5910371 2.649336
## 7 3.108444 0.6299684 2.537062
## 9 3.090673 0.6553105 2.506983
## 11 3.070640 0.6798579 2.489940
## 13 3.064958 0.6966689 2.467551
## 15 3.082000 0.7111449 2.491659
## 17 3.102544 0.7142288 2.506028
## 19 3.115994 0.7231515 2.517837
## 21 3.148223 0.7237919 2.560250
## 23 3.177131 0.7236183 2.589908
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was k = 13.
knnPred <- predict(knnModel, newdata = testData$x)
postResample(pred = knnPred, obs = testData$y)
## RMSE Rsquared MAE
## 3.1481557 0.6747755 2.5236041
Which models perform best. MARS is clearly the best:
its resampled RMSE is 1.241 and its test RMSE is 1.172 (R-squared
0.945), close to the noise floor of about 1. The SVM is second (test
RMSE 2.054), the neural network third (2.459) and KNN last (3.148). The
diff() table says MARS differs from each of the other three
with adjusted p-values below 0.0001. The SVM and the neural network
cannot be told apart on the resamples (adjusted p-value about 0.62 for
RMSE), although the SVM is ahead on the large test set. The resampled
and test RMSE are close for KNN and MARS, differ by about 0.2 for the
SVM, and differ most for the neural network (test 2.46 against resampled
1.95), so the 5,000-sample test set is the more trustworthy referee
here.
The ranking has a reason. The true function is additive in
X3, X4 and X5 and has one
interaction, sin(pi * X1 * X2), which is the structure a
degree-2 MARS model with hinge terms can build, and MARS also drops
predictors it does not need. KNN, the SVM and the neural network use all
ten predictors, so the five noise predictors may dilute the distances
(Section 7.4 notes that noisy predictors push similar samples apart) and
add parameters to fit; that explanation is consistent with the results,
though this comparison did not isolate the effect of the noise
predictors. KNN’s test RMSE of 3.15 is close to the 3.23 the book
reports for its example call (which used the bootstrap). All of this
comes from one simulated training set of 200 samples; a different seed
would move the numbers.
train() reports the best SVM settings, but it helps to
see what each parameter does. The chunk below fits the SVM directly with
ksvm() on the scaled training data and changes one setting
at a time, reporting the 10-fold cross-validated RMSE and the number of
support vectors. This is a supplementary sensitivity analysis: it uses
ksvm()’s internal 10-fold cross-validation after scaling on
the full training set, so its errors are not directly comparable with
the main repeated cross-validation results, where preprocessing is
fitted within each resample. It only illustrates how the parameters
affect the fit. The external test set is not used here.
x_scaled <- scale(trainingData$x)
try_svm <- function(cost, sigma, epsilon) {
set.seed(1)
fit <- ksvm(x_scaled, trainingData$y, type = "eps-svr", kernel = "rbfdot",
kpar = list(sigma = sigma), C = cost, epsilon = epsilon, cross = 10)
c(cost = cost, sigma = sigma, epsilon = epsilon,
CV_RMSE = sqrt(cross(fit)), support_vectors = nSV(fit))
}
# cost and sigma, with epsilon fixed at 0.1
svm_cs <- expand.grid(cost = c(0.25, 1, 8, 64), sigma = c(0.005, 0.063, 0.5))
svm_cs <- t(apply(svm_cs, 1, function(p) try_svm(p[["cost"]], p[["sigma"]], 0.1)))
round(svm_cs, 3)
## cost sigma epsilon CV_RMSE support_vectors
## [1,] 0.25 0.005 0.1 3.797 175
## [2,] 1.00 0.005 0.1 2.674 174
## [3,] 8.00 0.005 0.1 2.345 171
## [4,] 64.00 0.005 0.1 1.969 161
## [5,] 0.25 0.063 0.1 2.526 167
## [6,] 1.00 0.063 0.1 2.072 160
## [7,] 8.00 0.063 0.1 1.786 152
## [8,] 64.00 0.063 0.1 1.794 150
## [9,] 0.25 0.500 0.1 4.702 187
## [10,] 1.00 0.500 0.1 4.254 183
## [11,] 8.00 0.500 0.1 4.143 184
## [12,] 64.00 0.500 0.1 4.143 184
# epsilon, with cost 8 and sigma 0.063
svm_eps <- t(sapply(c(0.01, 0.1, 0.5, 1, 2), function(e) try_svm(8, 0.063, e)))
round(svm_eps, 3)
## cost sigma epsilon CV_RMSE support_vectors
## [1,] 8 0.063 0.01 1.810 198
## [2,] 8 0.063 0.10 1.786 152
## [3,] 8 0.063 0.50 2.279 52
## [4,] 8 0.063 1.00 3.077 19
## [5,] 8 0.063 2.00 4.656 4
sigma (0.063) the
cross-validated RMSE falls from 2.53 at a cost of 0.25 to 1.79 at 8 and
does not improve at 64: the validation error decreased as cost increased
to 8, with no improvement at 64 in this experiment, which matches the
plateau in the cost profile above.train() (0.063) sits in the good middle region.
These results show poorer validation performance at the extreme
settings; they do not by themselves show why.ksvm() standardizes the response by
default, so these epsilon values are in standardized units: with a
response standard deviation of 4.95, epsilon 0.1 is about 0.5 in
original units and epsilon 1 is about 4.95, while the RMSE is reported
on the original scale. A tiny epsilon makes almost every point a support
vector (198 of 200 at 0.01); a wide one leaves only a few (19 at 1) and
the fit is much worse, with RMSE 3.1 (4.7 at 2, close to the standard
deviation of y, about 5). The default of 0.1 gives the
lowest error of those tried, and train() keeps epsilon
fixed at that default and tunes only the cost.The values train() chose (cost 8, sigma 0.063) land
where the by-hand exploration shows the lowest error. The
set.seed(1) inside try_svm() only fixes the
cross-validation folds so the table is reproducible.
marsModel$finalModel
## Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 9 4
## GCV 1.62945 RSS 225.8601 GRSq 0.9338437 RSq 0.953688
summary(marsModel$finalModel)
## Call: earth(x=data.frame[200,10], y=c(18.46,16.1,17...), keepxy=TRUE, degree=2,
## nprune=14)
##
## coefficients
## (Intercept) 21.905319
## h(0.621722-X1) -15.726181
## h(X1-0.621722) 9.234027
## h(0.601063-X2) -18.253527
## h(X2-0.601063) 10.448545
## h(0.447442-X3) 9.700589
## h(X3-0.606015) 12.674694
## h(0.734892-X4) -9.863526
## h(X4-0.734892) 10.297964
## h(0.850094-X5) -5.324175
## h(0.218266-X1) * h(X2-0.601063) -55.358726
## h(X1-0.218266) * h(X2-0.601063) -29.100250
## h(X1-0.621722) * h(X2-0.295997) -26.833129
## h(0.649253-X1) * h(0.601063-X2) 27.120721
##
## Selected 14 of 18 terms, and 5 of 10 predictors (nprune=14)
## Termination condition: Reached nk 21
## Importance: X1, X4, X2, X5, X3, X6-unused, X7-unused, X8-unused, X9-unused, ...
## Number of terms at each degree of interaction: 1 9 4
## GCV 1.62945 RSS 225.8601 GRSq 0.9338437 RSq 0.953688
varImp(marsModel, scale = FALSE) # scale = FALSE: show earth's own values, not caret's 0-100 rescaling
## earth variable importance
##
## Overall
## X1 100.00
## X4 85.13
## X2 69.22
## X5 49.28
## X3 39.95
plotmo(marsModel$finalModel, caption = "MARS fitted response profiles: Friedman data")
## plotmo grid: X1 X2 X3 X4 X5 X6 X7
## 0.5139349 0.5106664 0.537307 0.4445841 0.5343299 0.4975981 0.4688035
## X8 X9 X10
## 0.497961 0.5288716 0.5359218
# predictors that appear in at least one retained (not pruned) MARS term
fm <- marsModel$finalModel
retained <- fm$dirs[fm$selected.terms, , drop = FALSE]
used <- colnames(retained)[colSums(retained != 0) > 0]
used
## [1] "X1" "X2" "X3" "X4" "X5"
Does MARS select the informative predictors? Yes.
The final model keeps 5 of the 10 predictors, X1, X2, X3, X4, X5, and
none of X6 to X10 appears in any term. The
equation has two hinge terms for each of X1 to
X4, one hinge for X5, and four degree-2 terms
that are products of X1 and X2 hinges, which
matches the X1 * X2 interaction in the formula. The two
X3 hinges have different knots (0.447 and 0.606, on either
side of 0.5 where the true quadratic term 20 * (X3 - 0.5)^2
has its minimum), so together they give a falling segment, a flat middle
segment and a rising segment, a rough piecewise-linear version of that
curve. X3 is retained in the final equation, yet the
default importance table gives it 0. That zero comes from caret’s extra
rescaling, which maps the lowest reported importance to 0; with
scale = FALSE (table above) X3 has an
importance of about 40, the lowest of the five. Which predictors were
selected is therefore determined from the retained terms in the
equation, not from whether the scaled importance is positive.
7.5. Exercise 6.3 describes data for a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several nonlinear regression models.
The data, the 80/20 split (seed 624), the removal of near-zero-variance predictors, and the KNN imputation fitted on the training rows only are the same as in Exercise 6.3.
data(ChemicalManufacturingProcess)
process_predictors <- ChemicalManufacturingProcess[, -1]
yield <- ChemicalManufacturingProcess$Yield
set.seed(624)
cmp_rows <- createDataPartition(yield, p = 0.80, list = FALSE)
cmp_train_x <- process_predictors[cmp_rows, ]
cmp_test_x <- process_predictors[-cmp_rows, ]
cmp_train_y <- yield[cmp_rows]
cmp_test_y <- yield[-cmp_rows]
cmp_nzv <- nearZeroVar(cmp_train_x, names = TRUE)
cmp_nzv
## [1] "BiologicalMaterial07"
cmp_train_x <- cmp_train_x[, !(names(cmp_train_x) %in% cmp_nzv)]
cmp_test_x <- cmp_test_x[, !(names(cmp_test_x) %in% cmp_nzv)]
c(train = nrow(cmp_train_x), test = nrow(cmp_test_x), predictors = ncol(cmp_train_x),
missing_cells_train = sum(is.na(cmp_train_x)))
## train test predictors missing_cells_train
## 144 32 56 99
As in Exercise 6.3, the imputation is fitted inside each resample
with preProc = "knnImpute" (which also centers and scales),
so a held-out fold never influences the values its own predictors
receive, and predict() applies the training imputation to
the test rows.
The neural network is the model most hurt by correlated predictors
(Section 7.1), so for it the book removes predictors until no pair has a
correlation above 0.75. Here that filter ("corr") is
applied inside each training resample through
preProcOptions. caret runs it first, on pairwise complete
observations with an absolute-correlation cutoff of 0.75; the remaining
predictors are then centered, scaled and imputed using the training data
of that resample (writing c("knnImpute", "corr") does not
set the order).
Which nonlinear regression model gives the optimal resampling and test set performance?
cmp_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)
cmp_models <- fit_once("cmp_models", {
set.seed(100)
cmp_knn <- train(cmp_train_x, cmp_train_y, method = "knn",
preProc = "knnImpute",
tuneGrid = data.frame(k = 1:20),
trControl = cmp_ctrl)
set.seed(100)
cmp_nnet <- train(cmp_train_x, cmp_train_y, method = "avNNet",
# "corr" is not in the book's code (the book calls findCorrelation() directly);
# it does the same filtering inside each resample, with the cutoff set below
preProc = c("knnImpute", "corr"),
tuneGrid = expand.grid(decay = c(0, 0.01, 0.1), size = 1:7, bag = FALSE),
linout = TRUE, trace = FALSE, maxit = 500,
MaxNWts = 7 * (ncol(cmp_train_x) + 1) + 7 + 1,
trControl = trainControl(method = "repeatedcv", number = 10, repeats = 5,
preProcOptions = list(thresh = 0.95, ICAcomp = 3,
k = 5, freqCut = 19,
uniqueCut = 10,
cutoff = 0.75)))
set.seed(100)
cmp_mars <- train(cmp_train_x, cmp_train_y, method = "earth",
preProc = "knnImpute",
tuneGrid = expand.grid(degree = 1:2, nprune = 2:30),
trControl = cmp_ctrl)
set.seed(100)
cmp_svm <- train(cmp_train_x, cmp_train_y, method = "svmRadial",
preProc = "knnImpute",
tuneLength = 14,
trControl = cmp_ctrl)
list(KNN = cmp_knn, NNet = cmp_nnet, MARS = cmp_mars, SVM = cmp_svm)
})
cmp_knn <- cmp_models$KNN; cmp_nnet <- cmp_models$NNet
cmp_mars <- cmp_models$MARS; cmp_svm <- cmp_models$SVM
lapply(cmp_models, function(m) m$bestTune)
## $KNN
## k
## 5 5
##
## $NNet
## size decay bag
## 8 1 0.01 FALSE
##
## $MARS
## nprune degree
## 32 4 2
##
## $SVM
## sigma C
## 6 0.01396075 8
Why these grids, and whether the optimum sits on an edge.
MaxNWts limit is set to fit the largest network; the
correlation filter leaves 35 of the 56 predictors for this model). The
best network has 1 hidden unit, which is the lower edge
of the size grid. One unit is the smallest possible network, so the grid
cannot go lower, and the profile shows every larger size doing worse;
the decay of 0.01 is inside its grid.tuneLength = 14: the same cost
grid as in 7.2 (0.25 to 2048). The best cost is 8; larger costs give
nearly identical resampled RMSE, and train() selected 8
because it had the lowest value.cmp_resampled <- t(sapply(cmp_models, function(m) {
best <- m$results[which.min(m$results$RMSE), ]
c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))
cmp_test_perf <- t(sapply(cmp_models, function(m)
postResample(pred = predict(m, cmp_test_x), obs = cmp_test_y)))
cmp_perf <- cbind(cmp_resampled, cmp_test_perf)
colnames(cmp_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(cmp_perf, 3)
## CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## KNN 1.294 0.500 1.063 1.402 0.613 1.082
## NNet 1.413 0.423 1.150 1.496 0.499 1.183
## MARS 1.193 0.570 0.971 1.201 0.713 0.918
## SVM 1.130 0.603 0.930 1.147 0.748 0.796
cmp_resamp <- resamples(cmp_models)
bwplot(cmp_resamp, metric = "RMSE")
summary(diff(cmp_resamp))
##
## Call:
## summary.diff.resamples(object = diff(cmp_resamp))
##
## p-value adjustment: bonferroni
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
##
## MAE
## KNN NNet MARS SVM
## KNN -0.08781 0.09128 0.13274
## NNet 0.16378 0.17909 0.22055
## MARS 0.01382 4.618e-06 0.04145
## SVM 4.558e-05 9.029e-10 0.75487
##
## RMSE
## KNN NNet MARS SVM
## KNN -0.11900 0.10090 0.16348
## NNet 0.06635 0.21990 0.28248
## MARS 0.03578 3.251e-06 0.06258
## SVM 1.120e-05 3.574e-10 0.37273
##
## Rsquared
## KNN NNet MARS SVM
## KNN 0.07695 -0.07014 -0.10274
## NNet 0.1584712 -0.14709 -0.17969
## MARS 0.0470613 9.632e-07 -0.03259
## SVM 0.0005413 8.860e-09 0.9500346
par(mfrow = c(2, 2))
plot(cmp_knn$results$k, cmp_knn$results$RMSE, type = "b", pch = 19,
xlab = "Neighbors (k)", ylab = "RMSE (repeated CV)", main = "KNN")
plot(cmp_svm$results$C, cmp_svm$results$RMSE, type = "b", pch = 19, log = "x",
xlab = "Cost (log scale)", ylab = "RMSE (repeated CV)", main = "SVM, radial kernel")
cm <- cmp_mars$results
plot(cm$nprune[cm$degree == 1], cm$RMSE[cm$degree == 1], type = "b", pch = 19,
ylim = range(cm$RMSE) + c(0, 0.3 * diff(range(cm$RMSE))),
xlab = "Terms retained (nprune)", ylab = "RMSE (repeated CV)", main = "MARS")
lines(cm$nprune[cm$degree == 2], cm$RMSE[cm$degree == 2], type = "b", pch = 17, col = "firebrick")
legend("top", c("degree 1", "degree 2"), pch = c(19, 17), col = c("black", "firebrick"), bty = "n",
horiz = TRUE)
cn <- cmp_nnet$results
plot(cn$size[cn$decay == 0], cn$RMSE[cn$decay == 0], type = "b", pch = 19,
ylim = range(cn$RMSE) + c(0, 0.3 * diff(range(cn$RMSE))),
xlab = "Hidden units", ylab = "RMSE (repeated CV)", main = "Neural network")
lines(cn$size[cn$decay == 0.01], cn$RMSE[cn$decay == 0.01], type = "b", pch = 17, col = "steelblue")
lines(cn$size[cn$decay == 0.1], cn$RMSE[cn$decay == 0.1], type = "b", pch = 15, col = "firebrick")
legend("top", c("decay 0", "decay 0.01", "decay 0.1"), pch = c(19, 17, 15),
col = c("black", "steelblue", "firebrick"), bty = "n", horiz = TRUE)
par(mfrow = c(1, 1))
The SVM with a radial kernel gives the best resampling and test set
performance: resampled RMSE 1.13 (R-squared 0.603) and test RMSE 1.147
(R-squared 0.748). The ranking by RMSE is the same on resampling and on
the test set: SVM, then MARS (1.193 and 1.201), KNN (1.294 and 1.402)
and the neural network (1.413 and 1.496). The SVM’s resampled and test
RMSE agree closely. The diff() table shows the SVM is
clearly better than KNN and the neural network (adjusted p-values near
0.00001 and below), but the SVM and MARS cannot be separated (adjusted
p-value about 0.37 for RMSE; about 0.19 in the three-model comparison
further down, because the adjustment depends on how many models are
compared), so the paired comparison did not detect a difference between
the SVM and MARS, and the SVM’s lead of about 0.06 RMSE should not be
treated as established. The mean cross-validated R-squared is
approximately 0.60, while the test-set R-squared is approximately 0.75,
so the best model’s predictions are only moderately correlated with
yield. A test RMSE near 1.15 yield points is the lowest among the tested
candidates, but whether that is accurate enough for plant decisions
depends on an error tolerance the exercise does not give.
The tuning profiles show how each model was chosen. The SVM’s RMSE
falls as the cost rises and is essentially flat from a cost of about 8
upward. train() picked the cost with the lowest resampled
RMSE (8, with sigma estimated analytically as 0.014); the
differences among the larger costs are tiny, so the plateau supports the
choice but does not single out one value. KNN is best at 5 neighbors,
with RMSE worse for a single neighbor and rising slowly beyond. MARS is
best as a very small additive model: 4 terms involving only two
predictors (shown in the MARS output in part b). Its degree-2 profile
rises steeply with more terms and ends near RMSE 2.1, the instability
with higher-degree models that the chapter warns about. The neural
network is best at 1 hidden unit with decay 0.01, the edge of the grid,
and no larger network does better: RMSE is higher than its one-unit
value for every larger network at every decay value, with the decay 0.01
profile peaking at four units and partly recovering at six and seven but
staying above its one-unit value. Larger networks generally did worse on
resampling; over-fitting with only 144 training runs and several dozen
predictors is a plausible reason, though the profile alone does not
separate it from other limits of the fit. A one-unit network cannot be
shrunk further, so the grid did not need to extend below it.
The test set holds only 32 runs, so before reading too much into the differences, the largest absolute test errors of each model show whether a few runs drive the results (row numbers are the run’s position in the original data):
lapply(cmp_models, function(m) {
err <- predict(m, cmp_test_x) - cmp_test_y
names(err) <- rownames(cmp_test_x)
round(head(sort(abs(err), decreasing = TRUE), 3), 2)
})
## $KNN
## 49 44 130
## 4.13 3.61 2.16
##
## $NNet
## 49 28 44
## 4.42 2.46 2.35
##
## $MARS
## 49 44 121
## 3.52 2.93 1.92
##
## $SVM
## 49 44 77
## 4.42 2.51 1.50
Run 49 is the worst-predicted test run for all four models (errors of about 3.5 to 4.4 yield points) and run 44 is second for three of them, so these two runs account for a large share of every model’s test RMSE. Differences of a few hundredths between models on this test set mean little; the resampled comparison, which uses all 144 training runs, is the firmer evidence.
Which predictors are most important in the optimal nonlinear regression model? Do either the biological or process variables dominate the list? How do the top ten important predictors compare to the top ten predictors from the optimal linear model?
cmp_best_name <- names(which.min(cmp_resampled[, "RMSE"]))
cmp_best <- cmp_models[[cmp_best_name]]
cmp_best_name
## [1] "SVM"
cmp_imp <- varImp(cmp_best)
cmp_imp
## loess r-squared variable importance
##
## only 20 most important variables shown (out of 56)
##
## Overall
## ManufacturingProcess32 100.00
## BiologicalMaterial06 89.32
## ManufacturingProcess36 79.78
## BiologicalMaterial03 77.48
## ManufacturingProcess09 73.90
## ManufacturingProcess13 73.23
## ManufacturingProcess31 70.84
## BiologicalMaterial02 66.91
## ManufacturingProcess11 65.19
## BiologicalMaterial12 64.94
## ManufacturingProcess06 61.04
## ManufacturingProcess17 53.06
## ManufacturingProcess29 50.91
## BiologicalMaterial11 49.10
## BiologicalMaterial04 48.27
## ManufacturingProcess33 47.62
## ManufacturingProcess30 47.38
## BiologicalMaterial01 40.69
## BiologicalMaterial08 38.18
## ManufacturingProcess25 34.07
plot(cmp_imp, top = 10,
main = paste("Top ten predictors,", cmp_best_name, "(marginal loess R-squared)"))
imp_order <- order(cmp_imp$importance$Overall, decreasing = TRUE)
nl_top10 <- rownames(cmp_imp$importance)[imp_order][1:10]
nl_top10
## [1] "ManufacturingProcess32" "BiologicalMaterial06" "ManufacturingProcess36"
## [4] "BiologicalMaterial03" "ManufacturingProcess09" "ManufacturingProcess13"
## [7] "ManufacturingProcess31" "BiologicalMaterial02" "ManufacturingProcess11"
## [10] "BiologicalMaterial12"
table(ifelse(grepl("^Biological", nl_top10), "Biological", "Process"))
##
## Biological Process
## 4 6
varImp() is model-specific for MARS (caret rescales it
to 0-100) but, for the SVM, caret has no model-based measure and falls
back to a filter: the R-squared of a loess fit of each predictor against
the response, computed one predictor at a time (the “loess r-squared
variable importance” in the heading above). That ranking therefore shows
which predictors relate to yield on their own, not which predictors the
fitted SVM leans on. As a model-based cross-check, the MARS model
(second best on the test set) has its own importance, based on the
reduction in GCV error each predictor’s terms give (shown with
scale = FALSE, earth’s own relative values):
varImp(cmp_mars, scale = FALSE)
## earth variable importance
##
## Overall
## ManufacturingProcess32 100.00
## ManufacturingProcess09 58.09
summary(cmp_mars$finalModel)
## Call: earth(x=data.frame[144,56], y=c(38,42.44,42.0...), keepxy=TRUE, degree=2,
## nprune=4)
##
## coefficients
## (Intercept) 37.822440
## h(-1.16089-ManufacturingProcess09) -1.043508
## h(ManufacturingProcess09- -1.16089) 0.742130
## h(ManufacturingProcess32- -1.24119) 1.160811
##
## Selected 4 of 50 terms, and 2 of 56 predictors (nprune=4)
## Termination condition: RSq changed by less than 0.001 at 50 terms
## Importance: ManufacturingProcess32, ManufacturingProcess09, ...
## Number of terms at each degree of interaction: 1 3 (additive model)
## GCV 1.423486 RSS 181.497 GRSq 0.5542264 RSq 0.5997596
The tuned MARS model is a four-term additive model that uses only
ManufacturingProcess32 and
ManufacturingProcess09, so no other predictor appears in
the model. With caret’s default rescaling
ManufacturingProcess09 would be shown as 0 even though it
has two terms in the equation, because that rescaling maps the lowest
reported importance to 0; its importance before rescaling is about 58,
against 100 for ManufacturingProcess32. The model equation
is the direct evidence of which predictors it uses. That is far fewer
predictors than the SVM works with, yet its test RMSE is within about
0.05 of the SVM’s.
The linear model used for comparison is the elastic net, the best of the linear models fitted to these data in Exercise 6.3, refit here with exactly the same resampling so the comparison is fair. Its importance is the absolute size of the coefficients at the selected fraction, on the standardized predictors (Section 6.5). Its grid is lambda 0.01, 0.1 and 1 (two orders of magnitude of ridge penalty) crossed with fraction 0.05 to 1 in 20 steps (this spans most of the elastic-net coefficient path for each penalty); the selected values are listed below and neither is on the edge of its grid.
cmp_enet <- fit_once("cmp_enet", {
set.seed(100)
train(cmp_train_x, cmp_train_y, method = "enet",
preProc = "knnImpute",
tuneGrid = expand.grid(lambda = c(0.01, 0.1, 1),
fraction = seq(0.05, 1, length = 20)),
trControl = cmp_ctrl)
})
cmp_enet$bestTune
## fraction lambda
## 26 0.3 0.1
c(resampled_RMSE = min(cmp_enet$results$RMSE),
test_RMSE = unname(postResample(predict(cmp_enet, cmp_test_x), cmp_test_y)["RMSE"]))
## resampled_RMSE test_RMSE
## 1.196128 1.219538
lin_coef <- predict(cmp_enet$finalModel, s = cmp_enet$bestTune$fraction,
mode = "fraction", type = "coefficients")$coefficients
lin_rank <- sort(abs(lin_coef[lin_coef != 0]), decreasing = TRUE)
lin_top10 <- names(lin_rank)[1:10]
c(nonzero_coefficients = length(lin_rank))
## nonzero_coefficients
## 13
The elastic net has a resampled RMSE of 1.196 and a test RMSE of 1.22, against 1.13 and 1.147 for the SVM. The SVM is lower by about 0.07 RMSE on both measures, and MARS (resampled 1.193) is no better than the linear model on resampling. The elastic net was tuned on the same resamples (same seed), so the three models can be compared on paired resampled RMSE like the models above:
cmp_vs_linear <- resamples(list(SVM = cmp_svm, MARS = cmp_mars, Enet = cmp_enet))
summary(diff(cmp_vs_linear), metric = "RMSE")
##
## Call:
## summary.diff.resamples(object = diff(cmp_vs_linear), metric = "RMSE")
##
## p-value adjustment: bonferroni
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
##
## MAE
## SVM MARS Enet
## SVM -0.04145 -0.03943
## MARS 0.3774 0.00202
## Enet 0.4599 1.0000
##
## RMSE
## SVM MARS Enet
## SVM -0.062585 -0.065702
## MARS 0.1864 -0.003117
## Enet 0.3965 1.0000
##
## Rsquared
## SVM MARS Enet
## SVM 0.032595 0.026803
## MARS 0.4750 -0.005791
## Enet 0.8051 1.0000
The SVM’s RMSE is lower than the elastic net’s by about 0.07, but the paired comparison gives an adjusted p-value of about 0.40 for that difference, and MARS is practically equal to the elastic net, so the resamples give no evidence that either nonlinear model predicts better than the linear one. The tested nonlinear models did not demonstrate a clear improvement over the elastic net with this sample and tuning procedure; similar predictive performance does not establish the underlying form of the relationship. The SVM is still the best of the four nonlinear models, and its test RMSE is also about 0.07 lower than the elastic net’s, but one small test set cannot confirm a gain the resamples do not show.
Choosing between near-ties. The resamples cannot separate the SVM, MARS and the elastic net, so a simpler model is the better practical choice when predictive error is tied. The SVM is called the best model above because it has the lowest resampled RMSE and the exercise asks for the optimal nonlinear model, and its importance list is used for part (b). If one model had to be used, though, MARS (four terms, two predictors) or the elastic net (linear, 13 nonzero coefficients) would be preferable to the SVM, which uses all 56 predictors and cannot be written as a short equation. MARS is the smallest of the three; the elastic net is the easiest to explain to a plant engineer. In Exercise 7.2 there is no such tie at the top: MARS beats the other three clearly, and it is also the model that matches the true structure.
top10_table <- data.frame(
Rank = 1:10,
Nonlinear = nl_top10,
Linear = lin_top10)
top10_table
## Rank Nonlinear Linear
## 1 1 ManufacturingProcess32 ManufacturingProcess32
## 2 2 BiologicalMaterial06 ManufacturingProcess09
## 3 3 ManufacturingProcess36 ManufacturingProcess36
## 4 4 BiologicalMaterial03 ManufacturingProcess17
## 5 5 ManufacturingProcess09 ManufacturingProcess34
## 6 6 ManufacturingProcess13 BiologicalMaterial06
## 7 7 ManufacturingProcess31 ManufacturingProcess11
## 8 8 BiologicalMaterial02 ManufacturingProcess06
## 9 9 ManufacturingProcess11 ManufacturingProcess13
## 10 10 BiologicalMaterial12 BiologicalMaterial03
shared <- intersect(nl_top10, lin_top10)
unique_nonlinear <- setdiff(nl_top10, lin_top10)
unique_linear <- setdiff(lin_top10, nl_top10)
list(shared = shared, only_nonlinear = unique_nonlinear, only_linear = unique_linear)
## $shared
## [1] "ManufacturingProcess32" "BiologicalMaterial06" "ManufacturingProcess36"
## [4] "BiologicalMaterial03" "ManufacturingProcess09" "ManufacturingProcess13"
## [7] "ManufacturingProcess11"
##
## $only_nonlinear
## [1] "ManufacturingProcess31" "BiologicalMaterial02" "BiologicalMaterial12"
##
## $only_linear
## [1] "ManufacturingProcess17" "ManufacturingProcess34" "ManufacturingProcess06"
table(Linear = ifelse(grepl("^Biological", lin_top10), "Biological", "Process"))
## Linear
## Biological Process
## 2 8
Neither group dominates the SVM’s list: six of its top ten are
process predictors and four are biological, with
ManufacturingProcess32 first and
BiologicalMaterial06 second. Process predictors lead in the
linear model’s list (eight of ten, two biological). Process predictors
also lead in the MARS model, which uses only two of them. Because the
SVM score is a one-predictor-at-a-time filter, the biological predictors
rank high partly because they are correlated with each other and with
yield individually; the elastic net, which holds the other predictors
fixed, has only two biological predictors among its top ten.
The two top-ten lists share seven predictors:
ManufacturingProcess32, 09, 36,
13 and 11, and
BiologicalMaterial06 and 03. All three
approaches (the SVM filter, MARS and the elastic net) place
ManufacturingProcess32 first or near the top and
ManufacturingProcess09 high. Three predictors are in the
nonlinear top ten only: ManufacturingProcess31,
BiologicalMaterial02 and BiologicalMaterial12.
Three are in the linear top ten only:
ManufacturingProcess17, 34 and
06. Since the lists are built from different measures (a
filter on one predictor at a time against coefficients of a penalized
multiple regression), differences between them partly reflect the
measures and not only linearity versus nonlinearity.
Explore the relationships between the top predictors and the response for the predictors that are unique to the optimal nonlinear regression model. Do these plots reveal intuition about the biological or process predictors and their relationship with yield?
The plots use the original, un-imputed and un-scaled values, so the
axes are in process units. The smooth curve is a loess smoother, as in
the book’s featurePlot() examples. These plots are
descriptive and were made after the test set evaluation; they are not
used for tuning.
unique_plot <- head(unique_nonlinear, 6)
featurePlot(x = ChemicalManufacturingProcess[, unique_plot, drop = FALSE],
y = ChemicalManufacturingProcess$Yield,
plot = "scatter", type = c("p", "smooth"),
labels = c("Predictor value", "Yield (%)"),
between = list(x = 1, y = 1),
layout = c(min(3, length(unique_plot)), ceiling(length(unique_plot) / 3)))
The ManufacturingProcess31 panel is dominated by run
108, so the same plot is repeated without that run:
no108 <- ChemicalManufacturingProcess[ChemicalManufacturingProcess$ManufacturingProcess31 > 0 &
!is.na(ChemicalManufacturingProcess$ManufacturingProcess31), ]
featurePlot(x = no108[, "ManufacturingProcess31", drop = FALSE], y = no108$Yield,
plot = "scatter", type = c("p", "smooth"),
labels = c("Predictor value", "Yield (%)"))
round(sapply(unique_nonlinear, function(v)
cor(ChemicalManufacturingProcess[[v]], ChemicalManufacturingProcess$Yield,
use = "complete.obs")), 3)
## ManufacturingProcess31 BiologicalMaterial02 BiologicalMaterial12
## -0.069 0.482 0.367
# the one run with ManufacturingProcess31 equal to zero
zero31 <- which(ChemicalManufacturingProcess$ManufacturingProcess31 == 0)
zero31
## [1] 108
ChemicalManufacturingProcess[zero31, c("Yield", "ManufacturingProcess31")]
## Yield ManufacturingProcess31
## 108 39.75 0
c(zeros_in_that_run = sum(process_predictors[zero31, ] == 0, na.rm = TRUE),
in_training_set = zero31 %in% cmp_rows)
## zeros_in_that_run in_training_set
## 11 1
# correlation with yield with and without that run
c(with_run = cor(ChemicalManufacturingProcess$ManufacturingProcess31,
ChemicalManufacturingProcess$Yield, use = "complete.obs"),
without_run = cor(ChemicalManufacturingProcess$ManufacturingProcess31[-zero31],
ChemicalManufacturingProcess$Yield[-zero31], use = "complete.obs"))
## with_run without_run
## -0.06890417 -0.36587278
# how the two biological predictors relate to the other biological predictors
bio <- ChemicalManufacturingProcess[, grep("^Biological", names(ChemicalManufacturingProcess))]
round(cor(bio)[c("BiologicalMaterial02", "BiologicalMaterial12"),
c("BiologicalMaterial01", "BiologicalMaterial03", "BiologicalMaterial06",
"BiologicalMaterial12", "BiologicalMaterial02")], 2)
## BiologicalMaterial01 BiologicalMaterial03
## BiologicalMaterial02 0.74 0.86
## BiologicalMaterial12 0.52 0.70
## BiologicalMaterial06 BiologicalMaterial12
## BiologicalMaterial02 0.95 0.78
## BiologicalMaterial12 0.81 1.00
## BiologicalMaterial02
## BiologicalMaterial02 1.00
## BiologicalMaterial12 0.78
The unique predictors are ManufacturingProcess31,
BiologicalMaterial02 and
BiologicalMaterial12.
ManufacturingProcess31 is a process
predictor and can be adjusted. Almost every run has a value between
about 68 and 73, so the plot is a narrow vertical band, and one run (run
108, yield 39.75) has the value 0 along with zeros in ten other process
predictors. That one run bends the loess curve in the first panel so far
that it leaves the plotted range, and a second run near 60 stretches the
left end of the curve. Without run 108 (second plot) the smooth curve
falls steadily from the low end of the band and flattens near its upper
end, so higher values go with lower yield over that range. Run 108 pulls
the overall correlation with yield to -0.07. Without it the correlation
is -0.37, a clearly negative relationship: runs with a higher value tend
to have lower yield. A linear model fitted with that run present sees a
weak relationship, which is consistent with
ManufacturingProcess31 missing from the elastic net’s top
ten, although this plot cannot prove that is the reason. The run is in
the training set, so it was seen by every model during fitting. In
practice a run with eleven predictors at exactly zero looks like a
recording or sensor problem and not a real operating point, and it
should be checked in the plant records before it is trusted.
BiologicalMaterial02 rises steadily
with yield: the smooth curve climbs from the lowest to the highest
values (roughly 38 to 42.5 on the yield axis), a correlation of about
0.48, with a lot of scatter around it.
BiologicalMaterial12 is curved: the smooth
curve rises toward a peak near a value of 21 and then declines modestly
at the high end, where there are relatively few runs (correlation about
0.37). That bend is the kind of shape a nonlinear model can follow and a
straight line cannot, but with so few runs at the high end the downturn
should be interpreted cautiously. Both biological predictors are
strongly correlated with the other biological predictors
(BiologicalMaterial02 with 06 at 0.95 and with
03 at 0.86, BiologicalMaterial12 with
06 at 0.81), so each is partly the same signal as
BiologicalMaterial06 and 03, which both lists
already rank highly. Their separate effects cannot be read from these
plots.
Taken together, the plots give some intuition but not a recipe. The
biological predictors cannot be changed, but higher
BiologicalMaterial02 and a middle-to-high
BiologicalMaterial12 go with higher yield on average, so
they may be worth watching when raw material batches are screened,
although the plots do not establish a threshold. The only unique process
predictor, ManufacturingProcess31, points to something that
could be controlled (lower values go with higher yield once the zero run
is set aside), but the data are observational, so the way to confirm it
is a designed experiment in the plant. With a 1% gain in yield worth
about one hundred thousand dollars per batch, a confirmed improvement
from one process setting would be worth that experiment.