All four exercises are from Chapter 8 of Kuhn and Johnson, Applied Predictive Modeling (regression trees and rule-based models). Exercises 8.1 and 8.3 are about how tree ensembles measure predictor importance, 8.2 is about a known bias in how single trees choose their splits, and 8.7 returns to the chemical manufacturing data from the last two homeworks.
Recreate the simulated data from Exercise 7.2:
library(mlbench)
set.seed(200)
simulated <- mlbench.friedman1(200, sd = 1)
simulated <- cbind(simulated$x, simulated$y)
simulated <- as.data.frame(simulated)
colnames(simulated)[ncol(simulated)] <- "y"
The response is \(y = 10 \sin(\pi x_1 x_2) + 20 (x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + N(0, 1)\), so V1 to V5 are informative and V6 to V10 are pure noise. That known answer is what makes this a fair test of the importance measures.
Fit a random forest model to all of the predictors, then estimate the variable importance scores:
library(randomForest) library(caret) model1 <- randomForest(y ~ ., data = simulated, importance = TRUE, ntree = 1000) rfImp1 <- varImp(model1, scale = FALSE)Did the random forest model significantly use the uninformative predictors (V6 – V10)?
# the book's code; the seed is added so the forest and its importance
# values can be reproduced
set.seed(624)
model1 <- randomForest(y ~ ., data = simulated,
importance = TRUE,
ntree = 1000)
rfImp1 <- varImp(model1, scale = FALSE)
rfImp1[order(-rfImp1$Overall), , drop = FALSE]
## Overall
## V1 8.950480717
## V4 7.848396144
## V2 6.560901913
## V5 2.143163628
## V3 0.740942146
## V6 0.091400119
## V7 0.076258170
## V10 0.009408601
## V9 -0.032033548
## V8 -0.153191514
varImp() on a randomForest object returns
the permutation importance described in the book: for each tree, the
out-of-bag values of one predictor are shuffled and the increase in the
out-of-bag mean squared error is recorded, then averaged over the
forest. A predictor the forest does not rely on gives an increase near
zero, and can even give a small negative value by chance.
A single forest is one random draw, so the size of the noise scores is easier to judge when the forest is refit with a few different seeds:
# five more forests on the same data, to see how much the scores move from seed to seed
rf_seed_imp <- fit_once("rf_seed_imp", sapply(1:5, function(s) {
set.seed(s)
fit <- randomForest(y ~ ., data = simulated, importance = TRUE, ntree = 1000)
varImp(fit, scale = FALSE)$Overall
}))
rownames(rf_seed_imp) <- rownames(rfImp1)
round(cbind(min = apply(rf_seed_imp, 1, min), max = apply(rf_seed_imp, 1, max)), 2)
## min max
## V1 8.53 8.98
## V2 6.54 6.78
## V3 0.61 0.83
## V4 7.52 7.83
## V5 2.05 2.28
## V6 0.08 0.17
## V7 -0.02 0.07
## V8 -0.17 -0.10
## V9 -0.02 0.03
## V10 -0.17 0.10
No. The five informative predictors have clearly positive scores (V1, V4 and V2 highest, then V5 and V3), while V6 to V10 sit around zero, between about -0.17 and 0.17 across the six forests. The weakest informative predictor, V3, never falls below 0.61 in any of the forests, so there is a clear gap between the known signal and the known noise. The small negative values for V8 mean that shuffling it made the out-of-bag error slightly smaller. That is consistent with V8 being noise here, which the simulation formula tells us, although a negative estimate on its own would not prove that a predictor carries no information. The forest does split on the noise predictors now and then (with 1,000 deep trees it has to split on something), but those splits add essentially nothing to prediction. The repeated seeds show how much the scores move from one forest to the next on this one data set; they are not a formal significance test, so the fair summary is that V6 to V10 get no substantial importance compared with V1 to V5.
V3 is low compared with V1, V2 and V4 because its effect, \(20 (x_3 - 0.5)^2\), is a U shape centered in the middle of its range and contributes at most 5 units to \(y\), against 10 for V4 and up to 10 for the sine term.
Use the
cforestfunction in thepartypackage to fit a random forest model using conditional inference trees. Thepartypackage functionvarimpcan calculate predictor importance. Theconditionalargument of that function toggles between the traditional importance measure and the modified version described in Strobl et al. (2007). Do these importances show the same pattern as the traditional random forest model?
cforest() is fitted to the same three data sets with the
same number of trees as the random forests.
cforest_unbiased() is the control setting the
party package recommends for importance (subsampling
without replacement); its default mtry is 5.
cf_imp <- fit_once("cf_imp", lapply(list(base = sim_base, dup1 = sim_dup1, dup2 = sim_dup2),
function(d) {
set.seed(624)
cf <- party::cforest(y ~ ., data = d, controls = cforest_unbiased(ntree = 1000))
# traditional (marginal) permutation importance and the conditional version
list(traditional = varimp(cf),
conditional = varimp(cf, conditional = TRUE))
}))
as_imp <- function(v) data.frame(Overall = v, row.names = names(v))
cf_trad_table <- importance_table(as_imp(cf_imp$base$traditional),
as_imp(cf_imp$dup1$traditional),
as_imp(cf_imp$dup2$traditional))
cf_cond_table <- importance_table(as_imp(cf_imp$base$conditional),
as_imp(cf_imp$dup1$conditional),
as_imp(cf_imp$dup2$conditional))
cf_trad_table
## original plus_duplicate1 plus_both_duplicates
## V1 8.70 6.46 6.13
## V2 6.41 6.02 5.84
## V3 0.02 0.02 0.02
## V4 8.36 7.75 7.34
## V5 1.80 1.79 1.61
## V6 -0.02 -0.01 0.01
## V7 0.08 0.06 0.02
## V8 -0.03 -0.03 0.00
## V9 -0.02 -0.01 -0.02
## V10 -0.01 -0.04 -0.05
## duplicate1 NA 2.96 2.53
## duplicate2 NA NA 1.03
cf_cond_table
## original plus_duplicate1 plus_both_duplicates
## V1 5.53 2.96 2.50
## V2 5.27 4.81 4.56
## V3 0.01 0.00 0.03
## V4 6.61 6.06 5.76
## V5 1.14 1.12 0.91
## V6 0.00 0.00 0.02
## V7 -0.01 0.01 0.00
## V8 -0.01 0.00 0.00
## V9 -0.01 0.01 0.01
## V10 0.00 -0.03 -0.03
## duplicate1 NA 0.93 0.77
## duplicate2 NA NA 0.19
The traditional cforest importance shows the same
pattern as the random forest. V1 falls from about 8.7 to 6.5 when
duplicate1 is added and to 6.1 with both copies, and the
copies pick up about 3.0, then 2.5 and 1.0. The noise predictors V6 to
V10 stay near zero again.
The conditional importance changes the picture. The copies get much
less credit: duplicate1 falls to about 0.9 and
duplicate2 to about 0.2, because the conditional version
permutes each predictor within groups defined by the predictors it is
correlated with, so a copy only gets credit for what it adds beyond V1.
This is the redundancy correction the book describes. V1 itself, though,
still drops, from about 5.5 to 3.0 and 2.5. The conditional measure
removes most of the credit given to the copies but does not give it back
to V1, which matches the book’s statement that the method reduces the
effect of redundancy but does not adjust for dilution.
One difference from the random forest has nothing to do with the
copies: cforest gives V3 an importance of almost exactly
zero in every fit. Conditional inference trees choose splits with a test
of association between the predictor and the response. V3’s effect is a
symmetric U shape around 0.5, so its overall linear association with
\(y\) is close to zero and the test
rarely selects it, while CART’s exhaustive SSE search can still find
splits that pick up part of the U shape. That is likely a cost of the
unbiased splitting rule when an effect is not monotone.
Repeat this process with different tree models, such as boosted trees and Cubist. Does the same pattern occur?
Both models are tuned by 10-fold cross-validation on each of the three data sets. The boosting grid follows the book’s (depth 1 to 7, 100 to 1,000 trees, shrinkage 0.01 or 0.1) and the Cubist grid covers 1 to 100 committees with and without neighbor adjustment.
sim_ctrl <- trainControl(method = "cv", number = 10)
sim_gbm_grid <- expand.grid(interaction.depth = seq(1, 7, by = 2),
n.trees = seq(100, 1000, by = 50),
shrinkage = c(0.01, 0.1),
n.minobsinnode = 10)
sim_cubist_grid <- expand.grid(committees = c(1, 10, 50, 100), neighbors = c(0, 5, 9))
sim_boost_cubist <- fit_once("sim_boost_cubist", lapply(
list(base = sim_base, dup1 = sim_dup1, dup2 = sim_dup2), function(d) {
x <- d[, names(d) != "y"]
set.seed(624)
gbm_fit <- train(x, d$y, method = "gbm", tuneGrid = sim_gbm_grid,
trControl = sim_ctrl, verbose = FALSE)
set.seed(624)
cubist_fit <- train(x, d$y, method = "cubist", tuneGrid = sim_cubist_grid,
trControl = sim_ctrl)
list(gbm = gbm_fit, cubist = cubist_fit)
}))
lapply(sim_boost_cubist, function(m) list(gbm = m$gbm$bestTune, cubist = m$cubist$bestTune))
## $base
## $base$gbm
## n.trees interaction.depth shrinkage n.minobsinnode
## 38 1000 3 0.01 10
##
## $base$cubist
## committees neighbors
## 12 100 9
##
##
## $dup1
## $dup1$gbm
## n.trees interaction.depth shrinkage n.minobsinnode
## 38 1000 3 0.01 10
##
## $dup1$cubist
## committees neighbors
## 12 100 9
##
##
## $dup2
## $dup2$gbm
## n.trees interaction.depth shrinkage n.minobsinnode
## 38 1000 3 0.01 10
##
## $dup2$cubist
## committees neighbors
## 9 50 9
For boosting, varImp() is the reduction in squared error
from each predictor’s splits, summed over the trees. For Cubist, caret’s
varImp() averages two of Cubist’s usage statistics: how
much a predictor is used in rule conditions and how much it is used in
the rules’ linear models (the book says Cubist has no established
importance measure and suggests counts of this kind). The result is a
usage score, not the percentage of rules that contain the predictor.
gbm_imp_table <- importance_table(varImp(sim_boost_cubist$base$gbm, scale = FALSE)$importance,
varImp(sim_boost_cubist$dup1$gbm, scale = FALSE)$importance,
varImp(sim_boost_cubist$dup2$gbm, scale = FALSE)$importance)
cubist_usage_of <- function(fit) varImp(fit, scale = FALSE)$importance
cubist_imp_table <- importance_table(cubist_usage_of(sim_boost_cubist$base$cubist),
cubist_usage_of(sim_boost_cubist$dup1$cubist),
cubist_usage_of(sim_boost_cubist$dup2$cubist))
gbm_imp_table
## original plus_duplicate1 plus_both_duplicates
## V1 39802.70 30326.38 28720.72
## V2 34075.90 33882.15 33853.83
## V3 11873.87 11708.40 11670.94
## V4 44228.29 44736.52 44775.94
## V5 18079.63 17475.84 17630.25
## V6 1899.34 1901.75 1772.22
## V7 1758.72 1579.90 1581.57
## V8 890.91 1021.66 1034.96
## V9 932.02 915.83 1010.20
## V10 606.90 597.86 553.15
## duplicate1 NA 11377.79 10504.76
## duplicate2 NA NA 2778.53
cubist_imp_table
## original plus_duplicate1 plus_both_duplicates
## V1 71.5 64.0 61.0
## V2 58.5 60.0 64.5
## V3 47.0 42.0 33.5
## V4 48.0 47.5 50.5
## V5 33.0 27.5 39.0
## V6 0.0 11.0 13.5
## V7 0.0 0.0 0.0
## V8 0.0 1.5 3.0
## V9 0.0 0.0 0.0
## V10 0.0 0.0 0.0
## duplicate1 NA 5.0 4.0
## duplicate2 NA NA 21.0
The models measure importance on different scales, so to compare the pattern across them the table below gives V1’s importance as a share of the total importance in each fit, and the combined share of V1 and its copies:
v1_share <- function(tab) {
# descriptive shares: absent predictors and negative scores set to zero
tab[is.na(tab) | tab < 0] <- 0
shares <- sweep(tab, 2, colSums(tab), "/")
rbind(V1 = shares["V1", ],
V1_and_copies = colSums(shares[c("V1", "duplicate1", "duplicate2"), ]))
}
share_tables <- list("random forest" = rf_imp_table, "cforest, traditional" = cf_trad_table,
"cforest, conditional" = cf_cond_table, "boosted trees" = gbm_imp_table,
"Cubist" = cubist_imp_table)
v1_shares <- do.call(rbind, lapply(names(share_tables), function(nm)
data.frame(model = nm, measure = c("V1", "V1 and copies"),
round(v1_share(share_tables[[nm]]), 3), row.names = NULL)))
v1_shares
## model measure original plus_duplicate1
## 1 random forest V1 0.339 0.235
## 2 random forest V1 and copies 0.339 0.376
## 3 cforest, traditional V1 0.343 0.258
## 4 cforest, traditional V1 and copies 0.343 0.376
## 5 cforest, conditional V1 0.298 0.186
## 6 cforest, conditional V1 and copies 0.298 0.245
## 7 boosted trees V1 0.258 0.195
## 8 boosted trees V1 and copies 0.258 0.268
## 9 Cubist V1 0.277 0.248
## 10 Cubist V1 and copies 0.277 0.267
## plus_both_duplicates
## 1 0.221
## 2 0.404
## 3 0.250
## 4 0.395
## 5 0.169
## 6 0.235
## 7 0.184
## 8 0.269
## 9 0.210
## 10 0.297
Boosted trees. The same pattern occurs. The selected
model (1,000 trees, depth 3, shrinkage 0.01) is the same on all three
data sets. V1’s importance falls from about 39,800 to 30,300 with one
copy and 28,700 with two, duplicate1 takes about 11,400 and
then 10,500, and duplicate2 only about 2,800. V1’s share of
the total falls from 0.26 to 0.20 and 0.18. Boosting gives the second
copy even less than the random forest does. A likely reason is that each
tree is fit to the residuals of the trees before it, so once V1 and
duplicate1 have been used to fit that part of the signal,
little is left for duplicate2.
Cubist. The pattern is weaker but in the same
direction. V1’s usage score falls from 71.5 to 64 and 61, and its share
from 0.28 to 0.25 and 0.21. duplicate1 has a usage score of
only 4 to 5, while duplicate2 reaches 21 in the last model,
so the usage is uneven between the copies and changes from one fit to
the next. Usage scores count how often a predictor appears in rule
conditions or linear models, not how much it reduces the error, so a
copy that enters many linear models with a small coefficient can look
important. Cubist also gives the noise predictor V6 a usage score of 11
to 14 on the data with copies, the only model here where a noise
predictor gets a noticeable score, which is another reason to treat
Cubist’s usage as a rough guide (the book says as much).
Across all five measures, adding a near-copy of an important predictor lowers that predictor’s own score and its share. The shares only describe how each measure redistributes its own scores; they do not turn permutation loss, squared-error reduction and usage counts into the same quantity. The conditional importance is the one that keeps the copies from collecting a large share (together about 0.07 of the total in the last fit), but V1’s share under it still falls from 0.30 to 0.19 and 0.17, so it cuts the credit given to redundant copies without restoring V1’s standing, as the book says. None of the measures recover V1’s original standing. In practice that means correlated predictors should be found and handled (removed or combined) before reading any tree ensemble’s importance list.
Use a simulation to show tree bias with different granularities.
The book says single trees suffer from selection bias: a predictor with many distinct values offers many more candidate split points than a coarse one, so it has more chances to find a split that lowers the SSE by luck. The simulation builds four predictors that differ only in granularity:
x_binary: two values (0 or 1), so one possible
splitx_10: whole numbers 1 to 10, nine possible splitsx_50: whole numbers 1 to 50, up to 49 possible
splitsx_cont: uniform on 0 to 1, so every one of the 200
values is distinct, giving up to 199 boundaries between ordered values
(rpart’s minimum node size of 7 rules out the few nearest
the ends)All four are generated independently. In the first setting the
response is pure noise, so no predictor is related to it, and an
unbiased method that has to choose a predictor would pick each one about
a quarter of the time. In the next two settings only the binary
predictor is related to the response: the mean of \(y\) is 0.3 higher, then 0.5 higher, when
x_binary is 1, in units of the noise standard deviation
(which is 1). Any split on the other three predictors is a split on
noise. Each setting is simulated 500 times, and each time a one-split
CART tree (rpart) and a one-split conditional inference
tree (ctree) record which predictor they choose for the
first split.
make_granularity_data <- function(n = 200, effect = 0) {
d <- data.frame(x_binary = sample(0:1, n, replace = TRUE),
x_10 = sample(1:10, n, replace = TRUE),
x_50 = sample(1:50, n, replace = TRUE),
x_cont = runif(n))
d$y <- effect * d$x_binary + rnorm(n) # only the binary predictor carries any signal
d
}
# cp = 0 and maxdepth = 1 force CART to make its single best split,
# so the choice can be recorded
first_split_rpart <- function(d) {
fit <- rpart(y ~ ., data = d, control = rpart.control(maxdepth = 1, cp = 0))
as.character(fit$frame$var[1])
}
# ctree only splits when the best predictor passes its significance test (1 - p > 0.95)
first_split_ctree <- function(d) {
fit <- party::ctree(y ~ ., data = d, controls = party::ctree_control(maxdepth = 1))
if (fit@tree$terminal) "no split" else fit@tree$psplit$variableName
}
set.seed(624)
gran_results <- do.call(rbind, lapply(c(0, 0.3, 0.5), function(effect) {
picks <- replicate(500, {
d <- make_granularity_data(effect = effect)
c(rpart = first_split_rpart(d), ctree = first_split_ctree(d))
})
data.frame(effect = effect, method = rep(c("CART (rpart)", "conditional (ctree)"), 500),
first_split = as.vector(picks))
}))
gran_levels <- c("x_binary", "x_10", "x_50", "x_cont", "no split")
gran_results$first_split <- factor(gran_results$first_split, levels = gran_levels)
gran_table <- with(gran_results, table(effect, method, first_split))
ftable(gran_table)
## first_split x_binary x_10 x_50 x_cont no split
## effect method
## 0 CART (rpart) 21 90 174 215 0
## conditional (ctree) 12 3 6 10 469
## 0.3 CART (rpart) 207 47 95 151 0
## conditional (ctree) 173 3 2 5 317
## 0.5 CART (rpart) 420 10 28 42 0
## conditional (ctree) 428 1 1 1 69
par(mfrow = c(1, 3), mar = c(5, 4, 3, 1))
for (e in c(0, 0.3, 0.5)) {
counts <- gran_table[as.character(e), , ]
barplot(counts, beside = TRUE, col = c("steelblue", "darkorange"), ylim = c(0, 500),
las = 2, ylab = "Times chosen (out of 500)",
main = paste("Effect of x_binary =", e))
if (e == 0) legend("topleft", c("CART", "ctree"), fill = c("steelblue", "darkorange"),
bty = "n")
}
par(mfrow = c(1, 1))
With a pure-noise response, a method that is forced to split and has no preference would pick each predictor about 125 times out of 500. CART picks the continuous predictor 215 times, the 50-level predictor 174 times, the 10-level predictor 90 times and the binary predictor only 21 times, so the more distinct values a predictor has, the more often it wins the first split. The likely reason is the number of chances: CART evaluates every allowed split point and keeps the one with the lowest SSE, so the continuous predictor gets up to 199 tries at a lucky reduction and the binary predictor gets one. Here cp = 0 deliberately forces a split so that the choice can be recorded. Choosing the tree size by resampling can remove some unnecessary splits, but that comparison is not part of this simulation, and in a deeper tree the same bias acts at every node.
With a real effect on the binary predictor, the bias reduces how often CART picks the informative predictor first. With an effect of 0.3 CART picks the binary predictor only 207 times out of 500; in the other 293 runs it splits first on a predictor that has no relation to the response, mostly the continuous one. Only when the effect grows to 0.5 does the informative predictor win most of the time (420 of 500). This is the danger the book warns about: when informative predictors are coarse and noise predictors are granular, the noise ends up at the top of the tree.
The conditional inference tree behaves very
differently. With pure noise it makes no split in 469 of 500 runs. It
splits only 31 times (about 6%, close to the 5% false-positive rate of
its 0.95 threshold), and those splits are spread over all four
predictors (12, 3, 6 and 10 times), with no sign of a preference for the
granular ones, although 31 splits are too few to show the choice is
exactly even. With an effect of 0.3 it splits less often (183 times),
but when it splits it chooses the binary predictor 173 times out of 183.
With an effect of 0.5 it picks the binary predictor 428 times and a
noise predictor only 3 times. ctree first tests the
association between the response and each candidate predictor, with a
multiplicity adjustment for the decision whether to split at all. If it
does split, it takes the predictor with the strongest association and
only then chooses that predictor’s split point. Separating the choice of
predictor from the choice of cut point removes the advantage that CART’s
exhaustive search gives predictors with more possible splits, which is
the correction the book describes. The cost is power: with a weak effect
it often makes no split at all.
In stochastic gradient boosting the bagging fraction and learning rate will govern the construction of the trees as they are guided by the gradient. Although the optimal values of these parameters should be obtained through the tuning process, it is helpful to understand how the magnitudes of these parameters affect magnitudes of variable importance. Figure 8.24 provides the variable importance plots for boosting using two extreme values for the bagging fraction (0.1 and 0.9) and the learning rate (0.1 and 0.9) for the solubility data. The left-hand plot has both parameters set to 0.1, and the right-hand plot has both set to 0.9:
The answers below are reasoned from how the algorithm works and then checked by refitting both models on the solubility data. The book does not state the number of trees or the depth behind Figure 8.24, so the refit uses the settings of the book’s tuned boosted model (400 trees of depth 3) with the two extreme combinations.
data(solubility)
sol_settings <- data.frame(name = c("slow (0.1, 0.1)", "fast (0.9, 0.9)"),
bag.fraction = c(0.1, 0.9), shrinkage = c(0.1, 0.9))
sol_gbm <- fit_once("sol_gbm", lapply(1:2, function(i) {
set.seed(100)
gbm.fit(solTrainXtrans, solTrainY, distribution = "gaussian",
n.trees = 400, interaction.depth = 3,
shrinkage = sol_settings$shrinkage[i],
bag.fraction = sol_settings$bag.fraction[i], verbose = FALSE)
}))
names(sol_gbm) <- sol_settings$name
# relative influence: each predictor's share (in percent) of the total squared-error reduction
sol_imp <- lapply(sol_gbm, function(m) summary(m, plotit = FALSE))
par(mfrow = c(1, 2), mar = c(4, 9, 3, 1))
for (nm in names(sol_imp)) {
top <- head(sol_imp[[nm]], 25)
barplot(rev(top$rel.inf), names.arg = rev(as.character(top$var)), horiz = TRUE, las = 1,
cex.names = 0.7, col = "steelblue", xlab = "Relative influence (%)",
main = nm)
}
par(mfrow = c(1, 1))
importance_slope <- function(imp) {
c(top1 = imp$rel.inf[1], top5 = sum(imp$rel.inf[1:5]), top10 = sum(imp$rel.inf[1:10]),
used_predictors = sum(imp$rel.inf > 0))
}
round(sapply(sol_imp, importance_slope), 1)
## slow (0.1, 0.1) fast (0.9, 0.9)
## top1 19.3 35.6
## top5 44.0 79.0
## top10 56.4 85.6
## used_predictors 172.0 197.0
Why does the model on the right focus its importance on just the first few of predictors, whereas the model on the left spreads importance across more predictors?
The two parameters push in the same direction. The two fits change
both at once, so this comparison cannot separate their effects; what
follows is the likely mechanism. The learning rate decides how much of
each new tree’s prediction is added to the model. At 0.9, the first few
trees add most of what they find, and what they find is driven by the
strongest predictors, which here are the continuous ones like
MolWeight, NumCarbon and the surface areas.
The next trees are fit to the residuals, and the same strong predictors
tend to keep winning splits there, collecting more importance. At 0.1,
each tree adds only a tenth of what it found, so the strong predictors
still have signal left for many more iterations, and as the residuals
shrink slowly other predictors get turns as well.
The bagging fraction decides how much of the training data each tree sees. At 0.9 every tree sees almost the same data, so consecutive trees tend to make similar choices and the importance concentrates. At 0.1 each tree sees a different 10% of the samples (about 95 of 951 compounds), so the best split varies more from tree to tree and the importance is shared more evenly among the predictors that carry signal. The book makes the same point about the steep boosting importance curve: trees that depend on each other have correlated structure and keep picking the same predictors.
The refit confirms this. In the fast model the top predictor holds about 36% of the total relative influence and the top five about 79%; in the slow model the top predictor holds about 19% and the top five 44%. The plot shows the same steep drop on the right and gradual drop on the left as Figure 8.24. The fast model does give some importance to more predictors in total (197 against 172 with any importance), but most of that is tiny and in the tail; the slope at the top is what differs.
Which model do you think would be more predictive of other samples?
I would expect the left-hand model to predict new samples better. A learning rate of 0.9 is close to no shrinkage at all, and the book describes boosting without regularization as greedy and prone to over-fitting: each step fits the current residuals as well as it can, including the noise in them. The book recommends a learning rate below 0.01 and a bagging fraction around 0.5, so the left-hand settings are much closer to the recommended ones on the learning rate. A bagging fraction of 0.1 is low and makes each tree noisy, but the small learning rate averages that noise over many trees, which is the point of stochastic gradient boosting.
To check, both settings are evaluated by 10-fold cross-validation on the training set and on the solubility test set:
sol_ctrl <- trainControl(method = "cv", number = 10)
sol_cv <- fit_once("sol_cv", lapply(1:2, function(i) {
set.seed(100)
train(solTrainXtrans, solTrainY, method = "gbm",
tuneGrid = data.frame(n.trees = 400, interaction.depth = 3,
shrinkage = sol_settings$shrinkage[i], n.minobsinnode = 10),
bag.fraction = sol_settings$bag.fraction[i],
trControl = sol_ctrl, verbose = FALSE)
}))
names(sol_cv) <- sol_settings$name
sol_perf <- t(sapply(names(sol_gbm), function(nm) {
c(CV_RMSE = sol_cv[[nm]]$results$RMSE, CV_R2 = sol_cv[[nm]]$results$Rsquared,
postResample(predict(sol_gbm[[nm]], solTestXtrans, n.trees = 400), solTestY))
}))
round(sol_perf, 3)
## CV_RMSE CV_R2 RMSE Rsquared MAE
## slow (0.1, 0.1) 0.661 0.896 0.698 0.888 0.515
## fast (0.9, 0.9) 0.781 0.857 0.744 0.876 0.530
The slow model is better on both: cross-validated RMSE 0.661 against 0.781, and test RMSE 0.698 against 0.744, with a higher R-squared on both. The difference on the test set is smaller than on resampling, but it points the same way. Neither setting would be the final choice; both are extreme, and the book’s tuned model (shrinkage 0.1, bagging fraction 0.5) sits between them.
How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?
Interaction depth sets the maximum depth of each tree, so deeper trees allow more splits and higher-order interactions. Those extra splits can bring in new predictors or reuse ones already in the tree, so a flatter importance profile is possible but not guaranteed. My expectation is that greater depth spreads importance across more predictors, with a bigger effect on the slow model, because in the fast model the first trees already take most of the signal from the top predictors. Both settings are refit at depths 1, 3 and 7:
sol_depth <- fit_once("sol_depth", lapply(1:2, function(i) {
lapply(c(1, 3, 7), function(depth) {
set.seed(100)
fit <- gbm.fit(solTrainXtrans, solTrainY, distribution = "gaussian",
n.trees = 400, interaction.depth = depth,
shrinkage = sol_settings$shrinkage[i],
bag.fraction = sol_settings$bag.fraction[i], verbose = FALSE)
summary(fit, plotit = FALSE)
})
}))
names(sol_depth) <- sol_settings$name
depth_table <- do.call(rbind, lapply(names(sol_depth), function(nm) {
t(sapply(1:3, function(j) c(depth = c(1, 3, 7)[j], importance_slope(sol_depth[[nm]][[j]]))))
}))
data.frame(model = rep(names(sol_depth), each = 3), round(depth_table, 1))
## model depth top1 top5 top10 used_predictors
## 1 slow (0.1, 0.1) 1 21.3 54.7 64.9 133
## 2 slow (0.1, 0.1) 3 19.3 44.0 56.4 172
## 3 slow (0.1, 0.1) 7 19.2 45.1 57.3 178
## 4 fast (0.9, 0.9) 1 40.7 76.2 85.0 118
## 5 fast (0.9, 0.9) 3 35.6 79.0 85.6 197
## 6 fast (0.9, 0.9) 7 36.2 78.0 86.1 225
par(mfrow = c(1, 2))
for (nm in names(sol_depth)) {
top_value <- max(sapply(sol_depth[[nm]], function(s) s$rel.inf[1]))
plot(NULL, xlim = c(1, 40), ylim = c(0, top_value),
xlab = "Predictor rank", ylab = "Relative influence (%)", main = nm)
cols <- c("gray40", "steelblue", "firebrick")
for (j in 1:3) lines(1:40, sol_depth[[nm]][[j]]$rel.inf[1:40], type = "b", pch = 19,
cex = 0.5, col = cols[j])
legend("topright", paste("depth", c(1, 3, 7)), col = cols, lwd = 1, pch = 19, bty = "n")
}
par(mfrow = c(1, 1))
The results partly match that expectation. In both models deeper trees bring in many more predictors: from 133 to 178 with any importance in the slow model, and from 118 to 225 in the fast one. In the slow model the top of the curve also flattens when going from depth 1 to depth 3 (the top five fall from about 55% to 44% of the importance), and depth 7 is about the same as depth 3. In the fast model the top of the curve barely changes: the top five hold about 76 to 79% at every depth, and the top predictor’s share drops only from 41% to about 36%. So increasing the interaction depth makes the tail of the importance curve longer and flatter for both models, but it only flattens the top of the curve for the slow model. A likely reason is that with a learning rate of 0.9 the first trees take most of the signal from the strongest predictors whatever their depth, so the extra splits in deeper trees land on predictors with little left to explain.
Refer to Exercises 6.3 and 7.5 which describe a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several tree-based models:
The data, the 80/20 split (seed 624), the removal of near-zero-variance predictors, and the KNN imputation fitted inside each training resample are the same as in Exercises 6.3 and 7.5.
data(ChemicalManufacturingProcess)
process_predictors <- ChemicalManufacturingProcess[, -1]
yield <- ChemicalManufacturingProcess$Yield
set.seed(624)
cmp_rows <- createDataPartition(yield, p = 0.80, list = FALSE)
cmp_train_x <- process_predictors[cmp_rows, ]
cmp_test_x <- process_predictors[-cmp_rows, ]
cmp_train_y <- yield[cmp_rows]
cmp_test_y <- yield[-cmp_rows]
cmp_nzv <- nearZeroVar(cmp_train_x, names = TRUE)
cmp_nzv
## [1] "BiologicalMaterial07"
cmp_train_x <- cmp_train_x[, !(names(cmp_train_x) %in% cmp_nzv)]
cmp_test_x <- cmp_test_x[, !(names(cmp_test_x) %in% cmp_nzv)]
c(train = nrow(cmp_train_x), test = nrow(cmp_test_x), predictors = ncol(cmp_train_x),
missing_cells_train = sum(is.na(cmp_train_x)))
## train test predictors missing_cells_train
## 144 32 56 99
Trees do not need centered and scaled predictors, but
preProc = "knnImpute" centers and scales as part of the
imputation. That changes nothing for a tree: a split on a standardized
value is the same split as on the raw value, only the threshold is
expressed in standard deviations. The KNN imputation is kept because the
exercise asks for the same imputation as Exercises 6.3 and 7.5, and it
puts every model on exactly the same data. randomForest
could not use the rows with missing values anyway (the book notes this),
while some of the other implementations have their own ways of handling
them.
Which tree-based regression model gives the optimal resampling and test set performance?
Seven tree-based models from the chapter, each tuned on the same
repeated cross-validation folds (the seed is reset before each
train() call):
cmp_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)
cmp_trees <- fit_once("cmp_trees", {
set.seed(100)
cmp_cart <- train(cmp_train_x, cmp_train_y, method = "rpart",
preProc = "knnImpute", tuneLength = 20, trControl = cmp_ctrl)
set.seed(100)
cmp_ctree <- train(cmp_train_x, cmp_train_y, method = "ctree",
preProc = "knnImpute",
tuneGrid = data.frame(mincriterion = seq(0.75, 0.99, length = 16)),
trControl = cmp_ctrl)
set.seed(100)
cmp_bag <- train(cmp_train_x, cmp_train_y, method = "treebag",
preProc = "knnImpute", nbagg = 50, trControl = cmp_ctrl)
set.seed(100)
cmp_rf <- train(cmp_train_x, cmp_train_y, method = "rf",
preProc = "knnImpute",
tuneGrid = data.frame(mtry = c(2, 15, 29, 42, 56)),
ntree = 1000, importance = TRUE, trControl = cmp_ctrl)
set.seed(100)
cmp_gbm <- train(cmp_train_x, cmp_train_y, method = "gbm",
preProc = "knnImpute",
tuneGrid = expand.grid(interaction.depth = seq(1, 7, by = 2),
n.trees = seq(100, 1000, by = 50),
shrinkage = c(0.01, 0.1),
n.minobsinnode = 10),
trControl = cmp_ctrl, verbose = FALSE)
set.seed(100)
cmp_cubist <- train(cmp_train_x, cmp_train_y, method = "cubist",
preProc = "knnImpute",
tuneGrid = expand.grid(committees = c(1, 5, 10, 25, 50, 100),
neighbors = c(0, 1, 5, 9)),
trControl = cmp_ctrl)
list(CART = cmp_cart, CondTree = cmp_ctree, Bagged = cmp_bag,
RF = cmp_rf, GBM = cmp_gbm, Cubist = cmp_cubist)
})
## Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,
## : There were missing values in resampled performance measures.
# M5 is cached on its own: the Weka model lives in Java, and .jcache() stores a copy
# inside the R object so the model still predicts after it is saved and reloaded
cmp_m5 <- fit_once("cmp_m5", {
set.seed(100)
m5 <- train(cmp_train_x, cmp_train_y, method = "M5",
preProc = "knnImpute", trControl = cmp_ctrl,
control = Weka_control(M = 10))
rJava::.jcache(m5$finalModel$classifier)
m5
})
cmp_trees <- c(cmp_trees[1:2], M5 = list(cmp_m5), cmp_trees[3:6])
lapply(cmp_trees, function(m) m$bestTune)
## $CART
## cp
## 18 0.3323156
##
## $CondTree
## mincriterion
## 12 0.926
##
## $M5
## pruned smoothed rules
## 7 No No Yes
##
## $Bagged
## parameter
## 1 none
##
## $RF
## mtry
## 2 15
##
## $GBM
## n.trees interaction.depth shrinkage n.minobsinnode
## 76 1000 7 0.01 10
##
## $Cubist
## committees neighbors
## 22 100 1
The tuning results behind the choices described below:
# resampled RMSE across the CART and conditional inference tree grids
round(cmp_trees$CART$results[, c("cp", "RMSE", "Rsquared")], 3)
## cp RMSE Rsquared
## 1 0.000 1.440 0.426
## 2 0.020 1.431 0.423
## 3 0.039 1.478 0.391
## 4 0.059 1.450 0.388
## 5 0.078 1.438 0.381
## 6 0.098 1.446 0.377
## 7 0.117 1.424 0.393
## 8 0.137 1.423 0.394
## 9 0.156 1.423 0.394
## 10 0.176 1.423 0.394
## 11 0.195 1.423 0.394
## 12 0.215 1.423 0.394
## 13 0.235 1.423 0.394
## 14 0.254 1.423 0.394
## 15 0.274 1.423 0.394
## 16 0.293 1.423 0.394
## 17 0.313 1.423 0.394
## 18 0.332 1.423 0.394
## 19 0.352 1.493 0.364
## 20 0.371 1.644 0.274
round(cmp_trees$CondTree$results[, c("mincriterion", "RMSE", "Rsquared")], 3)
## mincriterion RMSE Rsquared
## 1 0.750 1.454 0.393
## 2 0.766 1.455 0.393
## 3 0.782 1.454 0.393
## 4 0.798 1.457 0.391
## 5 0.814 1.465 0.384
## 6 0.830 1.456 0.387
## 7 0.846 1.455 0.389
## 8 0.862 1.450 0.390
## 9 0.878 1.446 0.393
## 10 0.894 1.445 0.392
## 11 0.910 1.441 0.395
## 12 0.926 1.428 0.403
## 13 0.942 1.434 0.399
## 14 0.958 1.442 0.392
## 15 0.974 1.441 0.386
## 16 0.990 1.441 0.384
# boosted trees: resampled RMSE at 500 and at 1,000 trees, by depth and shrinkage
gbm_res <- cmp_trees$GBM$results
round(xtabs(RMSE ~ interaction.depth + shrinkage, gbm_res[gbm_res$n.trees == 500, ]), 3)
## shrinkage
## interaction.depth 0.01 0.1
## 1 1.182 1.140
## 3 1.121 1.086
## 5 1.103 1.080
## 7 1.104 1.100
round(xtabs(RMSE ~ interaction.depth + shrinkage, gbm_res[gbm_res$n.trees == 1000, ]), 3)
## shrinkage
## interaction.depth 0.01 0.1
## 1 1.142 1.137
## 3 1.082 1.083
## 5 1.070 1.075
## 7 1.067 1.096
# random forest: resampled RMSE for each mtry value
round(cmp_trees$RF$results[, c("mtry", "RMSE", "Rsquared")], 3)
## mtry RMSE Rsquared
## 1 2 1.210 0.628
## 2 15 1.108 0.657
## 3 29 1.110 0.649
## 4 42 1.118 0.639
## 5 56 1.131 0.628
# Cubist with one neighbor, by number of committees
cubist_res <- cmp_trees$Cubist$results
round(cubist_res[cubist_res$neighbors == 1, c("committees", "RMSE", "Rsquared")], 4)
## committees RMSE Rsquared
## 2 1 1.2318 0.6093
## 6 5 1.0276 0.6813
## 10 10 0.9955 0.6937
## 14 25 0.9790 0.6938
## 18 50 0.9698 0.6996
## 22 100 0.9682 0.7020
Why these grids, and where the best values sit.
tuneLength = 20: caret spreads
20 evenly spaced complexity-parameter values over the range of the full
tree’s cost-complexity sequence. The table above shows the resampled
RMSE is almost flat (about 1.42 to 1.48) from cp = 0 up to about 0.33,
and the lowest value picked a tree with a single split. At the largest
cp values some resamples prune the tree back to the root, which predicts
the same yield for every run; that is why train() warns
about missing values in the resampled performance (R-squared cannot be
computed for a constant prediction).train(method = "M5") call, with
M = 10 (the minimum node size the book uses). The best is
the unpruned, unsmoothed rule version.mtry
spread from 2 to P = 56 as the book suggests, with 1,000 trees. The
best, 15, is inside the grid and close to the default P/3 (about 19);
every value from 15 to 56 gives nearly the same RMSE (table above).cmp_resampled <- t(sapply(cmp_trees, function(m) {
best <- merge(m$bestTune, m$results)
c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))
cmp_test_perf <- t(sapply(cmp_trees, function(m)
postResample(pred = predict(m, cmp_test_x), obs = cmp_test_y)))
cmp_perf <- cbind(cmp_resampled, cmp_test_perf)
colnames(cmp_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(cmp_perf[order(cmp_perf[, "CV_RMSE"]), ], 3)
## CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## Cubist 0.968 0.702 0.698 0.882 0.834 0.597
## GBM 1.067 0.657 0.840 1.183 0.725 0.888
## RF 1.108 0.657 0.895 1.263 0.723 0.912
## Bagged 1.184 0.589 0.935 1.262 0.704 0.937
## CART 1.423 0.394 1.145 1.563 0.504 1.191
## CondTree 1.428 0.403 1.150 1.417 0.616 1.113
## M5 1.449 0.429 1.117 1.219 0.681 0.915
cmp_resamp <- resamples(cmp_trees)
bwplot(cmp_resamp, metric = "RMSE")
# paired differences on the shared folds, for the best model against the others
cmp_best_name <- rownames(cmp_resampled)[which.min(cmp_resampled[, "RMSE"])]
cmp_best_name
## [1] "Cubist"
cmp_diff <- summary(diff(cmp_resamp), metric = "RMSE")
cmp_diff
##
## Call:
## summary.diff.resamples(object = diff(cmp_resamp), metric = "RMSE")
##
## p-value adjustment: bonferroni
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
##
## MAE
## CART CondTree M5 Bagged RF GBM Cubist
## CART -0.004662 0.028599 0.209964 0.249789 0.304890 0.447043
## CondTree 1.000000 0.033260 0.214625 0.254450 0.309551 0.451705
## M5 1.000000 1.000000 0.181365 0.221190 0.276291 0.418445
## Bagged 1.915e-09 2.138e-12 9.457e-05 0.039825 0.094926 0.237079
## RF 8.896e-11 5.306e-13 5.577e-06 0.001522 0.055101 0.197255
## GBM 6.739e-11 2.470e-13 1.704e-07 6.679e-06 0.002876 0.142154
## Cubist 9.555e-15 7.188e-16 1.055e-11 6.291e-10 4.268e-09 1.773e-05
##
## RMSE
## CART CondTree M5 Bagged RF GBM Cubist
## CART -0.005494 -0.026239 0.239086 0.314649 0.355995 0.454589
## CondTree 1.000000 -0.020745 0.244580 0.320142 0.361489 0.460083
## M5 1.000000 1.000000 0.265325 0.340888 0.382234 0.480828
## Bagged 1.030e-09 1.049e-13 3.704e-06 0.075562 0.116909 0.215503
## RF 4.132e-11 9.177e-14 4.759e-08 6.611e-07 0.041346 0.139940
## GBM 2.638e-10 9.947e-14 1.988e-08 3.273e-06 0.257957 0.098594
## Cubist 3.054e-10 2.288e-11 8.881e-10 6.071e-05 0.009273 0.142992
##
## Rsquared
## CART CondTree M5 Bagged RF GBM
## CART -0.0085960 -0.0344124 -0.1944159 -0.2629772 -0.2631454
## CondTree 1.0000000 -0.0258165 -0.1858200 -0.2543812 -0.2545495
## M5 1.0000000 1.0000000 -0.1600035 -0.2285647 -0.2287330
## Bagged 2.919e-09 1.479e-12 5.830e-06 -0.0685612 -0.0687295
## RF 3.342e-11 6.207e-14 4.033e-09 1.848e-07 -0.0001683
## GBM 6.777e-11 2.269e-14 2.738e-08 1.199e-05 1.0000000
## Cubist 2.085e-11 2.952e-12 1.089e-09 0.0003733 0.7187865 0.6186397
## Cubist
## CART -0.3077701
## CondTree -0.2991742
## M5 -0.2733577
## Bagged -0.1133542
## RF -0.0447930
## GBM -0.0446247
## Cubist
plot(cmp_trees$GBM)
plot(cmp_trees$Cubist)
plot(cmp_trees$RF)
The test set has only 32 runs, so the largest test errors of each model show whether a few runs drive the test numbers (row numbers are the run’s position in the original data):
sapply(cmp_trees, function(m) {
err <- predict(m, cmp_test_x) - cmp_test_y
names(err) <- rownames(cmp_test_x)
top <- head(sort(abs(err), decreasing = TRUE), 3)
paste0(names(top), " (", round(top, 2), ")")
})
## CART CondTree M5 Bagged RF GBM
## [1,] "49 (4.92)" "49 (4.64)" "49 (3.79)" "49 (3.69)" "49 (3.8)" "49 (3.75)"
## [2,] "28 (3.11)" "28 (2.8)" "44 (2.5)" "44 (3.09)" "44 (3.39)" "44 (2.37)"
## [3,] "44 (2.74)" "44 (2.46)" "132 (2.21)" "6 (2.11)" "6 (2.25)" "121 (2.23)"
## Cubist
## [1,] "49 (2.43)"
## [2,] "121 (2.43)"
## [3,] "44 (2.1)"
# test RMSE with all 32 runs and without run 49, the run every model predicts worst
run49 <- rownames(cmp_test_x) == "49"
round(t(sapply(cmp_trees, function(m) {
pred <- predict(m, cmp_test_x)
c(all_32_runs = RMSE(pred, cmp_test_y),
without_run_49 = RMSE(pred[!run49], cmp_test_y[!run49]))
})), 3)
## all_32_runs without_run_49
## CART 1.563 1.320
## CondTree 1.417 1.174
## M5 1.219 1.034
## Bagged 1.262 1.097
## RF 1.263 1.086
## GBM 1.183 0.995
## Cubist 0.882 0.782
Cubist gives the best resampling and test set performance among the models tried: resampled RMSE 0.968 (R-squared 0.70) and test RMSE 0.882 (R-squared 0.83), the lowest of the seven on both. The rankings below it differ between the two measures. By resampled RMSE, boosted trees are second (1.067) and the random forest third (1.108); on the test set boosted trees are second (1.183), then M5 (1.219), bagged trees (1.262) and the random forest (1.263). The paired comparison on the shared folds shows Cubist’s resampled RMSE is lower than the random forest’s (adjusted p-value about 0.009) and everything below it, but it did not detect a difference between Cubist and boosted trees (a lead of about 0.10, adjusted p-value about 0.14).
The ensembles split clearly from the single-tree and rule models. CART, the conditional inference tree and M5 (whose selected version is a rule model) all have resampled RMSE around 1.42 to 1.45 and R-squared around 0.4, and the paired comparison finds no differences among them. That is the chapter’s main lesson repeated on new data: a single tree is easy to read but predicts poorly, and averaging many trees (bagging and random forests) or adding them in sequence (boosting, and Cubist’s committees) improves on it a lot. Cubist’s linear models inside the rules and its neighbor correction can follow gradual changes in yield that a tree’s steps cannot, which is a plausible reason it does best, although this comparison does not show which part of Cubist made the difference.
Run 49 is the worst-predicted test run for every model. Without it every model’s test RMSE drops, and Cubist still has the lowest (0.782, then boosted trees at 0.995), so Cubist’s test lead does not depend on that one run; the main evaluation still uses all 32 runs. Cubist’s test RMSE is lower than its resampled RMSE. With only 32 test runs that gap could simply reflect which runs ended up in the test set, so the resampled estimate (0.968) is the figure to rely on, keeping in mind that it was also used to choose the tuning values.
Which predictors are most important in the optimal tree-based regression model? Do either the biological or process variables dominate the list? How do the top 10 important predictors compare to the top 10 predictors from the optimal linear and nonlinear models?
cmp_best <- cmp_trees[[cmp_best_name]]
cmp_tree_imp <- varImp(cmp_best)
plot(cmp_tree_imp, top = 15, main = paste("Top 15 predictors,", cmp_best_name))
tree_top10 <- rownames(cmp_tree_imp$importance)[order(-cmp_tree_imp$importance$Overall)][1:10]
tree_top10
## [1] "ManufacturingProcess32" "ManufacturingProcess17" "ManufacturingProcess09"
## [4] "ManufacturingProcess33" "ManufacturingProcess01" "ManufacturingProcess37"
## [7] "ManufacturingProcess45" "ManufacturingProcess15" "ManufacturingProcess21"
## [10] "BiologicalMaterial02"
table(ifelse(grepl("^Biological", tree_top10), "Biological", "Process"))
##
## Biological Process
## 1 9
For Cubist, caret’s varImp() averages two of Cubist’s
usage statistics, use in the rule conditions and use in the rules’
linear models, across all 100 committees (scaled 0 to 100 in the plot).
That is the kind of usage count the book suggests; it shows what the
model leans on, unlike the loess filter caret used for the SVM in
Exercise 7.5, but it counts uses rather than error reduction, so a
predictor that appears in many linear models with small coefficients can
rank high. The unscaled scores show several ties:
# Cubist usage scores before rescaling; equal values are ties
cubist_usage <- varImp(cmp_best, scale = FALSE)$importance
head(cubist_usage[order(-cubist_usage$Overall), , drop = FALSE], 12)
## Overall
## ManufacturingProcess32 49.5
## ManufacturingProcess17 24.0
## ManufacturingProcess09 24.0
## ManufacturingProcess33 23.5
## ManufacturingProcess01 23.0
## ManufacturingProcess37 23.0
## ManufacturingProcess45 22.5
## ManufacturingProcess15 22.0
## ManufacturingProcess21 22.0
## BiologicalMaterial02 2.5
## BiologicalMaterial10 1.5
## BiologicalMaterial06 1.5
The optimal linear model (the elastic net from Exercise 6.3) and the optimal nonlinear model (the radial-kernel SVM from Exercise 7.5) are refit here with the same code, folds and seed as in those exercises, so their top ten lists can be compared directly.
cmp_others <- fit_once("cmp_others", {
set.seed(100)
cmp_enet <- train(cmp_train_x, cmp_train_y, method = "enet",
preProc = "knnImpute",
tuneGrid = expand.grid(lambda = c(0.01, 0.1, 1),
fraction = seq(0.05, 1, length = 20)),
trControl = cmp_ctrl)
set.seed(100)
cmp_svm <- train(cmp_train_x, cmp_train_y, method = "svmRadial",
preProc = "knnImpute", tuneLength = 14, trControl = cmp_ctrl)
list(Enet = cmp_enet, SVM = cmp_svm)
})
# linear: absolute elastic net coefficients at the chosen fraction (standardized predictors)
lin_coef <- predict(cmp_others$Enet$finalModel, s = cmp_others$Enet$bestTune$fraction,
mode = "fraction", type = "coefficients")$coefficients
lin_top10 <- names(sort(abs(lin_coef[lin_coef != 0]), decreasing = TRUE))[1:10]
# nonlinear: caret's importance for the SVM (a loess R-squared filter, as in Exercise 7.5)
svm_imp <- varImp(cmp_others$SVM)$importance
nl_top10 <- rownames(svm_imp)[order(-svm_imp$Overall)][1:10]
data.frame(Rank = 1:10, Tree = tree_top10, Nonlinear_SVM = nl_top10, Linear_Enet = lin_top10)
## Rank Tree Nonlinear_SVM Linear_Enet
## 1 1 ManufacturingProcess32 ManufacturingProcess32 ManufacturingProcess32
## 2 2 ManufacturingProcess17 BiologicalMaterial06 ManufacturingProcess09
## 3 3 ManufacturingProcess09 ManufacturingProcess36 ManufacturingProcess36
## 4 4 ManufacturingProcess33 BiologicalMaterial03 ManufacturingProcess17
## 5 5 ManufacturingProcess01 ManufacturingProcess09 ManufacturingProcess34
## 6 6 ManufacturingProcess37 ManufacturingProcess13 BiologicalMaterial06
## 7 7 ManufacturingProcess45 ManufacturingProcess31 ManufacturingProcess11
## 8 8 ManufacturingProcess15 BiologicalMaterial02 ManufacturingProcess06
## 9 9 ManufacturingProcess21 ManufacturingProcess11 ManufacturingProcess13
## 10 10 BiologicalMaterial02 BiologicalMaterial12 BiologicalMaterial03
list(in_all_three = Reduce(intersect, list(tree_top10, nl_top10, lin_top10)),
tree_only = setdiff(tree_top10, union(nl_top10, lin_top10)))
## $in_all_three
## [1] "ManufacturingProcess32" "ManufacturingProcess09"
##
## $tree_only
## [1] "ManufacturingProcess33" "ManufacturingProcess01" "ManufacturingProcess37"
## [4] "ManufacturingProcess45" "ManufacturingProcess15" "ManufacturingProcess21"
# all three families on the same folds
cmp_all <- resamples(c(cmp_trees[cmp_best_name], cmp_others))
summary(cmp_all, metric = "RMSE")
##
## Call:
## summary.resamples(object = cmp_all, metric = "RMSE")
##
## Models: Cubist, Enet, SVM
## Number of resamples: 50
##
## RMSE
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## Cubist 0.5232784 0.7466358 0.9763964 0.9682315 1.109176 1.734572 0
## Enet 0.8009208 1.0016908 1.1374785 1.1961284 1.302849 2.904083 0
## SVM 0.6626650 1.0035396 1.1342706 1.1304262 1.294400 1.616642 0
summary(diff(cmp_all), metric = "RMSE")
##
## Call:
## summary.diff.resamples(object = diff(cmp_all), metric = "RMSE")
##
## p-value adjustment: bonferroni
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
##
## MAE
## Cubist Enet SVM
## Cubist -0.27121 -0.23178
## Enet 6.197e-14 0.03943
## SVM 3.560e-10 0.4599
##
## RMSE
## Cubist Enet SVM
## Cubist -0.2279 -0.1622
## Enet 4.143e-06 0.0657
## SVM 9.012e-05 0.3965
##
## Rsquared
## Cubist Enet SVM
## Cubist 0.12612 0.09932
## Enet 4.610e-07 -0.02680
## SVM 8.074e-05 0.8051
The three biological predictors that the lists treat differently are closely related in the training data:
bio_three <- c("BiologicalMaterial02", "BiologicalMaterial03", "BiologicalMaterial06")
round(cor(cmp_train_x[, bio_three], use = "pairwise.complete.obs"), 3)
## BiologicalMaterial02 BiologicalMaterial03
## BiologicalMaterial02 1.000 0.849
## BiologicalMaterial03 0.849 1.000
## BiologicalMaterial06 0.954 0.862
## BiologicalMaterial06
## BiologicalMaterial02 0.954
## BiologicalMaterial03 0.862
## BiologicalMaterial06 1.000
Process predictors dominate. Nine of Cubist’s top
ten are manufacturing process predictors; the only biological one is
BiologicalMaterial02, in tenth place.
ManufacturingProcess32 is first, then
ManufacturingProcess17 and
ManufacturingProcess09, which are tied on the unscaled
usage score (as are 01 and 37, and
15 and 21), so the order within each tied pair
means nothing. The tree model is more process-heavy than either earlier
model: the SVM’s list (from a one-predictor-at-a-time filter) had four
biological predictors, and the elastic net’s had two.
ManufacturingProcess32 is first in all three lists, and
ManufacturingProcess09 is in all three as well, so those
two are the most consistent findings across the three families of
models. ManufacturingProcess17 is shared with the elastic
net (fourth there) and BiologicalMaterial02 with the SVM.
The other six, ManufacturingProcess33, 01,
37, 45, 15 and 21,
appear only in the Cubist list. Some of that is likely the measure:
usage counts reward predictors that Cubist puts into many of its rules’
linear models, and there are many process predictors, so different rules
can use different ones. The biological predictors the other two models
ranked high, BiologicalMaterial06 and 03, are
strongly correlated with each other and with 02 in the
training data (0.85 to 0.95 in the table above), so Cubist may be using
one of them where the others would do; the lists alone cannot show
that.
The comparison of performance is the more striking result. On the same 50 resamples Cubist’s RMSE is lower than the SVM’s by about 0.16 and the elastic net’s by about 0.23, with adjusted p-values well below 0.001. In Exercise 7.5 the best nonlinear model could not be separated from the elastic net; here a tree-based model is the first one in these three homeworks with a clear advantage over the linear model in this resampling comparison. For a plant engineer that makes Cubist the leading candidate for predicting yield, if an error of about one yield point is accurate enough for the decisions involved, with the caution that its 100 committees of rules are much harder to explain than the elastic net’s coefficients.
Plot the optimal single tree with the distribution of yield in the terminal nodes. Does this view of the data provide additional knowledge about the biological or process predictors and their relationship with yield?
The single tree is the tuned CART model from part (a), which has the
lowest resampled RMSE of the three single-tree and rule candidates (the
conditional inference tree and M5 have lower test RMSE, but the choice
is made on resampling). The tree in train() was fitted on
standardized predictors, so its thresholds are in standard deviations.
To make the plot readable in process units, the training data are
imputed with the same KNN imputation and then put back on their original
scale, and the tree is refit with the selected complexity parameter.
Because a tree split depends only on the order of the values, this gives
the same tree with thresholds in the original units. The plot uses the
partykit conversion shown in the book, which draws the
distribution of yield in each terminal node as a box plot.
cmp_impute <- preProcess(cmp_train_x, method = "knnImpute")
cmp_train_std <- predict(cmp_impute, cmp_train_x)
# undo the centering and scaling that knnImpute applies, keeping the imputed values
cmp_train_imputed <- sweep(sweep(cmp_train_std, 2, cmp_impute$std[names(cmp_train_std)], "*"),
2, cmp_impute$mean[names(cmp_train_std)], "+")
cmp_train_imputed$Yield <- cmp_train_y
cmp_single_tree <- rpart(Yield ~ ., data = cmp_train_imputed,
control = rpart.control(cp = cmp_trees$CART$bestTune$cp))
cmp_single_tree
## n= 144
##
## node), split, n, deviance, yval
## * denotes terminal node
##
## 1) root 144 453.4699 40.15799
## 2) ManufacturingProcess32< 159.5 83 152.3024 39.23084 *
## 3) ManufacturingProcess32>=159.5 61 132.7435 41.41951 *
# same splitting variables as the tuned model?
identical(as.character(cmp_single_tree$frame$var),
as.character(cmp_trees$CART$finalModel$frame$var))
## [1] TRUE
plot(as.party(cmp_single_tree), gp = gpar(fontsize = 8))
# yield summary in each terminal node, with the path that leads to it
cmp_leaf <- cmp_single_tree$where
leaf_summary <- aggregate(cmp_train_y, list(node = cmp_leaf),
function(v) c(n = length(v), mean = mean(v), min = min(v),
q25 = unname(quantile(v, 0.25)),
q75 = unname(quantile(v, 0.75)), max = max(v)))
leaf_summary <- data.frame(node = leaf_summary$node, round(leaf_summary$x, 2))
leaf_summary[order(leaf_summary$mean), ]
## node n mean min q25 q75 max
## 1 2 83 39.23 35.25 38.36 40.05 43.12
## 2 3 61 41.42 36.83 40.28 42.45 44.35
The optimal single tree is a single split. Runs with
ManufacturingProcess32 below 159.5 have a mean yield of
about 39.2, and runs at or above 159.5 have a mean of about 41.4. The
yield ranges of the two leaves overlap a lot (about 35 to 43 on the low
side and 37 to 44 on the high side), although the middle halves do not
(quartiles 38.4 to 40.1 against 40.3 to 42.5 in the table), so the split
shifts the typical yield by about two points without separating good
runs from bad ones. It still adds something the importance lists do not:
a threshold. It is the first split this CART fit chose, which agrees
with every model in this and the last two exercises ranking
ManufacturingProcess32 first. A single split also shows why
CART predicts so poorly: with only two possible predictions, it cannot
follow the gradual changes the ensembles pick up.
Because the CART profile is flat, a larger tree has almost the same resampled RMSE as the stump. To see what more splits would add, the tree below is the one with the lowest resampled RMSE among the settings that give more than two leaves on the training data:
# among the cp values that give more than two leaves when the tree is refit,
# the one with the lowest resampled RMSE
cart_res <- cmp_trees$CART$results
leaf_count <- function(cp) {
fit <- rpart(Yield ~ ., data = cmp_train_imputed, control = rpart.control(cp = cp))
sum(fit$frame$var == "<leaf>")
}
cart_leaves <- sapply(cart_res$cp, leaf_count)
cart_bigger <- cart_res[cart_leaves > 2, ]
cp_deeper <- cart_bigger$cp[which.min(cart_bigger$RMSE)]
# compare its resampled RMSE with the selected stump's
cart_best <- cart_res[cart_res$cp == cmp_trees$CART$bestTune$cp, ]
c(cp = cp_deeper, RMSE = min(cart_bigger$RMSE), best_RMSE = cart_best$RMSE)
## cp RMSE best_RMSE
## 0.01954798 1.43089107 1.42282055
cmp_deeper_tree <- rpart(Yield ~ ., data = cmp_train_imputed,
control = rpart.control(cp = cp_deeper))
cmp_deeper_tree # path, size and mean yield of every node
## n= 144
##
## node), split, n, deviance, yval
## * denotes terminal node
##
## 1) root 144 453.469900 40.15799
## 2) ManufacturingProcess32< 159.5 83 152.302400 39.23084
## 4) BiologicalMaterial12< 19.735 31 36.094620 38.32839
## 8) BiologicalMaterial05>=19.6 8 5.544200 37.21500 *
## 9) BiologicalMaterial05< 19.6 23 17.183970 38.71565 *
## 5) BiologicalMaterial12>=19.735 52 75.909330 39.76885
## 10) ManufacturingProcess25>=4824.4 39 27.750140 39.29744 *
## 11) ManufacturingProcess25< 4824.4 13 13.491680 41.18308 *
## 3) ManufacturingProcess32>=159.5 61 132.743500 41.41951
## 6) BiologicalMaterial03< 72 48 92.191830 41.08688
## 12) ManufacturingProcess06< 206.3 14 20.613490 40.11071 *
## 13) ManufacturingProcess06>=206.3 34 52.744750 41.48882
## 26) ManufacturingProcess14< 4902 27 34.123900 41.17037
## 52) BiologicalMaterial09>=12.67 15 18.705570 40.56133 *
## 53) BiologicalMaterial09< 12.67 12 2.899567 41.93167 *
## 27) ManufacturingProcess14>=4902 7 5.321343 42.71714 *
## 7) BiologicalMaterial03>=72 13 15.631030 42.64769 *
plot(as.party(cmp_deeper_tree), gp = gpar(fontsize = 8))
The larger tree has nine leaves and a resampled RMSE of 1.431 against
1.423 for the stump. The estimates are close, but that does not
establish that the two predict equally well; the stump has the lower
mean error and is simpler, so it stays the selected model, and the
larger tree is shown only to explore the training data further. Its
printed output gives the path, size and mean yield of every node. It
keeps the same first split on ManufacturingProcess32, and
most of the next splits are on biological predictors:
BiologicalMaterial12 and BiologicalMaterial05
on the low side, and BiologicalMaterial03 and
BiologicalMaterial09 on the high side. That is information
the importance lists do not give. Within each side of the main split,
the tree divides the runs further by biological measurements: on the
high side, runs with BiologicalMaterial03 at 72 or more
average about 42.6, and on the low side, runs with low
BiologicalMaterial12 and high
BiologicalMaterial05 average about 37.2, the lowest leaf.
Three process predictors also appear further down
(ManufacturingProcess25,
ManufacturingProcess06 and
ManufacturingProcess14); for example, among the
low-ManufacturingProcess32 runs with higher
BiologicalMaterial12, the runs with lower
ManufacturingProcess25 average about 1.9 points more yield
(41.2 against 39.3).
These leaves hold only 7 to 39 runs each, and the book warns that a
single tree is unstable (a slightly different sample can give different
splits), so the deeper splits are hypotheses to check, not settings to
adopt. Higher ManufacturingProcess32 values go with higher
yield in these training runs, but the data are observational, so that
does not show that raising the setting would raise yield. The biological
and secondary process splits point to relationships worth investigating
with the process engineers and testing in a designed experiment before
any operating or screening rule is changed.