All four exercises are from Chapter 8 of Kuhn and Johnson, Applied Predictive Modeling (regression trees and rule-based models). Exercises 8.1 and 8.3 are about how tree ensembles measure predictor importance, 8.2 is about a known bias in how single trees choose their splits, and 8.7 returns to the chemical manufacturing data from the last two homeworks.

Exercise 8.1

Recreate the simulated data from Exercise 7.2:

library(mlbench)
set.seed(200)
simulated <- mlbench.friedman1(200, sd = 1)
simulated <- cbind(simulated$x, simulated$y)
simulated <- as.data.frame(simulated)
colnames(simulated)[ncol(simulated)] <- "y"

The response is \(y = 10 \sin(\pi x_1 x_2) + 20 (x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + N(0, 1)\), so V1 to V5 are informative and V6 to V10 are pure noise. That known answer is what makes this a fair test of the importance measures.

(a) Random forest on all of the predictors

Fit a random forest model to all of the predictors, then estimate the variable importance scores:

library(randomForest)
library(caret)
model1 <- randomForest(y ~ ., data = simulated,
                       importance = TRUE,
                       ntree = 1000)
rfImp1 <- varImp(model1, scale = FALSE)

Did the random forest model significantly use the uninformative predictors (V6 – V10)?

# the book's code; the seed is added so the forest and its importance
# values can be reproduced
set.seed(624)
model1 <- randomForest(y ~ ., data = simulated,
                       importance = TRUE,
                       ntree = 1000)
rfImp1 <- varImp(model1, scale = FALSE)
rfImp1[order(-rfImp1$Overall), , drop = FALSE]
##          Overall
## V1   8.950480717
## V4   7.848396144
## V2   6.560901913
## V5   2.143163628
## V3   0.740942146
## V6   0.091400119
## V7   0.076258170
## V10  0.009408601
## V9  -0.032033548
## V8  -0.153191514

varImp() on a randomForest object returns the permutation importance described in the book: for each tree, the out-of-bag values of one predictor are shuffled and the increase in the out-of-bag mean squared error is recorded, then averaged over the forest. A predictor the forest does not rely on gives an increase near zero, and can even give a small negative value by chance.

A single forest is one random draw, so the size of the noise scores is easier to judge when the forest is refit with a few different seeds:

# five more forests on the same data, to see how much the scores move from seed to seed
rf_seed_imp <- fit_once("rf_seed_imp", sapply(1:5, function(s) {
  set.seed(s)
  fit <- randomForest(y ~ ., data = simulated, importance = TRUE, ntree = 1000)
  varImp(fit, scale = FALSE)$Overall
}))
rownames(rf_seed_imp) <- rownames(rfImp1)
round(cbind(min = apply(rf_seed_imp, 1, min), max = apply(rf_seed_imp, 1, max)), 2)
##       min   max
## V1   8.53  8.98
## V2   6.54  6.78
## V3   0.61  0.83
## V4   7.52  7.83
## V5   2.05  2.28
## V6   0.08  0.17
## V7  -0.02  0.07
## V8  -0.17 -0.10
## V9  -0.02  0.03
## V10 -0.17  0.10

No. The five informative predictors have clearly positive scores (V1, V4 and V2 highest, then V5 and V3), while V6 to V10 sit around zero, between about -0.17 and 0.17 across the six forests. The weakest informative predictor, V3, never falls below 0.61 in any of the forests, so there is a clear gap between the known signal and the known noise. The small negative values for V8 mean that shuffling it made the out-of-bag error slightly smaller. That is consistent with V8 being noise here, which the simulation formula tells us, although a negative estimate on its own would not prove that a predictor carries no information. The forest does split on the noise predictors now and then (with 1,000 deep trees it has to split on something), but those splits add essentially nothing to prediction. The repeated seeds show how much the scores move from one forest to the next on this one data set; they are not a formal significance test, so the fair summary is that V6 to V10 get no substantial importance compared with V1 to V5.

V3 is low compared with V1, V2 and V4 because its effect, \(20 (x_3 - 0.5)^2\), is a U shape centered in the middle of its range and contributes at most 5 units to \(y\), against 10 for V4 and up to 10 for the sine term.

(b) Adding a predictor highly correlated with V1

Now add an additional predictor that is highly correlated with one of the informative predictors. For example:

simulated$duplicate1 <- simulated$V1 + rnorm(200) * .1
cor(simulated$duplicate1, simulated$V1)

Fit another random forest model to these data. Did the importance score for V1 change? What happens when you add another predictor that is also highly correlated with V1?

sim_base <- simulated   # kept for parts (c) and (d)

set.seed(624)   # the seed fixes the noise added to V1
simulated$duplicate1 <- simulated$V1 + rnorm(200) * .1
cor(simulated$duplicate1, simulated$V1)
## [1] 0.9429376
sim_dup1 <- simulated

simulated$duplicate2 <- simulated$V1 + rnorm(200) * .1
cor(simulated$duplicate2, simulated$V1)
## [1] 0.9380982
sim_dup2 <- simulated
set.seed(624)
model2 <- randomForest(y ~ ., data = sim_dup1, importance = TRUE, ntree = 1000)
rfImp2 <- varImp(model2, scale = FALSE)

set.seed(624)
model3 <- randomForest(y ~ ., data = sim_dup2, importance = TRUE, ntree = 1000)
rfImp3 <- varImp(model3, scale = FALSE)

# one row per predictor, one column per data set; NA where the predictor is not in the data
importance_table <- function(base, dup1, dup2) {
  preds <- c(paste0("V", 1:10), "duplicate1", "duplicate2")
  pick <- function(imp) imp[preds, "Overall"]
  out <- data.frame(original = pick(base), plus_duplicate1 = pick(dup1),
                    plus_both_duplicates = pick(dup2), row.names = preds)
  round(out, 2)
}
rf_imp_table <- importance_table(rfImp1, rfImp2, rfImp3)
rf_imp_table
##            original plus_duplicate1 plus_both_duplicates
## V1             8.95            5.99                 6.06
## V2             6.56            6.11                 6.15
## V3             0.74            0.55                 0.50
## V4             7.85            7.04                 7.42
## V5             2.14            1.94                 2.04
## V6             0.09            0.20                 0.20
## V7             0.08            0.04                 0.00
## V8            -0.15           -0.06                -0.12
## V9            -0.03           -0.01                 0.03
## V10            0.01           -0.14                 0.00
## duplicate1       NA            3.59                 3.25
## duplicate2       NA              NA                 1.75
# randomForest's default mtry (a third of the predictors) in the three fits
c(original = model1$mtry, plus_duplicate1 = model2$mtry, plus_both_duplicates = model3$mtry)
##             original      plus_duplicate1 plus_both_duplicates 
##                    3                    3                    4

Yes, V1’s importance drops. With one copy (correlation about 0.94 with V1) V1 falls from about 9.0 to about 6.0, and duplicate1 gets about 3.6, while the other predictors move much less. This is the dilution the book describes: when two predictors carry nearly the same information, the forest can use either one at a split (each split only considers a random subset of the predictors), so each gets part of the credit, and V1 looks less important on its own than it did without the copy. Permutation scores do not divide up a fixed total, though, so the 3.6 is not literally the 3.0 that V1 lost.

Adding a second copy (correlation about 0.94 again) spreads the credit further: duplicate1 drops to about 3.3 and duplicate2 gets about 1.8, while V1 itself stays near 6. The split is not an even X/2 or X/3 as in the book’s idealized example, likely because the copies carry added noise (0.1 standard deviations), so V1 is the cleanest version of the signal. One more thing changes with the second copy: randomForest sets mtry to a third of the predictors, so it goes from 3 to 3 to 4 across the three fits (printed above), and the book notes that mtry itself affects importance. The sum of the scores for V1 and its copies rises from about 9.0 to 9.6 and 11.1, but a sum of individual scores is not a measured importance for the group. The practical point is that the ranking is sensitive to redundant predictors: in the second and third tables V1 ranks below V4 and V2, which would mislead a reader who did not know the copies were made from V1.

(c) Conditional inference random forest

Use the cforest function in the party package to fit a random forest model using conditional inference trees. The party package function varimp can calculate predictor importance. The conditional argument of that function toggles between the traditional importance measure and the modified version described in Strobl et al. (2007). Do these importances show the same pattern as the traditional random forest model?

cforest() is fitted to the same three data sets with the same number of trees as the random forests. cforest_unbiased() is the control setting the party package recommends for importance (subsampling without replacement); its default mtry is 5.

cf_imp <- fit_once("cf_imp", lapply(list(base = sim_base, dup1 = sim_dup1, dup2 = sim_dup2),
  function(d) {
    set.seed(624)
    cf <- party::cforest(y ~ ., data = d, controls = cforest_unbiased(ntree = 1000))
    # traditional (marginal) permutation importance and the conditional version
    list(traditional = varimp(cf),
         conditional = varimp(cf, conditional = TRUE))
  }))

as_imp <- function(v) data.frame(Overall = v, row.names = names(v))
cf_trad_table <- importance_table(as_imp(cf_imp$base$traditional),
                                  as_imp(cf_imp$dup1$traditional),
                                  as_imp(cf_imp$dup2$traditional))
cf_cond_table <- importance_table(as_imp(cf_imp$base$conditional),
                                  as_imp(cf_imp$dup1$conditional),
                                  as_imp(cf_imp$dup2$conditional))
cf_trad_table
##            original plus_duplicate1 plus_both_duplicates
## V1             8.70            6.46                 6.13
## V2             6.41            6.02                 5.84
## V3             0.02            0.02                 0.02
## V4             8.36            7.75                 7.34
## V5             1.80            1.79                 1.61
## V6            -0.02           -0.01                 0.01
## V7             0.08            0.06                 0.02
## V8            -0.03           -0.03                 0.00
## V9            -0.02           -0.01                -0.02
## V10           -0.01           -0.04                -0.05
## duplicate1       NA            2.96                 2.53
## duplicate2       NA              NA                 1.03
cf_cond_table
##            original plus_duplicate1 plus_both_duplicates
## V1             5.53            2.96                 2.50
## V2             5.27            4.81                 4.56
## V3             0.01            0.00                 0.03
## V4             6.61            6.06                 5.76
## V5             1.14            1.12                 0.91
## V6             0.00            0.00                 0.02
## V7            -0.01            0.01                 0.00
## V8            -0.01            0.00                 0.00
## V9            -0.01            0.01                 0.01
## V10            0.00           -0.03                -0.03
## duplicate1       NA            0.93                 0.77
## duplicate2       NA              NA                 0.19

The traditional cforest importance shows the same pattern as the random forest. V1 falls from about 8.7 to 6.5 when duplicate1 is added and to 6.1 with both copies, and the copies pick up about 3.0, then 2.5 and 1.0. The noise predictors V6 to V10 stay near zero again.

The conditional importance changes the picture. The copies get much less credit: duplicate1 falls to about 0.9 and duplicate2 to about 0.2, because the conditional version permutes each predictor within groups defined by the predictors it is correlated with, so a copy only gets credit for what it adds beyond V1. This is the redundancy correction the book describes. V1 itself, though, still drops, from about 5.5 to 3.0 and 2.5. The conditional measure removes most of the credit given to the copies but does not give it back to V1, which matches the book’s statement that the method reduces the effect of redundancy but does not adjust for dilution.

One difference from the random forest has nothing to do with the copies: cforest gives V3 an importance of almost exactly zero in every fit. Conditional inference trees choose splits with a test of association between the predictor and the response. V3’s effect is a symmetric U shape around 0.5, so its overall linear association with \(y\) is close to zero and the test rarely selects it, while CART’s exhaustive SSE search can still find splits that pick up part of the U shape. That is likely a cost of the unbiased splitting rule when an effect is not monotone.

(d) Boosted trees and Cubist

Repeat this process with different tree models, such as boosted trees and Cubist. Does the same pattern occur?

Both models are tuned by 10-fold cross-validation on each of the three data sets. The boosting grid follows the book’s (depth 1 to 7, 100 to 1,000 trees, shrinkage 0.01 or 0.1) and the Cubist grid covers 1 to 100 committees with and without neighbor adjustment.

sim_ctrl <- trainControl(method = "cv", number = 10)
sim_gbm_grid <- expand.grid(interaction.depth = seq(1, 7, by = 2),
                            n.trees = seq(100, 1000, by = 50),
                            shrinkage = c(0.01, 0.1),
                            n.minobsinnode = 10)
sim_cubist_grid <- expand.grid(committees = c(1, 10, 50, 100), neighbors = c(0, 5, 9))

sim_boost_cubist <- fit_once("sim_boost_cubist", lapply(
  list(base = sim_base, dup1 = sim_dup1, dup2 = sim_dup2), function(d) {
    x <- d[, names(d) != "y"]
    set.seed(624)
    gbm_fit <- train(x, d$y, method = "gbm", tuneGrid = sim_gbm_grid,
                     trControl = sim_ctrl, verbose = FALSE)
    set.seed(624)
    cubist_fit <- train(x, d$y, method = "cubist", tuneGrid = sim_cubist_grid,
                        trControl = sim_ctrl)
    list(gbm = gbm_fit, cubist = cubist_fit)
  }))

lapply(sim_boost_cubist, function(m) list(gbm = m$gbm$bestTune, cubist = m$cubist$bestTune))
## $base
## $base$gbm
##    n.trees interaction.depth shrinkage n.minobsinnode
## 38    1000                 3      0.01             10
## 
## $base$cubist
##    committees neighbors
## 12        100         9
## 
## 
## $dup1
## $dup1$gbm
##    n.trees interaction.depth shrinkage n.minobsinnode
## 38    1000                 3      0.01             10
## 
## $dup1$cubist
##    committees neighbors
## 12        100         9
## 
## 
## $dup2
## $dup2$gbm
##    n.trees interaction.depth shrinkage n.minobsinnode
## 38    1000                 3      0.01             10
## 
## $dup2$cubist
##   committees neighbors
## 9         50         9

For boosting, varImp() is the reduction in squared error from each predictor’s splits, summed over the trees. For Cubist, caret’s varImp() averages two of Cubist’s usage statistics: how much a predictor is used in rule conditions and how much it is used in the rules’ linear models (the book says Cubist has no established importance measure and suggests counts of this kind). The result is a usage score, not the percentage of rules that contain the predictor.

gbm_imp_table <- importance_table(varImp(sim_boost_cubist$base$gbm, scale = FALSE)$importance,
                                  varImp(sim_boost_cubist$dup1$gbm, scale = FALSE)$importance,
                                  varImp(sim_boost_cubist$dup2$gbm, scale = FALSE)$importance)
cubist_usage_of <- function(fit) varImp(fit, scale = FALSE)$importance
cubist_imp_table <- importance_table(cubist_usage_of(sim_boost_cubist$base$cubist),
                                     cubist_usage_of(sim_boost_cubist$dup1$cubist),
                                     cubist_usage_of(sim_boost_cubist$dup2$cubist))
gbm_imp_table
##            original plus_duplicate1 plus_both_duplicates
## V1         39802.70        30326.38             28720.72
## V2         34075.90        33882.15             33853.83
## V3         11873.87        11708.40             11670.94
## V4         44228.29        44736.52             44775.94
## V5         18079.63        17475.84             17630.25
## V6          1899.34         1901.75              1772.22
## V7          1758.72         1579.90              1581.57
## V8           890.91         1021.66              1034.96
## V9           932.02          915.83              1010.20
## V10          606.90          597.86               553.15
## duplicate1       NA        11377.79             10504.76
## duplicate2       NA              NA              2778.53
cubist_imp_table
##            original plus_duplicate1 plus_both_duplicates
## V1             71.5            64.0                 61.0
## V2             58.5            60.0                 64.5
## V3             47.0            42.0                 33.5
## V4             48.0            47.5                 50.5
## V5             33.0            27.5                 39.0
## V6              0.0            11.0                 13.5
## V7              0.0             0.0                  0.0
## V8              0.0             1.5                  3.0
## V9              0.0             0.0                  0.0
## V10             0.0             0.0                  0.0
## duplicate1       NA             5.0                  4.0
## duplicate2       NA              NA                 21.0

The models measure importance on different scales, so to compare the pattern across them the table below gives V1’s importance as a share of the total importance in each fit, and the combined share of V1 and its copies:

v1_share <- function(tab) {
  # descriptive shares: absent predictors and negative scores set to zero
  tab[is.na(tab) | tab < 0] <- 0
  shares <- sweep(tab, 2, colSums(tab), "/")
  rbind(V1 = shares["V1", ],
        V1_and_copies = colSums(shares[c("V1", "duplicate1", "duplicate2"), ]))
}
share_tables <- list("random forest" = rf_imp_table, "cforest, traditional" = cf_trad_table,
                     "cforest, conditional" = cf_cond_table, "boosted trees" = gbm_imp_table,
                     "Cubist" = cubist_imp_table)
v1_shares <- do.call(rbind, lapply(names(share_tables), function(nm)
  data.frame(model = nm, measure = c("V1", "V1 and copies"),
             round(v1_share(share_tables[[nm]]), 3), row.names = NULL)))
v1_shares
##                   model       measure original plus_duplicate1
## 1         random forest            V1    0.339           0.235
## 2         random forest V1 and copies    0.339           0.376
## 3  cforest, traditional            V1    0.343           0.258
## 4  cforest, traditional V1 and copies    0.343           0.376
## 5  cforest, conditional            V1    0.298           0.186
## 6  cforest, conditional V1 and copies    0.298           0.245
## 7         boosted trees            V1    0.258           0.195
## 8         boosted trees V1 and copies    0.258           0.268
## 9                Cubist            V1    0.277           0.248
## 10               Cubist V1 and copies    0.277           0.267
##    plus_both_duplicates
## 1                 0.221
## 2                 0.404
## 3                 0.250
## 4                 0.395
## 5                 0.169
## 6                 0.235
## 7                 0.184
## 8                 0.269
## 9                 0.210
## 10                0.297

Boosted trees. The same pattern occurs. The selected model (1,000 trees, depth 3, shrinkage 0.01) is the same on all three data sets. V1’s importance falls from about 39,800 to 30,300 with one copy and 28,700 with two, duplicate1 takes about 11,400 and then 10,500, and duplicate2 only about 2,800. V1’s share of the total falls from 0.26 to 0.20 and 0.18. Boosting gives the second copy even less than the random forest does. A likely reason is that each tree is fit to the residuals of the trees before it, so once V1 and duplicate1 have been used to fit that part of the signal, little is left for duplicate2.

Cubist. The pattern is weaker but in the same direction. V1’s usage score falls from 71.5 to 64 and 61, and its share from 0.28 to 0.25 and 0.21. duplicate1 has a usage score of only 4 to 5, while duplicate2 reaches 21 in the last model, so the usage is uneven between the copies and changes from one fit to the next. Usage scores count how often a predictor appears in rule conditions or linear models, not how much it reduces the error, so a copy that enters many linear models with a small coefficient can look important. Cubist also gives the noise predictor V6 a usage score of 11 to 14 on the data with copies, the only model here where a noise predictor gets a noticeable score, which is another reason to treat Cubist’s usage as a rough guide (the book says as much).

Across all five measures, adding a near-copy of an important predictor lowers that predictor’s own score and its share. The shares only describe how each measure redistributes its own scores; they do not turn permutation loss, squared-error reduction and usage counts into the same quantity. The conditional importance is the one that keeps the copies from collecting a large share (together about 0.07 of the total in the last fit), but V1’s share under it still falls from 0.30 to 0.19 and 0.17, so it cuts the credit given to redundant copies without restoring V1’s standing, as the book says. None of the measures recover V1’s original standing. In practice that means correlated predictors should be found and handled (removed or combined) before reading any tree ensemble’s importance list.

Exercise 8.2

Use a simulation to show tree bias with different granularities.

The book says single trees suffer from selection bias: a predictor with many distinct values offers many more candidate split points than a coarse one, so it has more chances to find a split that lowers the SSE by luck. The simulation builds four predictors that differ only in granularity:

All four are generated independently. In the first setting the response is pure noise, so no predictor is related to it, and an unbiased method that has to choose a predictor would pick each one about a quarter of the time. In the next two settings only the binary predictor is related to the response: the mean of \(y\) is 0.3 higher, then 0.5 higher, when x_binary is 1, in units of the noise standard deviation (which is 1). Any split on the other three predictors is a split on noise. Each setting is simulated 500 times, and each time a one-split CART tree (rpart) and a one-split conditional inference tree (ctree) record which predictor they choose for the first split.

make_granularity_data <- function(n = 200, effect = 0) {
  d <- data.frame(x_binary = sample(0:1, n, replace = TRUE),
                  x_10 = sample(1:10, n, replace = TRUE),
                  x_50 = sample(1:50, n, replace = TRUE),
                  x_cont = runif(n))
  d$y <- effect * d$x_binary + rnorm(n)   # only the binary predictor carries any signal
  d
}

# cp = 0 and maxdepth = 1 force CART to make its single best split,
# so the choice can be recorded
first_split_rpart <- function(d) {
  fit <- rpart(y ~ ., data = d, control = rpart.control(maxdepth = 1, cp = 0))
  as.character(fit$frame$var[1])
}
# ctree only splits when the best predictor passes its significance test (1 - p > 0.95)
first_split_ctree <- function(d) {
  fit <- party::ctree(y ~ ., data = d, controls = party::ctree_control(maxdepth = 1))
  if (fit@tree$terminal) "no split" else fit@tree$psplit$variableName
}

set.seed(624)
gran_results <- do.call(rbind, lapply(c(0, 0.3, 0.5), function(effect) {
  picks <- replicate(500, {
    d <- make_granularity_data(effect = effect)
    c(rpart = first_split_rpart(d), ctree = first_split_ctree(d))
  })
  data.frame(effect = effect, method = rep(c("CART (rpart)", "conditional (ctree)"), 500),
             first_split = as.vector(picks))
}))

gran_levels <- c("x_binary", "x_10", "x_50", "x_cont", "no split")
gran_results$first_split <- factor(gran_results$first_split, levels = gran_levels)
gran_table <- with(gran_results, table(effect, method, first_split))
ftable(gran_table)
##                            first_split x_binary x_10 x_50 x_cont no split
## effect method                                                            
## 0      CART (rpart)                          21   90  174    215        0
##        conditional (ctree)                   12    3    6     10      469
## 0.3    CART (rpart)                         207   47   95    151        0
##        conditional (ctree)                  173    3    2      5      317
## 0.5    CART (rpart)                         420   10   28     42        0
##        conditional (ctree)                  428    1    1      1       69
par(mfrow = c(1, 3), mar = c(5, 4, 3, 1))
for (e in c(0, 0.3, 0.5)) {
  counts <- gran_table[as.character(e), , ]
  barplot(counts, beside = TRUE, col = c("steelblue", "darkorange"), ylim = c(0, 500),
          las = 2, ylab = "Times chosen (out of 500)",
          main = paste("Effect of x_binary =", e))
  if (e == 0) legend("topleft", c("CART", "ctree"), fill = c("steelblue", "darkorange"),
                     bty = "n")
}

par(mfrow = c(1, 1))

With a pure-noise response, a method that is forced to split and has no preference would pick each predictor about 125 times out of 500. CART picks the continuous predictor 215 times, the 50-level predictor 174 times, the 10-level predictor 90 times and the binary predictor only 21 times, so the more distinct values a predictor has, the more often it wins the first split. The likely reason is the number of chances: CART evaluates every allowed split point and keeps the one with the lowest SSE, so the continuous predictor gets up to 199 tries at a lucky reduction and the binary predictor gets one. Here cp = 0 deliberately forces a split so that the choice can be recorded. Choosing the tree size by resampling can remove some unnecessary splits, but that comparison is not part of this simulation, and in a deeper tree the same bias acts at every node.

With a real effect on the binary predictor, the bias reduces how often CART picks the informative predictor first. With an effect of 0.3 CART picks the binary predictor only 207 times out of 500; in the other 293 runs it splits first on a predictor that has no relation to the response, mostly the continuous one. Only when the effect grows to 0.5 does the informative predictor win most of the time (420 of 500). This is the danger the book warns about: when informative predictors are coarse and noise predictors are granular, the noise ends up at the top of the tree.

The conditional inference tree behaves very differently. With pure noise it makes no split in 469 of 500 runs. It splits only 31 times (about 6%, close to the 5% false-positive rate of its 0.95 threshold), and those splits are spread over all four predictors (12, 3, 6 and 10 times), with no sign of a preference for the granular ones, although 31 splits are too few to show the choice is exactly even. With an effect of 0.3 it splits less often (183 times), but when it splits it chooses the binary predictor 173 times out of 183. With an effect of 0.5 it picks the binary predictor 428 times and a noise predictor only 3 times. ctree first tests the association between the response and each candidate predictor, with a multiplicity adjustment for the decision whether to split at all. If it does split, it takes the predictor with the strongest association and only then chooses that predictor’s split point. Separating the choice of predictor from the choice of cut point removes the advantage that CART’s exhaustive search gives predictors with more possible splits, which is the correction the book describes. The cost is power: with a weak effect it often makes no split at all.

Exercise 8.3

In stochastic gradient boosting the bagging fraction and learning rate will govern the construction of the trees as they are guided by the gradient. Although the optimal values of these parameters should be obtained through the tuning process, it is helpful to understand how the magnitudes of these parameters affect magnitudes of variable importance. Figure 8.24 provides the variable importance plots for boosting using two extreme values for the bagging fraction (0.1 and 0.9) and the learning rate (0.1 and 0.9) for the solubility data. The left-hand plot has both parameters set to 0.1, and the right-hand plot has both set to 0.9:

The answers below are reasoned from how the algorithm works and then checked by refitting both models on the solubility data. The book does not state the number of trees or the depth behind Figure 8.24, so the refit uses the settings of the book’s tuned boosted model (400 trees of depth 3) with the two extreme combinations.

data(solubility)

sol_settings <- data.frame(name = c("slow (0.1, 0.1)", "fast (0.9, 0.9)"),
                           bag.fraction = c(0.1, 0.9), shrinkage = c(0.1, 0.9))

sol_gbm <- fit_once("sol_gbm", lapply(1:2, function(i) {
  set.seed(100)
  gbm.fit(solTrainXtrans, solTrainY, distribution = "gaussian",
          n.trees = 400, interaction.depth = 3,
          shrinkage = sol_settings$shrinkage[i],
          bag.fraction = sol_settings$bag.fraction[i], verbose = FALSE)
}))
names(sol_gbm) <- sol_settings$name

# relative influence: each predictor's share (in percent) of the total squared-error reduction
sol_imp <- lapply(sol_gbm, function(m) summary(m, plotit = FALSE))
par(mfrow = c(1, 2), mar = c(4, 9, 3, 1))
for (nm in names(sol_imp)) {
  top <- head(sol_imp[[nm]], 25)
  barplot(rev(top$rel.inf), names.arg = rev(as.character(top$var)), horiz = TRUE, las = 1,
          cex.names = 0.7, col = "steelblue", xlab = "Relative influence (%)",
          main = nm)
}

par(mfrow = c(1, 1))
importance_slope <- function(imp) {
  c(top1 = imp$rel.inf[1], top5 = sum(imp$rel.inf[1:5]), top10 = sum(imp$rel.inf[1:10]),
    used_predictors = sum(imp$rel.inf > 0))
}
round(sapply(sol_imp, importance_slope), 1)
##                 slow (0.1, 0.1) fast (0.9, 0.9)
## top1                       19.3            35.6
## top5                       44.0            79.0
## top10                      56.4            85.6
## used_predictors           172.0           197.0

(a) Why the right-hand model concentrates its importance

Why does the model on the right focus its importance on just the first few of predictors, whereas the model on the left spreads importance across more predictors?

The two parameters push in the same direction. The two fits change both at once, so this comparison cannot separate their effects; what follows is the likely mechanism. The learning rate decides how much of each new tree’s prediction is added to the model. At 0.9, the first few trees add most of what they find, and what they find is driven by the strongest predictors, which here are the continuous ones like MolWeight, NumCarbon and the surface areas. The next trees are fit to the residuals, and the same strong predictors tend to keep winning splits there, collecting more importance. At 0.1, each tree adds only a tenth of what it found, so the strong predictors still have signal left for many more iterations, and as the residuals shrink slowly other predictors get turns as well.

The bagging fraction decides how much of the training data each tree sees. At 0.9 every tree sees almost the same data, so consecutive trees tend to make similar choices and the importance concentrates. At 0.1 each tree sees a different 10% of the samples (about 95 of 951 compounds), so the best split varies more from tree to tree and the importance is shared more evenly among the predictors that carry signal. The book makes the same point about the steep boosting importance curve: trees that depend on each other have correlated structure and keep picking the same predictors.

The refit confirms this. In the fast model the top predictor holds about 36% of the total relative influence and the top five about 79%; in the slow model the top predictor holds about 19% and the top five 44%. The plot shows the same steep drop on the right and gradual drop on the left as Figure 8.24. The fast model does give some importance to more predictors in total (197 against 172 with any importance), but most of that is tiny and in the tail; the slope at the top is what differs.

(b) Which model would predict other samples better?

Which model do you think would be more predictive of other samples?

I would expect the left-hand model to predict new samples better. A learning rate of 0.9 is close to no shrinkage at all, and the book describes boosting without regularization as greedy and prone to over-fitting: each step fits the current residuals as well as it can, including the noise in them. The book recommends a learning rate below 0.01 and a bagging fraction around 0.5, so the left-hand settings are much closer to the recommended ones on the learning rate. A bagging fraction of 0.1 is low and makes each tree noisy, but the small learning rate averages that noise over many trees, which is the point of stochastic gradient boosting.

To check, both settings are evaluated by 10-fold cross-validation on the training set and on the solubility test set:

sol_ctrl <- trainControl(method = "cv", number = 10)
sol_cv <- fit_once("sol_cv", lapply(1:2, function(i) {
  set.seed(100)
  train(solTrainXtrans, solTrainY, method = "gbm",
        tuneGrid = data.frame(n.trees = 400, interaction.depth = 3,
                              shrinkage = sol_settings$shrinkage[i], n.minobsinnode = 10),
        bag.fraction = sol_settings$bag.fraction[i],
        trControl = sol_ctrl, verbose = FALSE)
}))
names(sol_cv) <- sol_settings$name

sol_perf <- t(sapply(names(sol_gbm), function(nm) {
  c(CV_RMSE = sol_cv[[nm]]$results$RMSE, CV_R2 = sol_cv[[nm]]$results$Rsquared,
    postResample(predict(sol_gbm[[nm]], solTestXtrans, n.trees = 400), solTestY))
}))
round(sol_perf, 3)
##                 CV_RMSE CV_R2  RMSE Rsquared   MAE
## slow (0.1, 0.1)   0.661 0.896 0.698    0.888 0.515
## fast (0.9, 0.9)   0.781 0.857 0.744    0.876 0.530

The slow model is better on both: cross-validated RMSE 0.661 against 0.781, and test RMSE 0.698 against 0.744, with a higher R-squared on both. The difference on the test set is smaller than on resampling, but it points the same way. Neither setting would be the final choice; both are extreme, and the book’s tuned model (shrinkage 0.1, bagging fraction 0.5) sits between them.

(c) Effect of interaction depth on the importance slope

How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?

Interaction depth sets the maximum depth of each tree, so deeper trees allow more splits and higher-order interactions. Those extra splits can bring in new predictors or reuse ones already in the tree, so a flatter importance profile is possible but not guaranteed. My expectation is that greater depth spreads importance across more predictors, with a bigger effect on the slow model, because in the fast model the first trees already take most of the signal from the top predictors. Both settings are refit at depths 1, 3 and 7:

sol_depth <- fit_once("sol_depth", lapply(1:2, function(i) {
  lapply(c(1, 3, 7), function(depth) {
    set.seed(100)
    fit <- gbm.fit(solTrainXtrans, solTrainY, distribution = "gaussian",
                   n.trees = 400, interaction.depth = depth,
                   shrinkage = sol_settings$shrinkage[i],
                   bag.fraction = sol_settings$bag.fraction[i], verbose = FALSE)
    summary(fit, plotit = FALSE)
  })
}))
names(sol_depth) <- sol_settings$name

depth_table <- do.call(rbind, lapply(names(sol_depth), function(nm) {
  t(sapply(1:3, function(j) c(depth = c(1, 3, 7)[j], importance_slope(sol_depth[[nm]][[j]]))))
}))
data.frame(model = rep(names(sol_depth), each = 3), round(depth_table, 1))
##             model depth top1 top5 top10 used_predictors
## 1 slow (0.1, 0.1)     1 21.3 54.7  64.9             133
## 2 slow (0.1, 0.1)     3 19.3 44.0  56.4             172
## 3 slow (0.1, 0.1)     7 19.2 45.1  57.3             178
## 4 fast (0.9, 0.9)     1 40.7 76.2  85.0             118
## 5 fast (0.9, 0.9)     3 35.6 79.0  85.6             197
## 6 fast (0.9, 0.9)     7 36.2 78.0  86.1             225
par(mfrow = c(1, 2))
for (nm in names(sol_depth)) {
  top_value <- max(sapply(sol_depth[[nm]], function(s) s$rel.inf[1]))
  plot(NULL, xlim = c(1, 40), ylim = c(0, top_value),
       xlab = "Predictor rank", ylab = "Relative influence (%)", main = nm)
  cols <- c("gray40", "steelblue", "firebrick")
  for (j in 1:3) lines(1:40, sol_depth[[nm]][[j]]$rel.inf[1:40], type = "b", pch = 19,
                       cex = 0.5, col = cols[j])
  legend("topright", paste("depth", c(1, 3, 7)), col = cols, lwd = 1, pch = 19, bty = "n")
}

par(mfrow = c(1, 1))

The results partly match that expectation. In both models deeper trees bring in many more predictors: from 133 to 178 with any importance in the slow model, and from 118 to 225 in the fast one. In the slow model the top of the curve also flattens when going from depth 1 to depth 3 (the top five fall from about 55% to 44% of the importance), and depth 7 is about the same as depth 3. In the fast model the top of the curve barely changes: the top five hold about 76 to 79% at every depth, and the top predictor’s share drops only from 41% to about 36%. So increasing the interaction depth makes the tail of the importance curve longer and flatter for both models, but it only flattens the top of the curve for the slow model. A likely reason is that with a learning rate of 0.9 the first trees take most of the signal from the strongest predictors whatever their depth, so the extra splits in deeper trees land on predictors with little left to explain.

Exercise 8.7

Refer to Exercises 6.3 and 7.5 which describe a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several tree-based models:

The data, the 80/20 split (seed 624), the removal of near-zero-variance predictors, and the KNN imputation fitted inside each training resample are the same as in Exercises 6.3 and 7.5.

data(ChemicalManufacturingProcess)
process_predictors <- ChemicalManufacturingProcess[, -1]
yield <- ChemicalManufacturingProcess$Yield

set.seed(624)
cmp_rows <- createDataPartition(yield, p = 0.80, list = FALSE)

cmp_train_x <- process_predictors[cmp_rows, ]
cmp_test_x  <- process_predictors[-cmp_rows, ]
cmp_train_y <- yield[cmp_rows]
cmp_test_y  <- yield[-cmp_rows]

cmp_nzv <- nearZeroVar(cmp_train_x, names = TRUE)
cmp_nzv
## [1] "BiologicalMaterial07"
cmp_train_x <- cmp_train_x[, !(names(cmp_train_x) %in% cmp_nzv)]
cmp_test_x  <- cmp_test_x[, !(names(cmp_test_x) %in% cmp_nzv)]

c(train = nrow(cmp_train_x), test = nrow(cmp_test_x), predictors = ncol(cmp_train_x),
  missing_cells_train = sum(is.na(cmp_train_x)))
##               train                test          predictors missing_cells_train 
##                 144                  32                  56                  99

Trees do not need centered and scaled predictors, but preProc = "knnImpute" centers and scales as part of the imputation. That changes nothing for a tree: a split on a standardized value is the same split as on the raw value, only the threshold is expressed in standard deviations. The KNN imputation is kept because the exercise asks for the same imputation as Exercises 6.3 and 7.5, and it puts every model on exactly the same data. randomForest could not use the rows with missing values anyway (the book notes this), while some of the other implementations have their own ways of handling them.

(a) Which tree-based model gives the best resampling and test set performance?

Which tree-based regression model gives the optimal resampling and test set performance?

Seven tree-based models from the chapter, each tuned on the same repeated cross-validation folds (the seed is reset before each train() call):

cmp_ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5)

cmp_trees <- fit_once("cmp_trees", {
  set.seed(100)
  cmp_cart <- train(cmp_train_x, cmp_train_y, method = "rpart",
                    preProc = "knnImpute", tuneLength = 20, trControl = cmp_ctrl)

  set.seed(100)
  cmp_ctree <- train(cmp_train_x, cmp_train_y, method = "ctree",
                     preProc = "knnImpute",
                     tuneGrid = data.frame(mincriterion = seq(0.75, 0.99, length = 16)),
                     trControl = cmp_ctrl)

  set.seed(100)
  cmp_bag <- train(cmp_train_x, cmp_train_y, method = "treebag",
                   preProc = "knnImpute", nbagg = 50, trControl = cmp_ctrl)

  set.seed(100)
  cmp_rf <- train(cmp_train_x, cmp_train_y, method = "rf",
                  preProc = "knnImpute",
                  tuneGrid = data.frame(mtry = c(2, 15, 29, 42, 56)),
                  ntree = 1000, importance = TRUE, trControl = cmp_ctrl)

  set.seed(100)
  cmp_gbm <- train(cmp_train_x, cmp_train_y, method = "gbm",
                   preProc = "knnImpute",
                   tuneGrid = expand.grid(interaction.depth = seq(1, 7, by = 2),
                                          n.trees = seq(100, 1000, by = 50),
                                          shrinkage = c(0.01, 0.1),
                                          n.minobsinnode = 10),
                   trControl = cmp_ctrl, verbose = FALSE)

  set.seed(100)
  cmp_cubist <- train(cmp_train_x, cmp_train_y, method = "cubist",
                      preProc = "knnImpute",
                      tuneGrid = expand.grid(committees = c(1, 5, 10, 25, 50, 100),
                                             neighbors = c(0, 1, 5, 9)),
                      trControl = cmp_ctrl)

  list(CART = cmp_cart, CondTree = cmp_ctree, Bagged = cmp_bag,
       RF = cmp_rf, GBM = cmp_gbm, Cubist = cmp_cubist)
})
## Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,
## : There were missing values in resampled performance measures.
# M5 is cached on its own: the Weka model lives in Java, and .jcache() stores a copy
# inside the R object so the model still predicts after it is saved and reloaded
cmp_m5 <- fit_once("cmp_m5", {
  set.seed(100)
  m5 <- train(cmp_train_x, cmp_train_y, method = "M5",
              preProc = "knnImpute", trControl = cmp_ctrl,
              control = Weka_control(M = 10))
  rJava::.jcache(m5$finalModel$classifier)
  m5
})
cmp_trees <- c(cmp_trees[1:2], M5 = list(cmp_m5), cmp_trees[3:6])
lapply(cmp_trees, function(m) m$bestTune)
## $CART
##           cp
## 18 0.3323156
## 
## $CondTree
##    mincriterion
## 12        0.926
## 
## $M5
##   pruned smoothed rules
## 7     No       No   Yes
## 
## $Bagged
##   parameter
## 1      none
## 
## $RF
##   mtry
## 2   15
## 
## $GBM
##    n.trees interaction.depth shrinkage n.minobsinnode
## 76    1000                 7      0.01             10
## 
## $Cubist
##    committees neighbors
## 22        100         1

The tuning results behind the choices described below:

# resampled RMSE across the CART and conditional inference tree grids
round(cmp_trees$CART$results[, c("cp", "RMSE", "Rsquared")], 3)
##       cp  RMSE Rsquared
## 1  0.000 1.440    0.426
## 2  0.020 1.431    0.423
## 3  0.039 1.478    0.391
## 4  0.059 1.450    0.388
## 5  0.078 1.438    0.381
## 6  0.098 1.446    0.377
## 7  0.117 1.424    0.393
## 8  0.137 1.423    0.394
## 9  0.156 1.423    0.394
## 10 0.176 1.423    0.394
## 11 0.195 1.423    0.394
## 12 0.215 1.423    0.394
## 13 0.235 1.423    0.394
## 14 0.254 1.423    0.394
## 15 0.274 1.423    0.394
## 16 0.293 1.423    0.394
## 17 0.313 1.423    0.394
## 18 0.332 1.423    0.394
## 19 0.352 1.493    0.364
## 20 0.371 1.644    0.274
round(cmp_trees$CondTree$results[, c("mincriterion", "RMSE", "Rsquared")], 3)
##    mincriterion  RMSE Rsquared
## 1         0.750 1.454    0.393
## 2         0.766 1.455    0.393
## 3         0.782 1.454    0.393
## 4         0.798 1.457    0.391
## 5         0.814 1.465    0.384
## 6         0.830 1.456    0.387
## 7         0.846 1.455    0.389
## 8         0.862 1.450    0.390
## 9         0.878 1.446    0.393
## 10        0.894 1.445    0.392
## 11        0.910 1.441    0.395
## 12        0.926 1.428    0.403
## 13        0.942 1.434    0.399
## 14        0.958 1.442    0.392
## 15        0.974 1.441    0.386
## 16        0.990 1.441    0.384
# boosted trees: resampled RMSE at 500 and at 1,000 trees, by depth and shrinkage
gbm_res <- cmp_trees$GBM$results
round(xtabs(RMSE ~ interaction.depth + shrinkage, gbm_res[gbm_res$n.trees == 500, ]), 3)
##                  shrinkage
## interaction.depth  0.01   0.1
##                 1 1.182 1.140
##                 3 1.121 1.086
##                 5 1.103 1.080
##                 7 1.104 1.100
round(xtabs(RMSE ~ interaction.depth + shrinkage, gbm_res[gbm_res$n.trees == 1000, ]), 3)
##                  shrinkage
## interaction.depth  0.01   0.1
##                 1 1.142 1.137
##                 3 1.082 1.083
##                 5 1.070 1.075
##                 7 1.067 1.096
# random forest: resampled RMSE for each mtry value
round(cmp_trees$RF$results[, c("mtry", "RMSE", "Rsquared")], 3)
##   mtry  RMSE Rsquared
## 1    2 1.210    0.628
## 2   15 1.108    0.657
## 3   29 1.110    0.649
## 4   42 1.118    0.639
## 5   56 1.131    0.628
# Cubist with one neighbor, by number of committees
cubist_res <- cmp_trees$Cubist$results
round(cubist_res[cubist_res$neighbors == 1, c("committees", "RMSE", "Rsquared")], 4)
##    committees   RMSE Rsquared
## 2           1 1.2318   0.6093
## 6           5 1.0276   0.6813
## 10         10 0.9955   0.6937
## 14         25 0.9790   0.6938
## 18         50 0.9698   0.6996
## 22        100 0.9682   0.7020

Why these grids, and where the best values sit.

  • CART, tuneLength = 20: caret spreads 20 evenly spaced complexity-parameter values over the range of the full tree’s cost-complexity sequence. The table above shows the resampled RMSE is almost flat (about 1.42 to 1.48) from cp = 0 up to about 0.33, and the lowest value picked a tree with a single split. At the largest cp values some resamples prune the tree back to the root, which predicts the same yield for every run; that is why train() warns about missing values in the resampled performance (R-squared cannot be computed for a constant prediction).
  • Conditional inference tree, 16 thresholds from 0.75 to 0.99, the range the book uses. The best (0.926) is inside the grid, and the table shows the RMSE barely changes across it (about 1.43 to 1.47).
  • M5, all combinations of pruning, smoothing and rules, as in the book’s train(method = "M5") call, with M = 10 (the minimum node size the book uses). The best is the unpruned, unsmoothed rule version.
  • Bagged trees, 50 bootstrap samples, where the book says the gains stop. Nothing to tune.
  • Random forest, five values of mtry spread from 2 to P = 56 as the book suggests, with 1,000 trees. The best, 15, is inside the grid and close to the default P/3 (about 19); every value from 15 to 56 gives nearly the same RMSE (table above).
  • Boosted trees, the book’s grid (depth 1 to 7, 100 to 1,000 trees, shrinkage 0.01 or 0.1). The best uses 1,000 trees, depth 7 and shrinkage 0.01, which is on the edge of all three ranges, and the tables above show the shrinkage 0.01 models are still improving between 500 and 1,000 trees. So this grid does not establish whether a wider search would improve boosting further or close the gap to Cubist (1.067 against 0.968); answering that would need more trees on the same resamples, which was not run here.
  • Cubist, 1 to 100 committees and 0, 1, 5 or 9 neighbors. The best is 100 committees with 1 neighbor, on the edge of the grid. With one neighbor the RMSE falls by about 0.011 from 25 to 100 committees and about 0.002 from 50 to 100 (table above), so the curve has nearly levelled off, as on the solubility data where it stabilized around 50. Fifty committees would be a reasonable smaller choice with almost the same error; the lowest value is kept here.
cmp_resampled <- t(sapply(cmp_trees, function(m) {
  best <- merge(m$bestTune, m$results)
  c(RMSE = best$RMSE, Rsquared = best$Rsquared, MAE = best$MAE)
}))
cmp_test_perf <- t(sapply(cmp_trees, function(m)
  postResample(pred = predict(m, cmp_test_x), obs = cmp_test_y)))

cmp_perf <- cbind(cmp_resampled, cmp_test_perf)
colnames(cmp_perf) <- c("CV_RMSE", "CV_R2", "CV_MAE", "Test_RMSE", "Test_R2", "Test_MAE")
round(cmp_perf[order(cmp_perf[, "CV_RMSE"]), ], 3)
##          CV_RMSE CV_R2 CV_MAE Test_RMSE Test_R2 Test_MAE
## Cubist     0.968 0.702  0.698     0.882   0.834    0.597
## GBM        1.067 0.657  0.840     1.183   0.725    0.888
## RF         1.108 0.657  0.895     1.263   0.723    0.912
## Bagged     1.184 0.589  0.935     1.262   0.704    0.937
## CART       1.423 0.394  1.145     1.563   0.504    1.191
## CondTree   1.428 0.403  1.150     1.417   0.616    1.113
## M5         1.449 0.429  1.117     1.219   0.681    0.915
cmp_resamp <- resamples(cmp_trees)
bwplot(cmp_resamp, metric = "RMSE")

# paired differences on the shared folds, for the best model against the others
cmp_best_name <- rownames(cmp_resampled)[which.min(cmp_resampled[, "RMSE"])]
cmp_best_name
## [1] "Cubist"
cmp_diff <- summary(diff(cmp_resamp), metric = "RMSE")
cmp_diff
## 
## Call:
## summary.diff.resamples(object = diff(cmp_resamp), metric = "RMSE")
## 
## p-value adjustment: bonferroni 
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
## 
## MAE 
##          CART      CondTree  M5        Bagged    RF        GBM       Cubist   
## CART               -0.004662  0.028599  0.209964  0.249789  0.304890  0.447043
## CondTree 1.000000             0.033260  0.214625  0.254450  0.309551  0.451705
## M5       1.000000  1.000000             0.181365  0.221190  0.276291  0.418445
## Bagged   1.915e-09 2.138e-12 9.457e-05            0.039825  0.094926  0.237079
## RF       8.896e-11 5.306e-13 5.577e-06 0.001522             0.055101  0.197255
## GBM      6.739e-11 2.470e-13 1.704e-07 6.679e-06 0.002876             0.142154
## Cubist   9.555e-15 7.188e-16 1.055e-11 6.291e-10 4.268e-09 1.773e-05          
## 
## RMSE 
##          CART      CondTree  M5        Bagged    RF        GBM       Cubist   
## CART               -0.005494 -0.026239  0.239086  0.314649  0.355995  0.454589
## CondTree 1.000000            -0.020745  0.244580  0.320142  0.361489  0.460083
## M5       1.000000  1.000000             0.265325  0.340888  0.382234  0.480828
## Bagged   1.030e-09 1.049e-13 3.704e-06            0.075562  0.116909  0.215503
## RF       4.132e-11 9.177e-14 4.759e-08 6.611e-07            0.041346  0.139940
## GBM      2.638e-10 9.947e-14 1.988e-08 3.273e-06 0.257957             0.098594
## Cubist   3.054e-10 2.288e-11 8.881e-10 6.071e-05 0.009273  0.142992           
## 
## Rsquared 
##          CART      CondTree   M5         Bagged     RF         GBM       
## CART               -0.0085960 -0.0344124 -0.1944159 -0.2629772 -0.2631454
## CondTree 1.0000000            -0.0258165 -0.1858200 -0.2543812 -0.2545495
## M5       1.0000000 1.0000000             -0.1600035 -0.2285647 -0.2287330
## Bagged   2.919e-09 1.479e-12  5.830e-06             -0.0685612 -0.0687295
## RF       3.342e-11 6.207e-14  4.033e-09  1.848e-07             -0.0001683
## GBM      6.777e-11 2.269e-14  2.738e-08  1.199e-05  1.0000000            
## Cubist   2.085e-11 2.952e-12  1.089e-09  0.0003733  0.7187865  0.6186397 
##          Cubist    
## CART     -0.3077701
## CondTree -0.2991742
## M5       -0.2733577
## Bagged   -0.1133542
## RF       -0.0447930
## GBM      -0.0446247
## Cubist
plot(cmp_trees$GBM)

plot(cmp_trees$Cubist)

plot(cmp_trees$RF)

The test set has only 32 runs, so the largest test errors of each model show whether a few runs drive the test numbers (row numbers are the run’s position in the original data):

sapply(cmp_trees, function(m) {
  err <- predict(m, cmp_test_x) - cmp_test_y
  names(err) <- rownames(cmp_test_x)
  top <- head(sort(abs(err), decreasing = TRUE), 3)
  paste0(names(top), " (", round(top, 2), ")")
})
##      CART        CondTree    M5           Bagged      RF          GBM         
## [1,] "49 (4.92)" "49 (4.64)" "49 (3.79)"  "49 (3.69)" "49 (3.8)"  "49 (3.75)" 
## [2,] "28 (3.11)" "28 (2.8)"  "44 (2.5)"   "44 (3.09)" "44 (3.39)" "44 (2.37)" 
## [3,] "44 (2.74)" "44 (2.46)" "132 (2.21)" "6 (2.11)"  "6 (2.25)"  "121 (2.23)"
##      Cubist      
## [1,] "49 (2.43)" 
## [2,] "121 (2.43)"
## [3,] "44 (2.1)"
# test RMSE with all 32 runs and without run 49, the run every model predicts worst
run49 <- rownames(cmp_test_x) == "49"
round(t(sapply(cmp_trees, function(m) {
  pred <- predict(m, cmp_test_x)
  c(all_32_runs = RMSE(pred, cmp_test_y),
    without_run_49 = RMSE(pred[!run49], cmp_test_y[!run49]))
})), 3)
##          all_32_runs without_run_49
## CART           1.563          1.320
## CondTree       1.417          1.174
## M5             1.219          1.034
## Bagged         1.262          1.097
## RF             1.263          1.086
## GBM            1.183          0.995
## Cubist         0.882          0.782

Cubist gives the best resampling and test set performance among the models tried: resampled RMSE 0.968 (R-squared 0.70) and test RMSE 0.882 (R-squared 0.83), the lowest of the seven on both. The rankings below it differ between the two measures. By resampled RMSE, boosted trees are second (1.067) and the random forest third (1.108); on the test set boosted trees are second (1.183), then M5 (1.219), bagged trees (1.262) and the random forest (1.263). The paired comparison on the shared folds shows Cubist’s resampled RMSE is lower than the random forest’s (adjusted p-value about 0.009) and everything below it, but it did not detect a difference between Cubist and boosted trees (a lead of about 0.10, adjusted p-value about 0.14).

The ensembles split clearly from the single-tree and rule models. CART, the conditional inference tree and M5 (whose selected version is a rule model) all have resampled RMSE around 1.42 to 1.45 and R-squared around 0.4, and the paired comparison finds no differences among them. That is the chapter’s main lesson repeated on new data: a single tree is easy to read but predicts poorly, and averaging many trees (bagging and random forests) or adding them in sequence (boosting, and Cubist’s committees) improves on it a lot. Cubist’s linear models inside the rules and its neighbor correction can follow gradual changes in yield that a tree’s steps cannot, which is a plausible reason it does best, although this comparison does not show which part of Cubist made the difference.

Run 49 is the worst-predicted test run for every model. Without it every model’s test RMSE drops, and Cubist still has the lowest (0.782, then boosted trees at 0.995), so Cubist’s test lead does not depend on that one run; the main evaluation still uses all 32 runs. Cubist’s test RMSE is lower than its resampled RMSE. With only 32 test runs that gap could simply reflect which runs ended up in the test set, so the resampled estimate (0.968) is the figure to rely on, keeping in mind that it was also used to choose the tuning values.

(b) Most important predictors, and comparison with the linear and nonlinear models

Which predictors are most important in the optimal tree-based regression model? Do either the biological or process variables dominate the list? How do the top 10 important predictors compare to the top 10 predictors from the optimal linear and nonlinear models?

cmp_best <- cmp_trees[[cmp_best_name]]
cmp_tree_imp <- varImp(cmp_best)
plot(cmp_tree_imp, top = 15, main = paste("Top 15 predictors,", cmp_best_name))

tree_top10 <- rownames(cmp_tree_imp$importance)[order(-cmp_tree_imp$importance$Overall)][1:10]
tree_top10
##  [1] "ManufacturingProcess32" "ManufacturingProcess17" "ManufacturingProcess09"
##  [4] "ManufacturingProcess33" "ManufacturingProcess01" "ManufacturingProcess37"
##  [7] "ManufacturingProcess45" "ManufacturingProcess15" "ManufacturingProcess21"
## [10] "BiologicalMaterial02"
table(ifelse(grepl("^Biological", tree_top10), "Biological", "Process"))
## 
## Biological    Process 
##          1          9

For Cubist, caret’s varImp() averages two of Cubist’s usage statistics, use in the rule conditions and use in the rules’ linear models, across all 100 committees (scaled 0 to 100 in the plot). That is the kind of usage count the book suggests; it shows what the model leans on, unlike the loess filter caret used for the SVM in Exercise 7.5, but it counts uses rather than error reduction, so a predictor that appears in many linear models with small coefficients can rank high. The unscaled scores show several ties:

# Cubist usage scores before rescaling; equal values are ties
cubist_usage <- varImp(cmp_best, scale = FALSE)$importance
head(cubist_usage[order(-cubist_usage$Overall), , drop = FALSE], 12)
##                        Overall
## ManufacturingProcess32    49.5
## ManufacturingProcess17    24.0
## ManufacturingProcess09    24.0
## ManufacturingProcess33    23.5
## ManufacturingProcess01    23.0
## ManufacturingProcess37    23.0
## ManufacturingProcess45    22.5
## ManufacturingProcess15    22.0
## ManufacturingProcess21    22.0
## BiologicalMaterial02       2.5
## BiologicalMaterial10       1.5
## BiologicalMaterial06       1.5

The optimal linear model (the elastic net from Exercise 6.3) and the optimal nonlinear model (the radial-kernel SVM from Exercise 7.5) are refit here with the same code, folds and seed as in those exercises, so their top ten lists can be compared directly.

cmp_others <- fit_once("cmp_others", {
  set.seed(100)
  cmp_enet <- train(cmp_train_x, cmp_train_y, method = "enet",
                    preProc = "knnImpute",
                    tuneGrid = expand.grid(lambda = c(0.01, 0.1, 1),
                                           fraction = seq(0.05, 1, length = 20)),
                    trControl = cmp_ctrl)
  set.seed(100)
  cmp_svm <- train(cmp_train_x, cmp_train_y, method = "svmRadial",
                   preProc = "knnImpute", tuneLength = 14, trControl = cmp_ctrl)
  list(Enet = cmp_enet, SVM = cmp_svm)
})

# linear: absolute elastic net coefficients at the chosen fraction (standardized predictors)
lin_coef <- predict(cmp_others$Enet$finalModel, s = cmp_others$Enet$bestTune$fraction,
                    mode = "fraction", type = "coefficients")$coefficients
lin_top10 <- names(sort(abs(lin_coef[lin_coef != 0]), decreasing = TRUE))[1:10]

# nonlinear: caret's importance for the SVM (a loess R-squared filter, as in Exercise 7.5)
svm_imp <- varImp(cmp_others$SVM)$importance
nl_top10 <- rownames(svm_imp)[order(-svm_imp$Overall)][1:10]

data.frame(Rank = 1:10, Tree = tree_top10, Nonlinear_SVM = nl_top10, Linear_Enet = lin_top10)
##    Rank                   Tree          Nonlinear_SVM            Linear_Enet
## 1     1 ManufacturingProcess32 ManufacturingProcess32 ManufacturingProcess32
## 2     2 ManufacturingProcess17   BiologicalMaterial06 ManufacturingProcess09
## 3     3 ManufacturingProcess09 ManufacturingProcess36 ManufacturingProcess36
## 4     4 ManufacturingProcess33   BiologicalMaterial03 ManufacturingProcess17
## 5     5 ManufacturingProcess01 ManufacturingProcess09 ManufacturingProcess34
## 6     6 ManufacturingProcess37 ManufacturingProcess13   BiologicalMaterial06
## 7     7 ManufacturingProcess45 ManufacturingProcess31 ManufacturingProcess11
## 8     8 ManufacturingProcess15   BiologicalMaterial02 ManufacturingProcess06
## 9     9 ManufacturingProcess21 ManufacturingProcess11 ManufacturingProcess13
## 10   10   BiologicalMaterial02   BiologicalMaterial12   BiologicalMaterial03
list(in_all_three = Reduce(intersect, list(tree_top10, nl_top10, lin_top10)),
     tree_only = setdiff(tree_top10, union(nl_top10, lin_top10)))
## $in_all_three
## [1] "ManufacturingProcess32" "ManufacturingProcess09"
## 
## $tree_only
## [1] "ManufacturingProcess33" "ManufacturingProcess01" "ManufacturingProcess37"
## [4] "ManufacturingProcess45" "ManufacturingProcess15" "ManufacturingProcess21"
# all three families on the same folds
cmp_all <- resamples(c(cmp_trees[cmp_best_name], cmp_others))
summary(cmp_all, metric = "RMSE")
## 
## Call:
## summary.resamples(object = cmp_all, metric = "RMSE")
## 
## Models: Cubist, Enet, SVM 
## Number of resamples: 50 
## 
## RMSE 
##             Min.   1st Qu.    Median      Mean  3rd Qu.     Max. NA's
## Cubist 0.5232784 0.7466358 0.9763964 0.9682315 1.109176 1.734572    0
## Enet   0.8009208 1.0016908 1.1374785 1.1961284 1.302849 2.904083    0
## SVM    0.6626650 1.0035396 1.1342706 1.1304262 1.294400 1.616642    0
summary(diff(cmp_all), metric = "RMSE")
## 
## Call:
## summary.diff.resamples(object = diff(cmp_all), metric = "RMSE")
## 
## p-value adjustment: bonferroni 
## Upper diagonal: estimates of the difference
## Lower diagonal: p-value for H0: difference = 0
## 
## MAE 
##        Cubist    Enet     SVM     
## Cubist           -0.27121 -0.23178
## Enet   6.197e-14           0.03943
## SVM    3.560e-10 0.4599           
## 
## RMSE 
##        Cubist    Enet    SVM    
## Cubist           -0.2279 -0.1622
## Enet   4.143e-06          0.0657
## SVM    9.012e-05 0.3965         
## 
## Rsquared 
##        Cubist    Enet     SVM     
## Cubist            0.12612  0.09932
## Enet   4.610e-07          -0.02680
## SVM    8.074e-05 0.8051

The three biological predictors that the lists treat differently are closely related in the training data:

bio_three <- c("BiologicalMaterial02", "BiologicalMaterial03", "BiologicalMaterial06")
round(cor(cmp_train_x[, bio_three], use = "pairwise.complete.obs"), 3)
##                      BiologicalMaterial02 BiologicalMaterial03
## BiologicalMaterial02                1.000                0.849
## BiologicalMaterial03                0.849                1.000
## BiologicalMaterial06                0.954                0.862
##                      BiologicalMaterial06
## BiologicalMaterial02                0.954
## BiologicalMaterial03                0.862
## BiologicalMaterial06                1.000

Process predictors dominate. Nine of Cubist’s top ten are manufacturing process predictors; the only biological one is BiologicalMaterial02, in tenth place. ManufacturingProcess32 is first, then ManufacturingProcess17 and ManufacturingProcess09, which are tied on the unscaled usage score (as are 01 and 37, and 15 and 21), so the order within each tied pair means nothing. The tree model is more process-heavy than either earlier model: the SVM’s list (from a one-predictor-at-a-time filter) had four biological predictors, and the elastic net’s had two.

ManufacturingProcess32 is first in all three lists, and ManufacturingProcess09 is in all three as well, so those two are the most consistent findings across the three families of models. ManufacturingProcess17 is shared with the elastic net (fourth there) and BiologicalMaterial02 with the SVM. The other six, ManufacturingProcess33, 01, 37, 45, 15 and 21, appear only in the Cubist list. Some of that is likely the measure: usage counts reward predictors that Cubist puts into many of its rules’ linear models, and there are many process predictors, so different rules can use different ones. The biological predictors the other two models ranked high, BiologicalMaterial06 and 03, are strongly correlated with each other and with 02 in the training data (0.85 to 0.95 in the table above), so Cubist may be using one of them where the others would do; the lists alone cannot show that.

The comparison of performance is the more striking result. On the same 50 resamples Cubist’s RMSE is lower than the SVM’s by about 0.16 and the elastic net’s by about 0.23, with adjusted p-values well below 0.001. In Exercise 7.5 the best nonlinear model could not be separated from the elastic net; here a tree-based model is the first one in these three homeworks with a clear advantage over the linear model in this resampling comparison. For a plant engineer that makes Cubist the leading candidate for predicting yield, if an error of about one yield point is accurate enough for the decisions involved, with the caution that its 100 committees of rules are much harder to explain than the elastic net’s coefficients.

(c) The optimal single tree

Plot the optimal single tree with the distribution of yield in the terminal nodes. Does this view of the data provide additional knowledge about the biological or process predictors and their relationship with yield?

The single tree is the tuned CART model from part (a), which has the lowest resampled RMSE of the three single-tree and rule candidates (the conditional inference tree and M5 have lower test RMSE, but the choice is made on resampling). The tree in train() was fitted on standardized predictors, so its thresholds are in standard deviations. To make the plot readable in process units, the training data are imputed with the same KNN imputation and then put back on their original scale, and the tree is refit with the selected complexity parameter. Because a tree split depends only on the order of the values, this gives the same tree with thresholds in the original units. The plot uses the partykit conversion shown in the book, which draws the distribution of yield in each terminal node as a box plot.

cmp_impute <- preProcess(cmp_train_x, method = "knnImpute")
cmp_train_std <- predict(cmp_impute, cmp_train_x)
# undo the centering and scaling that knnImpute applies, keeping the imputed values
cmp_train_imputed <- sweep(sweep(cmp_train_std, 2, cmp_impute$std[names(cmp_train_std)], "*"),
                           2, cmp_impute$mean[names(cmp_train_std)], "+")
cmp_train_imputed$Yield <- cmp_train_y

cmp_single_tree <- rpart(Yield ~ ., data = cmp_train_imputed,
                         control = rpart.control(cp = cmp_trees$CART$bestTune$cp))
cmp_single_tree
## n= 144 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
## 1) root 144 453.4699 40.15799  
##   2) ManufacturingProcess32< 159.5 83 152.3024 39.23084 *
##   3) ManufacturingProcess32>=159.5 61 132.7435 41.41951 *
# same splitting variables as the tuned model?
identical(as.character(cmp_single_tree$frame$var),
          as.character(cmp_trees$CART$finalModel$frame$var))
## [1] TRUE
plot(as.party(cmp_single_tree), gp = gpar(fontsize = 8))

# yield summary in each terminal node, with the path that leads to it
cmp_leaf <- cmp_single_tree$where
leaf_summary <- aggregate(cmp_train_y, list(node = cmp_leaf),
                          function(v) c(n = length(v), mean = mean(v), min = min(v),
                                        q25 = unname(quantile(v, 0.25)),
                                        q75 = unname(quantile(v, 0.75)), max = max(v)))
leaf_summary <- data.frame(node = leaf_summary$node, round(leaf_summary$x, 2))
leaf_summary[order(leaf_summary$mean), ]
##   node  n  mean   min   q25   q75   max
## 1    2 83 39.23 35.25 38.36 40.05 43.12
## 2    3 61 41.42 36.83 40.28 42.45 44.35

The optimal single tree is a single split. Runs with ManufacturingProcess32 below 159.5 have a mean yield of about 39.2, and runs at or above 159.5 have a mean of about 41.4. The yield ranges of the two leaves overlap a lot (about 35 to 43 on the low side and 37 to 44 on the high side), although the middle halves do not (quartiles 38.4 to 40.1 against 40.3 to 42.5 in the table), so the split shifts the typical yield by about two points without separating good runs from bad ones. It still adds something the importance lists do not: a threshold. It is the first split this CART fit chose, which agrees with every model in this and the last two exercises ranking ManufacturingProcess32 first. A single split also shows why CART predicts so poorly: with only two possible predictions, it cannot follow the gradual changes the ensembles pick up.

Because the CART profile is flat, a larger tree has almost the same resampled RMSE as the stump. To see what more splits would add, the tree below is the one with the lowest resampled RMSE among the settings that give more than two leaves on the training data:

# among the cp values that give more than two leaves when the tree is refit,
# the one with the lowest resampled RMSE
cart_res <- cmp_trees$CART$results
leaf_count <- function(cp) {
  fit <- rpart(Yield ~ ., data = cmp_train_imputed, control = rpart.control(cp = cp))
  sum(fit$frame$var == "<leaf>")
}
cart_leaves <- sapply(cart_res$cp, leaf_count)
cart_bigger <- cart_res[cart_leaves > 2, ]
cp_deeper <- cart_bigger$cp[which.min(cart_bigger$RMSE)]

# compare its resampled RMSE with the selected stump's
cart_best <- cart_res[cart_res$cp == cmp_trees$CART$bestTune$cp, ]
c(cp = cp_deeper, RMSE = min(cart_bigger$RMSE), best_RMSE = cart_best$RMSE)
##         cp       RMSE  best_RMSE 
## 0.01954798 1.43089107 1.42282055
cmp_deeper_tree <- rpart(Yield ~ ., data = cmp_train_imputed,
                         control = rpart.control(cp = cp_deeper))
cmp_deeper_tree   # path, size and mean yield of every node
## n= 144 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
##  1) root 144 453.469900 40.15799  
##    2) ManufacturingProcess32< 159.5 83 152.302400 39.23084  
##      4) BiologicalMaterial12< 19.735 31  36.094620 38.32839  
##        8) BiologicalMaterial05>=19.6 8   5.544200 37.21500 *
##        9) BiologicalMaterial05< 19.6 23  17.183970 38.71565 *
##      5) BiologicalMaterial12>=19.735 52  75.909330 39.76885  
##       10) ManufacturingProcess25>=4824.4 39  27.750140 39.29744 *
##       11) ManufacturingProcess25< 4824.4 13  13.491680 41.18308 *
##    3) ManufacturingProcess32>=159.5 61 132.743500 41.41951  
##      6) BiologicalMaterial03< 72 48  92.191830 41.08688  
##       12) ManufacturingProcess06< 206.3 14  20.613490 40.11071 *
##       13) ManufacturingProcess06>=206.3 34  52.744750 41.48882  
##         26) ManufacturingProcess14< 4902 27  34.123900 41.17037  
##           52) BiologicalMaterial09>=12.67 15  18.705570 40.56133 *
##           53) BiologicalMaterial09< 12.67 12   2.899567 41.93167 *
##         27) ManufacturingProcess14>=4902 7   5.321343 42.71714 *
##      7) BiologicalMaterial03>=72 13  15.631030 42.64769 *
plot(as.party(cmp_deeper_tree), gp = gpar(fontsize = 8))

The larger tree has nine leaves and a resampled RMSE of 1.431 against 1.423 for the stump. The estimates are close, but that does not establish that the two predict equally well; the stump has the lower mean error and is simpler, so it stays the selected model, and the larger tree is shown only to explore the training data further. Its printed output gives the path, size and mean yield of every node. It keeps the same first split on ManufacturingProcess32, and most of the next splits are on biological predictors: BiologicalMaterial12 and BiologicalMaterial05 on the low side, and BiologicalMaterial03 and BiologicalMaterial09 on the high side. That is information the importance lists do not give. Within each side of the main split, the tree divides the runs further by biological measurements: on the high side, runs with BiologicalMaterial03 at 72 or more average about 42.6, and on the low side, runs with low BiologicalMaterial12 and high BiologicalMaterial05 average about 37.2, the lowest leaf. Three process predictors also appear further down (ManufacturingProcess25, ManufacturingProcess06 and ManufacturingProcess14); for example, among the low-ManufacturingProcess32 runs with higher BiologicalMaterial12, the runs with lower ManufacturingProcess25 average about 1.9 points more yield (41.2 against 39.3).

These leaves hold only 7 to 39 runs each, and the book warns that a single tree is unstable (a slightly different sample can give different splits), so the deeper splits are hypotheses to check, not settings to adopt. Higher ManufacturingProcess32 values go with higher yield in these training runs, but the data are observational, so that does not show that raising the setting would raise yield. The biological and secondary process splits point to relationships worth investigating with the process engineers and testing in a designed experiment before any operating or screening rule is changed.