I investigate how correlated predictors and predictor granularity affect tree-based importance, explain the boosting settings in Figure 8.24, and return to the chemical manufacturing data. For the chemical data I use the same split, training-set KNN imputation, and preprocessing as in Homework 8. The test set is reserved for a final comparison; cross-validation selects the models.
Question (Kuhn & Johnson 8.2). Use a simulation to show tree bias with different granularities.
I simulate six independent, equally uninformative predictors, with 2, 5, 10, 20, 50, and 100 distinct ordered values. The response is independent standard normal noise. For each of 500 simulations, I fit a one-split regression tree and record the selected variable. If the selection were unbiased, each would be selected about one-sixth of the time. More distinct values provide more candidate split points and more chances to find an accidental reduction in training error.
set.seed(8200)
granularities <- c(2, 5, 10, 20, 50, 100)
selected <- replicate(500, {
x <- as.data.frame(lapply(granularities, function(k) sample.int(k, 200, TRUE)))
names(x) <- paste0("Levels", granularities)
x$y <- rnorm(200)
tree <- rpart(y ~ ., data = x, method = "anova",
control = rpart.control(maxdepth = 1, cp = 0, minbucket = 10,
xval = 0))
as.character(tree$frame$var[1])
})
granularity_results <- data.frame(
Levels = granularities,
Selections = as.integer(table(factor(selected, levels = paste0("Levels", granularities))))
) |>
mutate(Percent = 100 * Selections / length(selected))
granularity_results
## Levels Selections Percent
## 1 2 14 2.8
## 2 5 50 10.0
## 3 10 62 12.4
## 4 20 95 19.0
## 5 50 111 22.2
## 6 100 168 33.6
ggplot(granularity_results, aes(x = factor(Levels), y = Percent)) +
geom_col(fill = "steelblue") +
geom_hline(yintercept = 100 / 6, linetype = "dashed", colour = "darkred") +
labs(title = "Root-split selection under a completely null signal",
subtitle = "Dashed line: equal selection probability",
x = "Number of distinct values", y = "Selected in simulations (%)")
This is selection bias in training splits, not evidence that the high-granularity variable predicts new responses. The simulation treats the levels as ordered numeric values; nominal factors with many possible partitions can show related but different degrees of bias. The 2-level variable is selected in 2.8% of simulations, compared with 33.6% for the 100-level variable, despite their identical lack of predictive signal.
Question (Kuhn & Johnson 8.3). In stochastic gradient boosting the bagging fraction and learning rate will govern the construction of the trees as they are guided by the gradient. Although the optimal values of these parameters should be obtained through the tuning process, it is helpful to understand how the magnitudes of these parameters affect magnitudes of variable importance. Figure 8.24 provides the variable importance plots for boosting using two extreme values for the bagging fraction (0.1 and 0.9) and the learning rate (0.1 and 0.9) for the solubility data. The left-hand plot has both parameters set to 0.1, and the right-hand plot has both set to 0.9.
Why does the model on the right focus its importance on just the first few of predictors, whereas the model on the left spreads importance across more predictors?
Which model do you think would be more predictive of other samples?
How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?
Figure 8.24 in the book shows gradient boosting importance on the
solubility data for two settings: shrinkage (learning rate) 0.1 with
bagging fraction 0.1, and shrinkage 0.9 with bagging fraction 0.9. I
refit these two models on the same data (solubility in
AppliedPredictiveModeling, transformed predictors) with
1000 trees, and also repeat them at interaction depths 1, 5, and 10 for
part (c). Importance is gbm’s relative influence, rescaled to sum to
100% within each model so that models are comparable. Test RMSE uses the
untouched solubility test set.
data(solubility)
sol_train <- cbind(solTrainXtrans, y = solTrainY)
boost_grid <- expand.grid(Setting = c("LR 0.1, bag 0.1", "LR 0.9, bag 0.9"),
Depth = c(1, 5, 10), stringsAsFactors = FALSE)
boost_grid$Shrinkage <- ifelse(grepl("0.1,", boost_grid$Setting), 0.1, 0.9)
boost_fits <- lapply(seq_len(nrow(boost_grid)), function(i) {
set.seed(830 + i)
fit <- gbm(y ~ ., data = sol_train, distribution = "gaussian",
n.trees = 1000, interaction.depth = boost_grid$Depth[i],
shrinkage = boost_grid$Shrinkage[i],
bag.fraction = boost_grid$Shrinkage[i],
n.minobsinnode = 10, verbose = FALSE)
imp <- gbm::relative.influence(fit, n.trees = 1000, scale. = FALSE)
imp <- sort(100 * imp / sum(imp), decreasing = TRUE)
pred <- predict(fit, solTestXtrans, n.trees = 1000)
list(importance = imp, test_rmse = sqrt(mean((pred - solTestY)^2)))
})
boost_summary <- bind_cols(boost_grid[, c("Setting", "Depth")],
bind_rows(lapply(boost_fits, function(f) data.frame(
Top_predictor_pct = f$importance[1],
Top5_pct = sum(f$importance[1:5]),
Top10_pct = sum(f$importance[1:10]),
Predictors_for_80pct = which(cumsum(f$importance) >= 80)[1],
Test_RMSE = f$test_rmse))))
rownames(boost_summary) <- NULL
boost_summary |> mutate(across(where(is.numeric), ~ round(.x, 2)))
## Setting Depth Top_predictor_pct Top5_pct Top10_pct
## 1 LR 0.1, bag 0.1 1 17.53 46.63 57.94
## 2 LR 0.9, bag 0.9 1 36.57 71.62 82.18
## 3 LR 0.1, bag 0.1 5 11.82 38.58 54.34
## 4 LR 0.9, bag 0.9 5 37.04 78.40 86.25
## 5 LR 0.1, bag 0.1 10 14.45 38.59 53.15
## 6 LR 0.9, bag 0.9 10 36.24 78.65 84.27
## Predictors_for_80pct Test_RMSE
## 1 42 0.68
## 2 9 0.69
## 3 53 0.66
## 4 6 0.81
## 5 54 0.62
## 6 6 0.84
curve_data <- bind_rows(lapply(seq_len(nrow(boost_grid)), function(i) {
imp <- boost_fits[[i]]$importance
data.frame(Setting = boost_grid$Setting[i], Depth = boost_grid$Depth[i],
Rank = seq_along(imp), Importance = as.numeric(imp))
}))
ggplot(filter(curve_data, Depth == 1, Rank <= 40),
aes(x = Rank, y = Importance, colour = Setting)) +
geom_line(linewidth = 1) +
labs(title = "Ranked importance, interaction depth 1",
subtitle = "Right-hand setting piles importance on a few predictors",
x = "Predictor rank", y = "Relative influence (%)", colour = NULL)
ggplot(filter(curve_data, Rank <= 40),
aes(x = Rank, y = Importance, colour = factor(Depth))) +
geom_line(linewidth = 1) +
facet_wrap(~ Setting) +
labs(title = "Effect of interaction depth on the importance curve",
x = "Predictor rank", y = "Relative influence (%)", colour = "Depth")
(a) The refit shows the book’s pattern. At depth 1 the 0.9/0.9 model puts 36.6% of its importance on a single predictor and 71.6% on its top five, needing only 9 predictors to reach 80% of total importance. The 0.1/0.1 model gives its top predictor 17.5% and its top five 46.6%, needing 42 predictors to reach 80%. With a large learning rate and nearly the full data in each tree, the first trees remove most of the residual that the strongest predictors can explain and every later tree is built on almost the same data, so the same predictors keep winning the splits. With small steps and small random subsamples, the strong predictors cannot absorb everything quickly, the subsamples differ from tree to tree, and weaker predictors win splits on some trees, which spreads the importance.
(b) On the untouched test set, the two models are effectively tied at depth 1: RMSE 0.679 for 0.1/0.1 versus 0.693 for 0.9/0.9. The difference appears at greater depth, where the 0.1/0.1 model is clearly better: 0.66 versus 0.806 at depth 5, and 0.621 versus 0.84 at depth 10. The 0.9/0.9 model gets worse as trees deepen while the 0.1/0.1 model improves, so I would expect the left-hand model (small learning rate, small bagging fraction) to be more predictive of other samples, particularly with deeper trees. This is one data set and one seed per setting, not a full tuning exercise.
(c) The effect of interaction depth is not the same for the two models. For the 0.1/0.1 model the share held by the top five predictors goes from 46.6% (depth 1) to 38.6% (depth 5) and 38.6% (depth 10), so its importance curve becomes flatter with deeper trees. For the 0.9/0.9 model the top-five share goes from 71.6% to 78.4% and 78.6%, so its slope stays steep or becomes slightly steeper. Deeper trees therefore flatten the curve when the learning rate is small, but do not spread importance when large steps already let a few strong predictors dominate. Test RMSE for the 0.9/0.9 model worsens with depth (0.693, 0.806, 0.84), consistent with overfitting.
Question (Kuhn & Johnson 8.7). Refer to Exercises 6.3 and 7.5 which describe a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several tree-based models:
Which tree-based regression model gives the optimal resampling and test set performance?
Which predictors are most important in the optimal tree-based regression model? Do either the biological or process variables dominate the list? How do the top 10 important predictors compare to the top 10 predictors from the optimal linear and nonlinear models?
Plot the optimal single tree with the distribution of yield in the terminal nodes. Does this view of the data provide additional knowledge about the biological or process predictors and their relationship with yield?
As in Homework 8, I use the seed-624, 75% stratified split, fit KNN imputation on the full training partition, filter near-zero-variance variables using that partition, and center and scale within model fitting. The same seed-6240 ten folds are used for all models. Because the imputer is fitted before the folds are formed, validation folds do influence the imputation step; CV may consequently be somewhat optimistic. The held-out test data do not influence imputation or tuning.
data(ChemicalManufacturingProcess)
cmp <- ChemicalManufacturingProcess
predictors_raw <- cmp[, -1, drop = FALSE]
yield <- as.numeric(cmp[, 1])
set.seed(624)
chem_idx <- createDataPartition(yield, p = 0.75, list = FALSE)
chem_imputer <- preProcess(predictors_raw[chem_idx, , drop = FALSE],
method = "knnImpute")
predictors_imputed <- predict(chem_imputer, predictors_raw)
nzv_chem <- nearZeroVar(predictors_imputed[chem_idx, , drop = FALSE])
if (length(nzv_chem) > 0) {
predictors_imputed <- predictors_imputed[, -nzv_chem, drop = FALSE]
}
chem_X_train <- predictors_imputed[chem_idx, , drop = FALSE]
chem_X_test <- predictors_imputed[-chem_idx, , drop = FALSE]
chem_y_train <- yield[chem_idx]
chem_y_test <- yield[-chem_idx]
set.seed(6240)
chem_folds <- createFolds(chem_y_train, k = 10, returnTrain = TRUE)
chem_ctrl <- trainControl(method = "cv", number = 10, index = chem_folds)
c(train_rows = nrow(chem_X_train), test_rows = nrow(chem_X_test),
predictors = ncol(chem_X_train), missing = sum(is.na(predictors_imputed)))
## train_rows test_rows predictors missing
## 132 44 56 0
I tune a single regression tree, random forest, stochastic gradient boosting, and Cubist on the same folds. I also refit the Homework 8 elastic net for the importance comparison, using its original tuning grid and seed. The forest is fit with the same settings and seed as the Homework 8 nonlinear winner.
fit_chemical <- function(method, seed, ...) {
set.seed(seed)
train(x = chem_X_train, y = chem_y_train, method = method,
trControl = chem_ctrl, preProcess = c("center", "scale"), ...)
}
chem_models <- list(
SingleTree = fit_chemical("rpart", 870, tuneLength = 12),
RandomForest = fit_chemical("rf", 629, tuneLength = 5),
BoostedTrees = fit_chemical(
"gbm", 871, verbose = FALSE,
tuneGrid = expand.grid(n.trees = c(100, 500),
interaction.depth = c(1, 3),
shrinkage = c(0.05, 0.1),
n.minobsinnode = 10)
),
Cubist = fit_chemical(
"cubist", 872,
tuneGrid = expand.grid(committees = c(1, 10, 50),
neighbors = c(0, 5, 9))
),
ElasticNet = fit_chemical(
"enet", 626,
tuneGrid = expand.grid(lambda = c(0.001, 0.01, 0.1),
fraction = seq(0.05, 1, length = 10))
)
)
tree_names <- c("SingleTree", "RandomForest", "BoostedTrees", "Cubist")
chemical_results <- bind_rows(lapply(names(chem_models), function(nm) {
model <- chem_models[[nm]]
cv <- model$results[which.min(model$results$RMSE), ]
test <- postResample(predict(model, newdata = chem_X_test), chem_y_test)
data.frame(Model = nm, CV_RMSE = cv$RMSE, CV_R2 = cv$Rsquared,
Test_RMSE = unname(test["RMSE"]), Test_R2 = unname(test["Rsquared"]))
})) |>
arrange(CV_RMSE)
chemical_results |> mutate(across(where(is.numeric), ~ round(.x, 4)))
## Model CV_RMSE CV_R2 Test_RMSE Test_R2
## 1 Cubist 0.9614 0.7253 1.1537 0.6908
## 2 BoostedTrees 0.9822 0.7208 1.2726 0.6357
## 3 RandomForest 1.0521 0.6932 1.4113 0.5657
## 4 ElasticNet 1.0662 0.6602 1.3191 0.5957
## 5 SingleTree 1.3710 0.4568 1.7158 0.3731
tree_results <- filter(chemical_results, Model %in% tree_names)
best_tree_name <- tree_results$Model[which.min(tree_results$CV_RMSE)]
best_test_tree <- tree_results$Model[which.min(tree_results$Test_RMSE)]
best_tree <- chem_models[[best_tree_name]]
best_tree$bestTune
## committees neighbors
## 8 50 5
The CV-selected tree-based model is Cubist, with CV RMSE 0.961 and test RMSE 1.154. The lowest observed test RMSE belongs to Cubist. I use CV, not the test ranking, to choose the model; the test set is small and should not be used to retune it. R-squared is caret’s squared correlation, not necessarily one minus the ratio of squared errors to total variation.
Cubist clearly beats the elastic net on both measures: CV RMSE 0.961 versus 1.066, and test RMSE 1.154 versus 1.319. This differs from Homework 8, where the leading linear and nonlinear models were separated by only a small margin on CV RMSE. Among tree models the single tree is by far the weakest (CV RMSE 1.371).
For the selected tree model I use caret’s model-specific importance. For the prior linear winner, elastic net, I rank absolute standardized coefficients. The Homework 8 nonlinear CV winner was random forest, so I refit it with its prior settings. Ranks and shared predictors are comparable; the numerical scales are not.
model_importance <- function(model) {
imp <- caret::varImp(model, scale = FALSE)$importance
scores <- imp[[1]]
names(scores) <- rownames(imp)
sort(scores[is.finite(scores)], decreasing = TRUE)
}
enet_coef <- predict(chem_models$ElasticNet$finalModel,
s = chem_models$ElasticNet$bestTune$fraction,
type = "coefficients", mode = "fraction")$coefficients
enet_imp <- sort(abs(enet_coef[names(enet_coef) != "(Intercept)"]),
decreasing = TRUE)
top_tree <- head(names(model_importance(best_tree)), 10)
top_linear <- head(names(enet_imp), 10)
top_nonlinear <- head(names(model_importance(chem_models$RandomForest)), 10)
importance_comparison <- full_join(
data.frame(Predictor = top_tree, Tree_Rank = seq_along(top_tree)),
data.frame(Predictor = top_linear, Linear_Rank = seq_along(top_linear)),
by = "Predictor"
) |>
full_join(data.frame(Predictor = top_nonlinear,
Prior_Nonlinear_Rank = seq_along(top_nonlinear)),
by = "Predictor") |>
arrange(coalesce(Tree_Rank, Inf), coalesce(Linear_Rank, Inf))
importance_comparison
## Predictor Tree_Rank Linear_Rank Prior_Nonlinear_Rank
## 1 ManufacturingProcess32 1 1 1
## 2 BiologicalMaterial06 2 3 4
## 3 ManufacturingProcess09 3 2 10
## 4 ManufacturingProcess02 4 NA NA
## 5 ManufacturingProcess17 5 4 NA
## 6 ManufacturingProcess39 6 8 NA
## 7 ManufacturingProcess06 7 5 NA
## 8 BiologicalMaterial02 8 NA NA
## 9 ManufacturingProcess01 9 7 NA
## 10 ManufacturingProcess04 10 NA NA
## 11 ManufacturingProcess34 NA 6 NA
## 12 ManufacturingProcess37 NA 9 NA
## 13 ManufacturingProcess45 NA 10 NA
## 14 ManufacturingProcess31 NA NA 2
## 15 BiologicalMaterial12 NA NA 3
## 16 BiologicalMaterial03 NA NA 5
## 17 ManufacturingProcess13 NA NA 6
## 18 ManufacturingProcess11 NA NA 7
## 19 BiologicalMaterial11 NA NA 8
## 20 BiologicalMaterial04 NA NA 9
tree_bio <- sum(grepl("^Biological", top_tree))
tree_process <- sum(grepl("^Manufacturing", top_tree))
shared_linear <- intersect(top_tree, top_linear)
shared_nonlinear <- intersect(top_tree, top_nonlinear)
data.frame(Group = c("Biological", "Manufacturing"),
Top_ten_count = c(tree_bio, tree_process))
## Group Top_ten_count
## 1 Biological 2
## 2 Manufacturing 8
The selected tree model’s top ten contain 2 biological and 8 manufacturing predictors. They overlap the elastic-net top ten in 7 predictors (ManufacturingProcess32, BiologicalMaterial06, ManufacturingProcess09, ManufacturingProcess17, ManufacturingProcess39, ManufacturingProcess06, ManufacturingProcess01) and the previous random-forest top ten in 3 predictors (ManufacturingProcess32, BiologicalMaterial06, ManufacturingProcess09). Rankings reflect the fitted models and correlated predictors can share or substitute for importance; they do not establish causal effects. Process variables dominate the top ten, 8 to 2. In particular, ManufacturingProcess32 ranks first in all three lists. Cubist’s usage-based importance describes rules and linear models, not the additional contribution of its nearest-neighbor adjustment.
The best single tree is the CV-tuned
rpart model, whether or not an ensemble wins part (a).
Split thresholds below use the centered/scaled predictors produced by
the previous preprocessing, not their original measurement units. The
distributions show training yields, not test-set performance.
single_tree <- chem_models$SingleTree$finalModel
plot(single_tree, uniform = TRUE, margin = 0.1)
text(single_tree, use.n = TRUE, cex = 0.7)
single_tree
## n= 132
##
## node), split, n, deviance, yval
## * denotes terminal node
##
## 1) root 132 407.16090 40.13909
## 2) ManufacturingProcess32< 0.1640716 75 123.26570 39.15813
## 4) BiologicalMaterial12< -0.605324 29 27.50061 38.20172 *
## 5) BiologicalMaterial12>=-0.605324 46 52.51485 39.76109
## 10) ManufacturingProcess17>=-0.6165343 32 14.84240 39.29469 *
## 11) ManufacturingProcess17< -0.6165343 14 14.80089 40.82714 *
## 3) ManufacturingProcess32>=0.1640716 57 116.76270 41.42982
## 6) ManufacturingProcess06< -0.930275 9 12.55120 39.87000 *
## 7) ManufacturingProcess06>=-0.930275 48 78.20825 41.72229
## 14) ManufacturingProcess32< 0.3557304 13 13.14260 40.76000 *
## 15) ManufacturingProcess32>=0.3557304 35 48.55630 42.07971 *
node_ids <- as.integer(rownames(single_tree$frame))[single_tree$where]
node_data <- data.frame(Node = factor(node_ids), Yield = chem_y_train)
node_summary <- node_data |>
group_by(Node) |>
summarise(Runs = n(), Mean = mean(Yield), Median = median(Yield),
Min = min(Yield), Max = max(Yield), .groups = "drop")
node_summary |> mutate(across(where(is.numeric), ~ round(.x, 2)))
## # A tibble: 6 × 6
## Node Runs Mean Median Min Max
## <fct> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 4 29 38.2 38.0 35.2 40.1
## 2 6 9 39.9 39.9 36.8 41
## 3 10 32 39.3 39.3 37.9 40.4
## 4 11 14 40.8 41.1 38.5 42.3
## 5 14 13 40.8 40.9 39.0 42.3
## 6 15 35 42.1 42.2 39.4 44.4
ggplot(node_data, aes(x = Node, y = Yield)) +
geom_boxplot(fill = "lightblue", outlier.shape = NA) +
geom_jitter(width = 0.12, alpha = 0.5, colour = "steelblue") +
labs(title = "Training yield distributions in the single-tree leaves",
x = "Terminal node number", y = "Yield (%)")
tree_split_vars <- unique(as.character(single_tree$frame$var))
tree_split_vars <- setdiff(tree_split_vars, "<leaf>")
tree_split_vars
## [1] "ManufacturingProcess32" "BiologicalMaterial12" "ManufacturingProcess17"
## [4] "ManufacturingProcess06"
The tree splits on ManufacturingProcess32, BiologicalMaterial12, ManufacturingProcess17, ManufacturingProcess06. Its branches show which measurements define groups and the leaf plots show the spread within those groups, information that a ranked importance list alone does not display. The printed tree numbers match the boxplot node labels. ManufacturingProcess32 forms the first split; on its lower side BiologicalMaterial12 separates a lower-yield group, node 4 (mean 38.2%). On the higher side, node 15 has mean yield 42.08%. Mean yield runs from 38.2% to 42.1% across the leaves. ManufacturingProcess32 is used twice (the root and again on the high side), whereas BiologicalMaterial12 matters only on the low-ManufacturingProcess32 branch. The node distributions overlap, so the splits do not perfectly separate yields. Differences between leaves are descriptive, may reflect correlated measurements, and do not prove that changing a process variable will cause a yield change. Small terminal groups and the single train/test split also limit the stability of the thresholds.