RPubs URL: https://rpubs.com/umaisabdullah/Data624_Homework9

Objective

I investigate how correlated predictors and predictor granularity affect tree-based importance, explain the boosting settings in Figure 8.24, and return to the chemical manufacturing data. For the chemical data I use the same split, training-set KNN imputation, and preprocessing as in Homework 8. The test set is reserved for a final comparison; cross-validation selects the models.

Exercise 8.1: Correlated Predictors and Importance

Question (Kuhn & Johnson 8.1). Recreate the simulated data from Exercise 7.2: library(mlbench); set.seed(200); simulated <- mlbench.friedman1(200, sd = 1).

  1. Fit a random forest model to all of the predictors, then estimate the variable importance scores. Did the random forest model significantly use the uninformative predictors (V6 – V10)?

  2. Now add an additional predictor that is highly correlated with one of the informative predictors (simulated$duplicate1 <- simulated$V1 + rnorm(200) * .1). Fit another random forest model to these data. Did the importance score for V1 change? What happens when you add another predictor that is also highly correlated with V1?

  3. Use the cforest function in the party package to fit a random forest model using conditional inference trees. The party::varimp function can calculate predictor importance; its conditional argument toggles between the traditional importance measure and the modified version described in Strobl et al. (2007). Do these importances show the same pattern as the traditional random forest model?

  4. Repeat this process with different tree models, such as boosted trees and Cubist. Does the same pattern occur?

The Friedman response depends on V1–V5; V6–V10 are independent noise predictors. I reproduce the requested 200 observations and add two noisy copies of V1. Each model is refit with zero, one, and two copies. Importance is model-specific and values from different model families are not on the same scale.

set.seed(200)
simulated <- mlbench.friedman1(200, sd = 1)
simulated <- as.data.frame(cbind(simulated$x, simulated$y))
names(simulated)[ncol(simulated)] <- "y"
set.seed(201)
simulated$duplicate1 <- simulated$V1 + rnorm(200) * 0.1
simulated$duplicate2 <- simulated$V1 + rnorm(200) * 0.1
round(c(cor_duplicate1 = cor(simulated$V1, simulated$duplicate1),
        cor_duplicate2 = cor(simulated$V1, simulated$duplicate2)), 3)
## cor_duplicate1 cor_duplicate2 
##          0.936          0.945
data_sets <- list(
  Original = simulated[, c(paste0("V", 1:10), "y")],
  One_copy = simulated[, c(paste0("V", 1:10), "duplicate1", "y")],
  Two_copies = simulated[, c(paste0("V", 1:10), "duplicate1",
                              "duplicate2", "y")]
)

(a) and (b) Random forest

I use the out-of-bag permutation increase in MSE, rather than the impurity-decrease score. This measures the fitted forest’s reliance on a predictor, not a significance test. Correlated variables can substitute for each other, making their individual permutation scores smaller even when the underlying signal remains important.

rf_models <- lapply(seq_along(data_sets), function(i) {
  set.seed(810 + i)
  randomForest(y ~ ., data = data_sets[[i]], importance = TRUE, ntree = 1000)
})
names(rf_models) <- names(data_sets)
rf_importance <- bind_rows(lapply(names(rf_models), function(nm) {
  imp <- caret::varImp(rf_models[[nm]], scale = FALSE)
  data.frame(Scenario = nm, Predictor = rownames(imp), Importance = imp[[1]])
}))
rf_importance |>
  pivot_wider(names_from = Scenario, values_from = Importance) |>
  mutate(across(where(is.numeric), ~ round(.x, 3))) |>
  arrange(desc(Original))
## # A tibble: 12 × 4
##    Predictor  Original One_copy Two_copies
##    <chr>         <dbl>    <dbl>      <dbl>
##  1 V1            8.81     6.73       5.27 
##  2 V4            7.42     7.33       7.27 
##  3 V2            6.51     6.28       6.28 
##  4 V5            2.13     2.00       2.10 
##  5 V3            0.767    0.561      0.456
##  6 V6            0.105    0.003      0.158
##  7 V7            0.086    0.041     -0.063
##  8 V9           -0.057   -0.055     -0.022
##  9 V10          -0.092   -0.041     -0.057
## 10 V8           -0.113   -0.142     -0.051
## 11 duplicate1   NA        2.85       1.70 
## 12 duplicate2   NA       NA          3.58

Noise predictors can have small positive or negative permutation scores by chance. Their magnitudes relative to the known signal variables are more informative here than the mere fact that a score is nonzero. The scores are not p-values, so they cannot establish formal statistical significance. (a) No noise variable has importance close to the true signal variables: the largest original noise score is about 0.11, versus 0.77 for the weakest signal variable, V3.

(b) V1’s importance falls from 8.81 to 6.73 with one copy and 5.27 with two. The copies obtain positive scores, consistent with the forest sharing reliance on the correlated V1 signal. Changes also include forest-sampling variation.

(c) Conditional inference forests

I fit unbiased conditional-inference forests with 300 trees and the same three data sets. The ordinary score permutes a variable without conditioning. The conditional score permutes within groups defined by associated predictors, asking what information the variable adds beyond its correlated partners. These are out-of-bag MSE increases; they can also be noisy and are not significance tests.

cf_importance <- bind_rows(lapply(seq_along(data_sets), function(i) {
  set.seed(820 + i)
  fit <- cforest(y ~ ., data = data_sets[[i]],
                 controls = cforest_unbiased(ntree = 300, mtry = 3))
  set.seed(830 + i)
  ordinary <- party::varimp(fit, conditional = FALSE)
  set.seed(840 + i)
  conditional <- party::varimp(fit, conditional = TRUE)
  bind_rows(
    data.frame(Scenario = names(data_sets)[i], Measure = "Ordinary",
               Predictor = names(ordinary), Importance = as.numeric(ordinary)),
    data.frame(Scenario = names(data_sets)[i], Measure = "Conditional",
               Predictor = names(conditional), Importance = as.numeric(conditional))
  )
}))
cf_importance |>
  filter(Predictor %in% c("V1", "duplicate1", "duplicate2", "V4", "V6")) |>
  pivot_wider(names_from = Measure, values_from = Importance) |>
  mutate(across(where(is.numeric), ~ round(.x, 3))) |>
  as.data.frame()
##      Scenario  Predictor Ordinary Conditional
## 1    Original         V1    6.963       4.300
## 2    Original         V4    6.432       4.874
## 3    Original         V6   -0.002       0.020
## 4    One_copy         V1    5.532       2.198
## 5    One_copy         V4    6.428       5.076
## 6    One_copy         V6    0.046       0.059
## 7    One_copy duplicate1    2.993       1.302
## 8  Two_copies         V1    4.980       1.863
## 9  Two_copies         V4    5.635       4.386
## 10 Two_copies         V6   -0.035       0.025
## 11 Two_copies duplicate1    2.290       0.666
## 12 Two_copies duplicate2    2.393       0.963

The ordinary conditional-forest score for V1 also decreases as copies are added: 6.96, 5.53, and 4.98. The conditional score falls as well, from 4.3 to 1.86. Thus the broad pattern is similar to random forest, but the conditional measure assigns less unique importance to the correlated copies. V4 stays important and the illustrative noise variable V6 remains near zero.

(d) Boosted trees and Cubist

I use the same tuning settings across the three scenarios so that the comparison focuses on adding correlated predictors. The boosted-tree score is the reduction in loss attributed to splits; Cubist’s score averages the percentages of rules and linear models using each variable. Neither is the same quantity as a permutation score, and Cubist’s rule/model usage does not measure a coefficient’s effect size.

other_importance <- bind_rows(lapply(seq_along(data_sets), function(i) {
  dat <- data_sets[[i]]
  set.seed(850 + i)
  boosted <- gbm(y ~ ., data = dat, distribution = "gaussian",
                 n.trees = 1000, interaction.depth = 3, shrinkage = 0.03,
                 bag.fraction = 0.7, n.minobsinnode = 10, verbose = FALSE)
  boost_imp <- gbm::relative.influence(boosted, n.trees = 1000, scale. = FALSE)
  set.seed(860 + i)
  cubist_fit <- cubist(x = dat[, setdiff(names(dat), "y")],
                       y = dat$y, committees = 50)
  cubist_imp <- caret::varImp(cubist_fit, scale = FALSE)
  bind_rows(
    data.frame(Model = "Boosted", Scenario = names(data_sets)[i],
               Predictor = names(boost_imp), Importance = as.numeric(boost_imp)),
    data.frame(Model = "Cubist", Scenario = names(data_sets)[i],
               Predictor = rownames(cubist_imp), Importance = cubist_imp[[1]])
  )
}))
other_importance |>
  filter(Predictor %in% c("V1", "duplicate1", "duplicate2", "V4", "V6")) |>
  pivot_wider(names_from = Scenario, values_from = Importance) |>
  mutate(across(where(is.numeric), ~ round(.x, 2)))
## # A tibble: 10 × 5
##    Model   Predictor  Original One_copy Two_copies
##    <chr>   <chr>         <dbl>    <dbl>      <dbl>
##  1 Boosted V1          17986.    17501.    11655. 
##  2 Boosted V4          18990.    18700.    19305. 
##  3 Boosted V6            727.      677.      525. 
##  4 Cubist  V1             71        71        70  
##  5 Cubist  V4             48.5      48        48  
##  6 Cubist  V6              0         0         8.5
##  7 Boosted duplicate1     NA      2317.     1833. 
##  8 Cubist  duplicate1     NA         0         0  
##  9 Boosted duplicate2     NA        NA      5723. 
## 10 Cubist  duplicate2     NA        NA         6.5

The copies do not create independent new signal. A model may distribute credit among them, but the exact ranking and even the direction of a change can depend on the algorithm, random sampling, and importance definition. The conditional score answers a different question from the ordinary score, so identical numerical patterns should not be expected. Here boosted-tree importance for V1 decreases as copies are added, particularly with two copies. Cubist is different: V1’s usage score stays near 70 while the copies have low usage, so the forest’s pattern is not universal.

Exercise 8.2: Tree Bias and Granularity

Question (Kuhn & Johnson 8.2). Use a simulation to show tree bias with different granularities.

I simulate six independent, equally uninformative predictors, with 2, 5, 10, 20, 50, and 100 distinct ordered values. The response is independent standard normal noise. For each of 500 simulations, I fit a one-split regression tree and record the selected variable. If the selection were unbiased, each would be selected about one-sixth of the time. More distinct values provide more candidate split points and more chances to find an accidental reduction in training error.

set.seed(8200)
granularities <- c(2, 5, 10, 20, 50, 100)
selected <- replicate(500, {
  x <- as.data.frame(lapply(granularities, function(k) sample.int(k, 200, TRUE)))
  names(x) <- paste0("Levels", granularities)
  x$y <- rnorm(200)
  tree <- rpart(y ~ ., data = x, method = "anova",
                control = rpart.control(maxdepth = 1, cp = 0, minbucket = 10,
                                        xval = 0))
  as.character(tree$frame$var[1])
})
granularity_results <- data.frame(
  Levels = granularities,
  Selections = as.integer(table(factor(selected, levels = paste0("Levels", granularities))))
) |>
  mutate(Percent = 100 * Selections / length(selected))
granularity_results
##   Levels Selections Percent
## 1      2         14     2.8
## 2      5         50    10.0
## 3     10         62    12.4
## 4     20         95    19.0
## 5     50        111    22.2
## 6    100        168    33.6
ggplot(granularity_results, aes(x = factor(Levels), y = Percent)) +
  geom_col(fill = "steelblue") +
  geom_hline(yintercept = 100 / 6, linetype = "dashed", colour = "darkred") +
  labs(title = "Root-split selection under a completely null signal",
       subtitle = "Dashed line: equal selection probability",
       x = "Number of distinct values", y = "Selected in simulations (%)")

This is selection bias in training splits, not evidence that the high-granularity variable predicts new responses. The simulation treats the levels as ordered numeric values; nominal factors with many possible partitions can show related but different degrees of bias. The 2-level variable is selected in 2.8% of simulations, compared with 33.6% for the 100-level variable, despite their identical lack of predictive signal.

Exercise 8.3: Boosting Parameters

Question (Kuhn & Johnson 8.3). In stochastic gradient boosting the bagging fraction and learning rate will govern the construction of the trees as they are guided by the gradient. Although the optimal values of these parameters should be obtained through the tuning process, it is helpful to understand how the magnitudes of these parameters affect magnitudes of variable importance. Figure 8.24 provides the variable importance plots for boosting using two extreme values for the bagging fraction (0.1 and 0.9) and the learning rate (0.1 and 0.9) for the solubility data. The left-hand plot has both parameters set to 0.1, and the right-hand plot has both set to 0.9.

  1. Why does the model on the right focus its importance on just the first few of predictors, whereas the model on the left spreads importance across more predictors?

  2. Which model do you think would be more predictive of other samples?

  3. How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?

Figure 8.24 in the book shows gradient boosting importance on the solubility data for two settings: shrinkage (learning rate) 0.1 with bagging fraction 0.1, and shrinkage 0.9 with bagging fraction 0.9. I refit these two models on the same data (solubility in AppliedPredictiveModeling, transformed predictors) with 1000 trees, and also repeat them at interaction depths 1, 5, and 10 for part (c). Importance is gbm’s relative influence, rescaled to sum to 100% within each model so that models are comparable. Test RMSE uses the untouched solubility test set.

data(solubility)
sol_train <- cbind(solTrainXtrans, y = solTrainY)
boost_grid <- expand.grid(Setting = c("LR 0.1, bag 0.1", "LR 0.9, bag 0.9"),
                          Depth = c(1, 5, 10), stringsAsFactors = FALSE)
boost_grid$Shrinkage <- ifelse(grepl("0.1,", boost_grid$Setting), 0.1, 0.9)

boost_fits <- lapply(seq_len(nrow(boost_grid)), function(i) {
  set.seed(830 + i)
  fit <- gbm(y ~ ., data = sol_train, distribution = "gaussian",
             n.trees = 1000, interaction.depth = boost_grid$Depth[i],
             shrinkage = boost_grid$Shrinkage[i],
             bag.fraction = boost_grid$Shrinkage[i],
             n.minobsinnode = 10, verbose = FALSE)
  imp <- gbm::relative.influence(fit, n.trees = 1000, scale. = FALSE)
  imp <- sort(100 * imp / sum(imp), decreasing = TRUE)
  pred <- predict(fit, solTestXtrans, n.trees = 1000)
  list(importance = imp, test_rmse = sqrt(mean((pred - solTestY)^2)))
})

boost_summary <- bind_cols(boost_grid[, c("Setting", "Depth")],
  bind_rows(lapply(boost_fits, function(f) data.frame(
    Top_predictor_pct = f$importance[1],
    Top5_pct = sum(f$importance[1:5]),
    Top10_pct = sum(f$importance[1:10]),
    Predictors_for_80pct = which(cumsum(f$importance) >= 80)[1],
    Test_RMSE = f$test_rmse))))
rownames(boost_summary) <- NULL
boost_summary |> mutate(across(where(is.numeric), ~ round(.x, 2)))
##           Setting Depth Top_predictor_pct Top5_pct Top10_pct
## 1 LR 0.1, bag 0.1     1             17.53    46.63     57.94
## 2 LR 0.9, bag 0.9     1             36.57    71.62     82.18
## 3 LR 0.1, bag 0.1     5             11.82    38.58     54.34
## 4 LR 0.9, bag 0.9     5             37.04    78.40     86.25
## 5 LR 0.1, bag 0.1    10             14.45    38.59     53.15
## 6 LR 0.9, bag 0.9    10             36.24    78.65     84.27
##   Predictors_for_80pct Test_RMSE
## 1                   42      0.68
## 2                    9      0.69
## 3                   53      0.66
## 4                    6      0.81
## 5                   54      0.62
## 6                    6      0.84
curve_data <- bind_rows(lapply(seq_len(nrow(boost_grid)), function(i) {
  imp <- boost_fits[[i]]$importance
  data.frame(Setting = boost_grid$Setting[i], Depth = boost_grid$Depth[i],
             Rank = seq_along(imp), Importance = as.numeric(imp))
}))
ggplot(filter(curve_data, Depth == 1, Rank <= 40),
       aes(x = Rank, y = Importance, colour = Setting)) +
  geom_line(linewidth = 1) +
  labs(title = "Ranked importance, interaction depth 1",
       subtitle = "Right-hand setting piles importance on a few predictors",
       x = "Predictor rank", y = "Relative influence (%)", colour = NULL)

ggplot(filter(curve_data, Rank <= 40),
       aes(x = Rank, y = Importance, colour = factor(Depth))) +
  geom_line(linewidth = 1) +
  facet_wrap(~ Setting) +
  labs(title = "Effect of interaction depth on the importance curve",
       x = "Predictor rank", y = "Relative influence (%)", colour = "Depth")

(a) The refit shows the book’s pattern. At depth 1 the 0.9/0.9 model puts 36.6% of its importance on a single predictor and 71.6% on its top five, needing only 9 predictors to reach 80% of total importance. The 0.1/0.1 model gives its top predictor 17.5% and its top five 46.6%, needing 42 predictors to reach 80%. With a large learning rate and nearly the full data in each tree, the first trees remove most of the residual that the strongest predictors can explain and every later tree is built on almost the same data, so the same predictors keep winning the splits. With small steps and small random subsamples, the strong predictors cannot absorb everything quickly, the subsamples differ from tree to tree, and weaker predictors win splits on some trees, which spreads the importance.

(b) On the untouched test set, the two models are effectively tied at depth 1: RMSE 0.679 for 0.1/0.1 versus 0.693 for 0.9/0.9. The difference appears at greater depth, where the 0.1/0.1 model is clearly better: 0.66 versus 0.806 at depth 5, and 0.621 versus 0.84 at depth 10. The 0.9/0.9 model gets worse as trees deepen while the 0.1/0.1 model improves, so I would expect the left-hand model (small learning rate, small bagging fraction) to be more predictive of other samples, particularly with deeper trees. This is one data set and one seed per setting, not a full tuning exercise.

(c) The effect of interaction depth is not the same for the two models. For the 0.1/0.1 model the share held by the top five predictors goes from 46.6% (depth 1) to 38.6% (depth 5) and 38.6% (depth 10), so its importance curve becomes flatter with deeper trees. For the 0.9/0.9 model the top-five share goes from 71.6% to 78.4% and 78.6%, so its slope stays steep or becomes slightly steeper. Deeper trees therefore flatten the curve when the learning rate is small, but do not spread importance when large steps already let a few strong predictors dominate. Test RMSE for the 0.9/0.9 model worsens with depth (0.693, 0.806, 0.84), consistent with overfitting.

Exercise 8.7: Chemical Manufacturing

Question (Kuhn & Johnson 8.7). Refer to Exercises 6.3 and 7.5 which describe a chemical manufacturing process. Use the same data imputation, data splitting, and pre-processing steps as before and train several tree-based models:

  1. Which tree-based regression model gives the optimal resampling and test set performance?

  2. Which predictors are most important in the optimal tree-based regression model? Do either the biological or process variables dominate the list? How do the top 10 important predictors compare to the top 10 predictors from the optimal linear and nonlinear models?

  3. Plot the optimal single tree with the distribution of yield in the terminal nodes. Does this view of the data provide additional knowledge about the biological or process predictors and their relationship with yield?

Data preparation and resampling

As in Homework 8, I use the seed-624, 75% stratified split, fit KNN imputation on the full training partition, filter near-zero-variance variables using that partition, and center and scale within model fitting. The same seed-6240 ten folds are used for all models. Because the imputer is fitted before the folds are formed, validation folds do influence the imputation step; CV may consequently be somewhat optimistic. The held-out test data do not influence imputation or tuning.

data(ChemicalManufacturingProcess)
cmp <- ChemicalManufacturingProcess
predictors_raw <- cmp[, -1, drop = FALSE]
yield <- as.numeric(cmp[, 1])
set.seed(624)
chem_idx <- createDataPartition(yield, p = 0.75, list = FALSE)
chem_imputer <- preProcess(predictors_raw[chem_idx, , drop = FALSE],
                           method = "knnImpute")
predictors_imputed <- predict(chem_imputer, predictors_raw)
nzv_chem <- nearZeroVar(predictors_imputed[chem_idx, , drop = FALSE])
if (length(nzv_chem) > 0) {
  predictors_imputed <- predictors_imputed[, -nzv_chem, drop = FALSE]
}
chem_X_train <- predictors_imputed[chem_idx, , drop = FALSE]
chem_X_test <- predictors_imputed[-chem_idx, , drop = FALSE]
chem_y_train <- yield[chem_idx]
chem_y_test <- yield[-chem_idx]
set.seed(6240)
chem_folds <- createFolds(chem_y_train, k = 10, returnTrain = TRUE)
chem_ctrl <- trainControl(method = "cv", number = 10, index = chem_folds)
c(train_rows = nrow(chem_X_train), test_rows = nrow(chem_X_test),
  predictors = ncol(chem_X_train), missing = sum(is.na(predictors_imputed)))
## train_rows  test_rows predictors    missing 
##        132         44         56          0

(a) Compare tree models

I tune a single regression tree, random forest, stochastic gradient boosting, and Cubist on the same folds. I also refit the Homework 8 elastic net for the importance comparison, using its original tuning grid and seed. The forest is fit with the same settings and seed as the Homework 8 nonlinear winner.

fit_chemical <- function(method, seed, ...) {
  set.seed(seed)
  train(x = chem_X_train, y = chem_y_train, method = method,
        trControl = chem_ctrl, preProcess = c("center", "scale"), ...)
}
chem_models <- list(
  SingleTree = fit_chemical("rpart", 870, tuneLength = 12),
  RandomForest = fit_chemical("rf", 629, tuneLength = 5),
  BoostedTrees = fit_chemical(
    "gbm", 871, verbose = FALSE,
    tuneGrid = expand.grid(n.trees = c(100, 500),
                           interaction.depth = c(1, 3),
                           shrinkage = c(0.05, 0.1),
                           n.minobsinnode = 10)
  ),
  Cubist = fit_chemical(
    "cubist", 872,
    tuneGrid = expand.grid(committees = c(1, 10, 50),
                           neighbors = c(0, 5, 9))
  ),
  ElasticNet = fit_chemical(
    "enet", 626,
    tuneGrid = expand.grid(lambda = c(0.001, 0.01, 0.1),
                           fraction = seq(0.05, 1, length = 10))
  )
)
tree_names <- c("SingleTree", "RandomForest", "BoostedTrees", "Cubist")
chemical_results <- bind_rows(lapply(names(chem_models), function(nm) {
  model <- chem_models[[nm]]
  cv <- model$results[which.min(model$results$RMSE), ]
  test <- postResample(predict(model, newdata = chem_X_test), chem_y_test)
  data.frame(Model = nm, CV_RMSE = cv$RMSE, CV_R2 = cv$Rsquared,
             Test_RMSE = unname(test["RMSE"]), Test_R2 = unname(test["Rsquared"]))
})) |>
  arrange(CV_RMSE)
chemical_results |> mutate(across(where(is.numeric), ~ round(.x, 4)))
##          Model CV_RMSE  CV_R2 Test_RMSE Test_R2
## 1       Cubist  0.9614 0.7253    1.1537  0.6908
## 2 BoostedTrees  0.9822 0.7208    1.2726  0.6357
## 3 RandomForest  1.0521 0.6932    1.4113  0.5657
## 4   ElasticNet  1.0662 0.6602    1.3191  0.5957
## 5   SingleTree  1.3710 0.4568    1.7158  0.3731
tree_results <- filter(chemical_results, Model %in% tree_names)
best_tree_name <- tree_results$Model[which.min(tree_results$CV_RMSE)]
best_test_tree <- tree_results$Model[which.min(tree_results$Test_RMSE)]
best_tree <- chem_models[[best_tree_name]]
best_tree$bestTune
##   committees neighbors
## 8         50         5

The CV-selected tree-based model is Cubist, with CV RMSE 0.961 and test RMSE 1.154. The lowest observed test RMSE belongs to Cubist. I use CV, not the test ranking, to choose the model; the test set is small and should not be used to retune it. R-squared is caret’s squared correlation, not necessarily one minus the ratio of squared errors to total variation.

Cubist clearly beats the elastic net on both measures: CV RMSE 0.961 versus 1.066, and test RMSE 1.154 versus 1.319. This differs from Homework 8, where the leading linear and nonlinear models were separated by only a small margin on CV RMSE. Among tree models the single tree is by far the weakest (CV RMSE 1.371).

(b) Compare the top predictors

For the selected tree model I use caret’s model-specific importance. For the prior linear winner, elastic net, I rank absolute standardized coefficients. The Homework 8 nonlinear CV winner was random forest, so I refit it with its prior settings. Ranks and shared predictors are comparable; the numerical scales are not.

model_importance <- function(model) {
  imp <- caret::varImp(model, scale = FALSE)$importance
  scores <- imp[[1]]
  names(scores) <- rownames(imp)
  sort(scores[is.finite(scores)], decreasing = TRUE)
}
enet_coef <- predict(chem_models$ElasticNet$finalModel,
                     s = chem_models$ElasticNet$bestTune$fraction,
                     type = "coefficients", mode = "fraction")$coefficients
enet_imp <- sort(abs(enet_coef[names(enet_coef) != "(Intercept)"]),
                 decreasing = TRUE)
top_tree <- head(names(model_importance(best_tree)), 10)
top_linear <- head(names(enet_imp), 10)
top_nonlinear <- head(names(model_importance(chem_models$RandomForest)), 10)
importance_comparison <- full_join(
  data.frame(Predictor = top_tree, Tree_Rank = seq_along(top_tree)),
  data.frame(Predictor = top_linear, Linear_Rank = seq_along(top_linear)),
  by = "Predictor"
) |>
  full_join(data.frame(Predictor = top_nonlinear,
                       Prior_Nonlinear_Rank = seq_along(top_nonlinear)),
            by = "Predictor") |>
  arrange(coalesce(Tree_Rank, Inf), coalesce(Linear_Rank, Inf))
importance_comparison
##                 Predictor Tree_Rank Linear_Rank Prior_Nonlinear_Rank
## 1  ManufacturingProcess32         1           1                    1
## 2    BiologicalMaterial06         2           3                    4
## 3  ManufacturingProcess09         3           2                   10
## 4  ManufacturingProcess02         4          NA                   NA
## 5  ManufacturingProcess17         5           4                   NA
## 6  ManufacturingProcess39         6           8                   NA
## 7  ManufacturingProcess06         7           5                   NA
## 8    BiologicalMaterial02         8          NA                   NA
## 9  ManufacturingProcess01         9           7                   NA
## 10 ManufacturingProcess04        10          NA                   NA
## 11 ManufacturingProcess34        NA           6                   NA
## 12 ManufacturingProcess37        NA           9                   NA
## 13 ManufacturingProcess45        NA          10                   NA
## 14 ManufacturingProcess31        NA          NA                    2
## 15   BiologicalMaterial12        NA          NA                    3
## 16   BiologicalMaterial03        NA          NA                    5
## 17 ManufacturingProcess13        NA          NA                    6
## 18 ManufacturingProcess11        NA          NA                    7
## 19   BiologicalMaterial11        NA          NA                    8
## 20   BiologicalMaterial04        NA          NA                    9
tree_bio <- sum(grepl("^Biological", top_tree))
tree_process <- sum(grepl("^Manufacturing", top_tree))
shared_linear <- intersect(top_tree, top_linear)
shared_nonlinear <- intersect(top_tree, top_nonlinear)
data.frame(Group = c("Biological", "Manufacturing"),
           Top_ten_count = c(tree_bio, tree_process))
##           Group Top_ten_count
## 1    Biological             2
## 2 Manufacturing             8

The selected tree model’s top ten contain 2 biological and 8 manufacturing predictors. They overlap the elastic-net top ten in 7 predictors (ManufacturingProcess32, BiologicalMaterial06, ManufacturingProcess09, ManufacturingProcess17, ManufacturingProcess39, ManufacturingProcess06, ManufacturingProcess01) and the previous random-forest top ten in 3 predictors (ManufacturingProcess32, BiologicalMaterial06, ManufacturingProcess09). Rankings reflect the fitted models and correlated predictors can share or substitute for importance; they do not establish causal effects. Process variables dominate the top ten, 8 to 2. In particular, ManufacturingProcess32 ranks first in all three lists. Cubist’s usage-based importance describes rules and linear models, not the additional contribution of its nearest-neighbor adjustment.

(c) Best single tree and terminal-node yield

The best single tree is the CV-tuned rpart model, whether or not an ensemble wins part (a). Split thresholds below use the centered/scaled predictors produced by the previous preprocessing, not their original measurement units. The distributions show training yields, not test-set performance.

single_tree <- chem_models$SingleTree$finalModel
plot(single_tree, uniform = TRUE, margin = 0.1)
text(single_tree, use.n = TRUE, cex = 0.7)

single_tree
## n= 132 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
##  1) root 132 407.16090 40.13909  
##    2) ManufacturingProcess32< 0.1640716 75 123.26570 39.15813  
##      4) BiologicalMaterial12< -0.605324 29  27.50061 38.20172 *
##      5) BiologicalMaterial12>=-0.605324 46  52.51485 39.76109  
##       10) ManufacturingProcess17>=-0.6165343 32  14.84240 39.29469 *
##       11) ManufacturingProcess17< -0.6165343 14  14.80089 40.82714 *
##    3) ManufacturingProcess32>=0.1640716 57 116.76270 41.42982  
##      6) ManufacturingProcess06< -0.930275 9  12.55120 39.87000 *
##      7) ManufacturingProcess06>=-0.930275 48  78.20825 41.72229  
##       14) ManufacturingProcess32< 0.3557304 13  13.14260 40.76000 *
##       15) ManufacturingProcess32>=0.3557304 35  48.55630 42.07971 *
node_ids <- as.integer(rownames(single_tree$frame))[single_tree$where]
node_data <- data.frame(Node = factor(node_ids), Yield = chem_y_train)
node_summary <- node_data |>
  group_by(Node) |>
  summarise(Runs = n(), Mean = mean(Yield), Median = median(Yield),
            Min = min(Yield), Max = max(Yield), .groups = "drop")
node_summary |> mutate(across(where(is.numeric), ~ round(.x, 2)))
## # A tibble: 6 × 6
##   Node   Runs  Mean Median   Min   Max
##   <fct> <dbl> <dbl>  <dbl> <dbl> <dbl>
## 1 4        29  38.2   38.0  35.2  40.1
## 2 6         9  39.9   39.9  36.8  41  
## 3 10       32  39.3   39.3  37.9  40.4
## 4 11       14  40.8   41.1  38.5  42.3
## 5 14       13  40.8   40.9  39.0  42.3
## 6 15       35  42.1   42.2  39.4  44.4
ggplot(node_data, aes(x = Node, y = Yield)) +
  geom_boxplot(fill = "lightblue", outlier.shape = NA) +
  geom_jitter(width = 0.12, alpha = 0.5, colour = "steelblue") +
  labs(title = "Training yield distributions in the single-tree leaves",
       x = "Terminal node number", y = "Yield (%)")

tree_split_vars <- unique(as.character(single_tree$frame$var))
tree_split_vars <- setdiff(tree_split_vars, "<leaf>")
tree_split_vars
## [1] "ManufacturingProcess32" "BiologicalMaterial12"   "ManufacturingProcess17"
## [4] "ManufacturingProcess06"

The tree splits on ManufacturingProcess32, BiologicalMaterial12, ManufacturingProcess17, ManufacturingProcess06. Its branches show which measurements define groups and the leaf plots show the spread within those groups, information that a ranked importance list alone does not display. The printed tree numbers match the boxplot node labels. ManufacturingProcess32 forms the first split; on its lower side BiologicalMaterial12 separates a lower-yield group, node 4 (mean 38.2%). On the higher side, node 15 has mean yield 42.08%. Mean yield runs from 38.2% to 42.1% across the leaves. ManufacturingProcess32 is used twice (the root and again on the high side), whereas BiologicalMaterial12 matters only on the low-ManufacturingProcess32 branch. The node distributions overlap, so the splits do not perfectly separate yields. Differences between leaves are descriptive, may reflect correlated measurements, and do not prove that changing a process variable will cause a yield change. Small terminal groups and the single train/test split also limit the stability of the thresholds.