Model selection and Examination Tree-Based Models

Author

Guibril Ramde

Exercise 8

##Exercise 8.1

#install.packages('')
library(mlbench)
Warning: package 'mlbench' was built under R version 4.5.2
set.seed(200)
simulated <- mlbench.friedman1(200, sd = 1)
simulated <- cbind(simulated$x, simulated$y)
simulated <- as.data.frame(simulated)
colnames(simulated)[ncol(simulated)] <- "y"

a. Fit a random forest model to all of the predictors, then estimate the variable importance scores:

library(randomForest)
randomForest 4.7-1.2
Type rfNews() to see new features/changes/bug fixes.
library(caret)
Loading required package: ggplot2
Warning: package 'ggplot2' was built under R version 4.5.2

Attaching package: 'ggplot2'
The following object is masked from 'package:randomForest':

    margin
Loading required package: lattice
model1 <- randomForest(y ~ ., data = simulated, importance = TRUE, ntree= 1000)
model1

Call:
 randomForest(formula = y ~ ., data = simulated, importance = TRUE,      ntree = 1000) 
               Type of random forest: regression
                     Number of trees: 1000
No. of variables tried at each split: 3

          Mean of squared residuals: 6.514982
                    % Var explained: 73.28
rfImp1 <- varImp(model1, scale = FALSE)
rfImp1
        Overall
V1   8.86329776
V2   6.72851763
V3   0.84145353
V4   7.60284159
V5   2.26864193
V6   0.11268425
V7   0.07374772
V8  -0.07210708
V9  -0.06913906
V10 -0.10577619
plot(rfImp1)

Did the random forest model significantly use the uninformative predictors (V6 – V10)?

##Answer:

The random forest model used all ten predictors (V1–V10) to predict the response variable y. However, the variable importance scores for the uninformative predictors (V6–V10) were relatively low, with some values near zero or negative. This suggests that these predictors contributed little to the model’s predictive performance compared with the informative predictors (V1–V5). Therefore, the random forest did not rely heavily on the uninformative predictors.

b. Now add an additional predictor that is highly correlated with one of the informative predictors. For example:

library(randomForest)
library(caret)

simulated2 <- simulated

set.seed(201)
simulated2$duplicate1 <- simulated2$V1 + rnorm(200) * 0.1

# Check correlation
cor(simulated2$duplicate1, simulated2$V1)
[1] 0.9358765
# Fit a new random forest
set.seed(200)

model2 <- randomForest(
  y ~ .,
  data = simulated2,
  importance = TRUE,
  ntree = 1000
)

model2

Call:
 randomForest(formula = y ~ ., data = simulated2, importance = TRUE,      ntree = 1000) 
               Type of random forest: regression
                     Number of trees: 1000
No. of variables tried at each split: 3

          Mean of squared residuals: 7.227598
                    % Var explained: 70.36
# Variable importance
rfImp2 <- varImp(model2, scale = FALSE)

rfImp2
               Overall
V1          6.66784016
V2          5.94883839
V3          0.68183118
V4          7.05948117
V5          2.00197462
V6          0.17103674
V7          0.02785474
V8         -0.13644481
V9         -0.06659097
V10        -0.12343994
duplicate1  2.90167899
plot(rfImp2)

Did the importance score for V1 change? What happens when you add another predictor that is also highly correlated with V1?

# Create a third dataset
simulated3 <- simulated2

# Add another correlated predictor
set.seed(202)

simulated3$duplicate2 <- simulated3$V1 + rnorm(200) * 0.1

# Check correlations
cor(simulated3[, c("V1", "duplicate1", "duplicate2")])
                  V1 duplicate1 duplicate2
V1         1.0000000  0.9358765  0.9429382
duplicate1 0.9358765  1.0000000  0.8725834
duplicate2 0.9429382  0.8725834  1.0000000
# Fit third random forest
set.seed(200)

model3 <- randomForest(
  y ~ .,
  data = simulated3,
  importance = TRUE,
  ntree = 1000
)

# Variable importance
rfImp3 <- varImp(model3, scale = FALSE)

rfImp3
                Overall
V1          4.895569743
V2          6.300210411
V3          0.440349997
V4          6.949353497
V5          2.062192732
V6          0.123605542
V7          0.031020095
V8         -0.068299597
V9         -0.009143988
V10         0.051067079
duplicate1  1.471278576
duplicate2  3.407370203
plot(rfImp3)

# Compare the importance of V1 across all three models
data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  V1_Importance = c(
    rfImp1["V1", 1],
    rfImp2["V1", 1],
    rfImp3["V1", 1]
  )
)
    Model V1_Importance
1 Model 1      8.863298
2 Model 2      6.667840
3 Model 3      4.895570

##Answer:

After adding a predictor highly correlated with V1, the variable importance score for V1 decreased from 8.6254 to 6.6678. When another highly correlated predictor was added, the importance score decreased further to 4.8956. This suggests that the random forest model distributes predictive information among correlated predictors, reducing the individual importance score of V1. Therefore, adding highly correlated predictors can affect variable importance scores even when the original predictor remains informative.

c. Use the cforest function in the party package to fit a random forest model using conditional inference trees.

library(party)
Warning: package 'party' was built under R version 4.5.2
Loading required package: grid
Loading required package: mvtnorm
Loading required package: modeltools
Warning: package 'modeltools' was built under R version 4.5.2
Loading required package: stats4
Loading required package: strucchange
Warning: package 'strucchange' was built under R version 4.5.2
Loading required package: zoo
Warning: package 'zoo' was built under R version 4.5.2

Attaching package: 'zoo'
The following objects are masked from 'package:base':

    as.Date, as.Date.numeric
Loading required package: sandwich
Warning: package 'sandwich' was built under R version 4.5.2
# Conditional inference forests
set.seed(200)
cforest1 <- cforest(y ~ ., data = simulated)

set.seed(200)
cforest2 <- cforest(y ~ ., data = simulated2)

set.seed(200)
cforest3 <- cforest(y ~ ., data = simulated3)

# Traditional permutation importance
imp1 <- varimp(cforest1, conditional = FALSE)
imp2 <- varimp(cforest2, conditional = FALSE)
imp3 <- varimp(cforest3, conditional = FALSE)

# Conditional permutation importance
cond_imp1 <- varimp(cforest1, conditional = TRUE)
cond_imp2 <- varimp(cforest2, conditional = TRUE)
cond_imp3 <- varimp(cforest3, conditional = TRUE)

# Compare V1 importance
comparison <- data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  
  RandomForest = c(
    rfImp1["V1", 1],
    rfImp2["V1", 1],
    rfImp3["V1", 1]
  ),
  
  Cforest_Traditional = c(
    imp1["V1"],
    imp2["V1"],
    imp3["V1"]
  ),
  
  Cforest_Conditional = c(
    cond_imp1["V1"],
    cond_imp2["V1"],
    cond_imp3["V1"]
  )
)

comparison
    Model RandomForest Cforest_Traditional Cforest_Conditional
1 Model 1     8.863298            8.672661            5.604587
2 Model 2     6.667840            6.730145            3.135758
3 Model 3     4.895570            4.513619            1.707996

Interpretation

No, the cforest importance scores did not show exactly the same pattern as the traditional random forest. In the traditional random forest, V1’s importance decreased consistently as correlated predictors were added. However, both the traditional and conditional importance scores from cforest initially increased after adding the first correlated predictor and then decreased after adding the second. This suggests that correlated predictors can affect variable importance differently depending on the method used. Overall, all three methods showed a decrease in V1’s importance after adding two correlated predictors.

d. repeating with different tree model Boosted tree and Cubist tree

##install.packages('gbm')
library(gbm)
Warning: package 'gbm' was built under R version 4.5.2
Loaded gbm 2.3.1
This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3
# Model 1: Original predictors
set.seed(200)
gbmModel1 <- gbm(
  y ~ .,
  data = simulated,
  distribution = "gaussian",
  n.trees = 1000,
  interaction.depth = 3,
  shrinkage = 0.01,
  verbose = FALSE
)

# Model 2: One correlated predictor
set.seed(200)
gbmModel2 <- gbm(
  y ~ .,
  data = simulated2,
  distribution = "gaussian",
  n.trees = 1000,
  interaction.depth = 3,
  shrinkage = 0.01,
  verbose = FALSE
)

# Model 3: Two correlated predictors
set.seed(200)
gbmModel3 <- gbm(
  y ~ .,
  data = simulated3,
  distribution = "gaussian",
  n.trees = 1000,
  interaction.depth = 3,
  shrinkage = 0.01,
  verbose = FALSE
)
gbmImp1 <- summary(gbmModel1, plotit = FALSE)
gbmImp2 <- summary(gbmModel2, plotit = FALSE)
gbmImp3 <- summary(gbmModel3, plotit = FALSE)

gbmImp1
    var    rel.inf
V4   V4 28.3972578
V1   V1 26.6377714
V2   V2 21.9806150
V5   V5 11.7327317
V3   V3  7.6887939
V7   V7  1.0762463
V6   V6  0.9827406
V9   V9  0.5598912
V10 V10  0.5056136
V8   V8  0.4383385
gbmImp2
                  var    rel.inf
V4                 V4 28.2530336
V1                 V1 23.8041343
V2                 V2 21.9286406
V5                 V5 11.7095916
V3                 V3  7.6446414
duplicate1 duplicate1  3.2524652
V7                 V7  1.0383308
V6                 V6  0.9291780
V9                 V9  0.5699957
V8                 V8  0.4785854
V10               V10  0.3914035
gbmImp3
                  var    rel.inf
V4                 V4 27.7005475
V2                 V2 21.4361282
V1                 V1 17.4301682
V5                 V5 11.9205610
duplicate2 duplicate2  7.7537852
V3                 V3  7.4187733
duplicate1 duplicate1  2.9885001
V7                 V7  1.0343191
V6                 V6  0.8973513
V9                 V9  0.5342906
V8                 V8  0.5179416
V10               V10  0.3676339
data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  GBM_V1_Importance = c(
    gbmImp1$rel.inf[gbmImp1$var == "V1"],
    gbmImp2$rel.inf[gbmImp2$var == "V1"],
    gbmImp3$rel.inf[gbmImp3$var == "V1"]
  )
)
    Model GBM_V1_Importance
1 Model 1          26.63777
2 Model 2          23.80413
3 Model 3          17.43017
library(Cubist)
Warning: package 'Cubist' was built under R version 4.5.2
# Model 1: Original predictors
set.seed(200)
cubistModel1 <- cubist(
  x = simulated[, setdiff(names(simulated), "y")],
  y = simulated$y,
  committees = 50
)

# Model 2: One correlated predictor
set.seed(200)
cubistModel2 <- cubist(
  x = simulated2[, setdiff(names(simulated2), "y")],
  y = simulated2$y,
  committees = 50
)

# Model 3: Two correlated predictors
set.seed(200)
cubistModel3 <- cubist(
  x = simulated3[, setdiff(names(simulated3), "y")],
  y = simulated3$y,
  committees = 50
)
cubistImp1 <- varImp(cubistModel1)
cubistImp2 <- varImp(cubistModel2)
cubistImp3 <- varImp(cubistModel3)

cubistImp1
    Overall
V1     71.0
V3     44.5
V2     57.5
V4     48.5
V5     35.0
V6      0.0
V7      0.0
V8      0.0
V9      0.0
V10     0.0
cubistImp2
           Overall
V1            71.0
V3            43.0
V2            54.5
V4            48.0
V5            35.0
V6             0.0
V7             0.0
V8             0.0
V9             0.0
V10            0.0
duplicate1     0.0
cubistImp3
           Overall
V3            45.0
V2            66.0
V1            57.5
V4            50.0
duplicate2    19.0
V5            33.0
V6            13.0
duplicate1     9.0
V7             0.0
V8             0.0
V9             0.0
V10            0.0
data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  Cubist_V1_Importance = c(
    cubistImp1["V1", 1],
    cubistImp2["V1", 1],
    cubistImp3["V1", 1]
  )
)
    Model Cubist_V1_Importance
1 Model 1                 71.0
2 Model 2                 71.0
3 Model 3                 57.5
comparison_d <- data.frame(
  Model = c("Model 1", "Model 2", "Model 3"),
  
  GBM = c(
    gbmImp1$rel.inf[gbmImp1$var == "V1"],
    gbmImp2$rel.inf[gbmImp2$var == "V1"],
    gbmImp3$rel.inf[gbmImp3$var == "V1"]
  ),
  
  Cubist = c(
    cubistImp1["V1", 1],
    cubistImp2["V1", 1],
    cubistImp3["V1", 1]
  )
)

comparison_d
    Model      GBM Cubist
1 Model 1 26.63777   71.0
2 Model 2 23.80413   71.0
3 Model 3 17.43017   57.5

Interpretation:

The boosted tree (GBM) and Cubist models did not show the same pattern as the traditional random forest. Both GBM and Cubist showed a similar pattern: the importance score of V1 increased from Model 1 to Model 2, then decreased in Model 3 after adding another highly correlated predictor. This suggests that correlated predictors affect variable importance differently depending on the modeling method used.

##Exercise 8.2

#install.packages('rpart')
library(rpart)
Warning: package 'rpart' was built under R version 4.5.2
library(dplyr)
Warning: package 'dplyr' was built under R version 4.5.2

Attaching package: 'dplyr'
The following object is masked from 'package:party':

    where
The following object is masked from 'package:randomForest':

    combine
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union
library(rpart)
#library(rpart.plot)

set.seed(200)

n <- 200

simulated_tree <- data.frame(
  X1 = sample(1:2, n, replace = TRUE),
  X2 = sample(1:5, n, replace = TRUE),
  X3 = sample(1:20, n, replace = TRUE),
  X4 = runif(n)
)

# Response independent of all predictors
simulated_tree$y <- rnorm(n)

# Check the number of unique values
sapply(simulated_tree, function(x) length(unique(x)))
 X1  X2  X3  X4   y 
  2   5  20 200 200 
tree_model <- rpart(
  y ~ .,
  data = simulated_tree,
  method = "anova",
  control = rpart.control(
    cp = 0,
    minsplit = 10
  )
)

#rpart.plot(tree_model)

tree_model$variable.importance
      X4       X3       X2       X1 
61.26549 33.15347 17.50742  4.43626 
library(rpart)

set.seed(200)

results <- replicate(100, {
  
  simulated_tree <- data.frame(
    X1 = sample(1:2, 200, replace = TRUE),
    X2 = sample(1:5, 200, replace = TRUE),
    X3 = sample(1:20, 200, replace = TRUE),
    X4 = runif(200),
    y = rnorm(200)
  )
  
  tree_model <- rpart(
    y ~ .,
    data = simulated_tree,
    method = "anova",
    control = rpart.control(
      cp = 0,
      minsplit = 10,
      xval = 0
    )
  )
  
  as.character(tree_model$frame$var[1])
})

# Count how frequently each predictor was selected
table(factor(results, levels = c("X1", "X2", "X3", "X4")))

X1 X2 X3 X4 
 3 10 24 63 

Interpretation:

The simulation shows how different predictor granularities can affect variable importance in a decision tree. All predictors have the same sample size of 200 observations, but they have different numbers of unique values. Predictors with more unique values provide more possible splitting points, which can cause the decision tree to assign them higher importance even when they are not related to the response variable. This demonstrates how decision trees can be biased toward predictors with higher granularity.

Exercise 8.3

a. Why does the model on the right focus its importance on just the first few of predictors, whereas the model on the left spreads importance across more predictors?

The model on the right focuses on fewer predictors because it uses a higher learning rate and bagging fraction. The higher learning rate allows the model to make larger adjustments, while the higher bagging fraction uses more observations to build each tree. As a result, the model concentrates importance on a few predictors. The model on the left uses smaller values, which spreads importance across more predictors.

b. Which model do you think would be more predictive of other samples?

I would expect the model on the left to perform better on new samples because it uses a lower learning rate, allowing the model to learn more gradually and potentially reduce overfitting. The model on the right makes larger adjustments, which may increase the risk of overfitting. However, we would need to compare their prediction errors on test data to determine which model performs better.

c. How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?

Increasing the interaction depth allows the boosted trees to capture more complex relationships between predictors. This may spread variable importance across more predictors and produce a flatter importance curve. However, if a few predictors dominate the interactions, the importance scores may remain concentrated among those predictors.

8.7

library(tidyverse)
Warning: package 'tibble' was built under R version 4.5.2
Warning: package 'tidyr' was built under R version 4.5.2
Warning: package 'readr' was built under R version 4.5.2
Warning: package 'purrr' was built under R version 4.5.2
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ lubridate 1.9.4     ✔ tibble    3.3.1
✔ purrr     1.2.1     ✔ tidyr     1.3.2
✔ readr     2.1.6     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ stringr::boundary() masks strucchange::boundary()
✖ dplyr::combine()    masks randomForest::combine()
✖ dplyr::filter()     masks stats::filter()
✖ dplyr::lag()        masks stats::lag()
✖ purrr::lift()       masks caret::lift()
✖ ggplot2::margin()   masks randomForest::margin()
✖ dplyr::where()      masks party::where()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(AppliedPredictiveModeling)
Warning: package 'AppliedPredictiveModeling' was built under R version 4.5.2
library(caret)
library(earth)
Warning: package 'earth' was built under R version 4.5.2
Loading required package: Formula
Loading required package: plotmo
Warning: package 'plotmo' was built under R version 4.5.2
Loading required package: plotrix
Warning: package 'plotrix' was built under R version 4.5.2
library(kernlab)

Attaching package: 'kernlab'

The following object is masked from 'package:purrr':

    cross

The following object is masked from 'package:modeltools':

    prior

The following object is masked from 'package:ggplot2':

    alpha
library(nnet)
Warning: package 'nnet' was built under R version 4.5.2
set.seed(42)

data(ChemicalManufacturingProcess)

# Dataset exploration
head(ChemicalManufacturingProcess)
  Yield BiologicalMaterial01 BiologicalMaterial02 BiologicalMaterial03
1 38.00                 6.25                49.58                56.97
2 42.44                 8.01                60.97                67.48
3 42.03                 8.01                60.97                67.48
4 41.42                 8.01                60.97                67.48
5 42.49                 7.47                63.33                72.25
6 43.57                 6.12                58.36                65.31
  BiologicalMaterial04 BiologicalMaterial05 BiologicalMaterial06
1                12.74                19.51                43.73
2                14.65                19.36                53.14
3                14.65                19.36                53.14
4                14.65                19.36                53.14
5                14.02                17.91                54.66
6                15.17                21.79                51.23
  BiologicalMaterial07 BiologicalMaterial08 BiologicalMaterial09
1                  100                16.66                11.44
2                  100                19.04                12.55
3                  100                19.04                12.55
4                  100                19.04                12.55
5                  100                18.22                12.80
6                  100                18.30                12.13
  BiologicalMaterial10 BiologicalMaterial11 BiologicalMaterial12
1                 3.46               138.09                18.83
2                 3.46               153.67                21.05
3                 3.46               153.67                21.05
4                 3.46               153.67                21.05
5                 3.05               147.61                21.05
6                 3.78               151.88                20.76
  ManufacturingProcess01 ManufacturingProcess02 ManufacturingProcess03
1                     NA                     NA                     NA
2                    0.0                      0                     NA
3                    0.0                      0                     NA
4                    0.0                      0                     NA
5                   10.7                      0                     NA
6                   12.0                      0                     NA
  ManufacturingProcess04 ManufacturingProcess05 ManufacturingProcess06
1                     NA                     NA                     NA
2                    917                 1032.2                  210.0
3                    912                 1003.6                  207.1
4                    911                 1014.6                  213.3
5                    918                 1027.5                  205.7
6                    924                 1016.8                  208.9
  ManufacturingProcess07 ManufacturingProcess08 ManufacturingProcess09
1                     NA                     NA                  43.00
2                    177                    178                  46.57
3                    178                    178                  45.07
4                    177                    177                  44.92
5                    178                    178                  44.96
6                    178                    178                  45.32
  ManufacturingProcess10 ManufacturingProcess11 ManufacturingProcess12
1                     NA                     NA                     NA
2                     NA                     NA                      0
3                     NA                     NA                      0
4                     NA                     NA                      0
5                     NA                     NA                      0
6                     NA                     NA                      0
  ManufacturingProcess13 ManufacturingProcess14 ManufacturingProcess15
1                   35.5                   4898                   6108
2                   34.0                   4869                   6095
3                   34.8                   4878                   6087
4                   34.8                   4897                   6102
5                   34.6                   4992                   6233
6                   34.0                   4985                   6222
  ManufacturingProcess16 ManufacturingProcess17 ManufacturingProcess18
1                   4682                   35.5                   4865
2                   4617                   34.0                   4867
3                   4617                   34.8                   4877
4                   4635                   34.8                   4872
5                   4733                   33.9                   4886
6                   4786                   33.4                   4862
  ManufacturingProcess19 ManufacturingProcess20 ManufacturingProcess21
1                   6049                   4665                    0.0
2                   6097                   4621                    0.0
3                   6078                   4621                    0.0
4                   6073                   4611                    0.0
5                   6102                   4659                   -0.7
6                   6115                   4696                   -0.6
  ManufacturingProcess22 ManufacturingProcess23 ManufacturingProcess24
1                     NA                     NA                     NA
2                      3                      0                      3
3                      4                      1                      4
4                      5                      2                      5
5                      8                      4                     18
6                      9                      1                      1
  ManufacturingProcess25 ManufacturingProcess26 ManufacturingProcess27
1                   4873                   6074                   4685
2                   4869                   6107                   4630
3                   4897                   6116                   4637
4                   4892                   6111                   4630
5                   4930                   6151                   4684
6                   4871                   6128                   4687
  ManufacturingProcess28 ManufacturingProcess29 ManufacturingProcess30
1                   10.7                   21.0                    9.9
2                   11.2                   21.4                    9.9
3                   11.1                   21.3                    9.4
4                   11.1                   21.3                    9.4
5                   11.3                   21.6                    9.0
6                   11.4                   21.7                   10.1
  ManufacturingProcess31 ManufacturingProcess32 ManufacturingProcess33
1                   69.1                    156                     66
2                   68.7                    169                     66
3                   69.3                    173                     66
4                   69.3                    171                     68
5                   69.4                    171                     70
6                   68.2                    173                     70
  ManufacturingProcess34 ManufacturingProcess35 ManufacturingProcess36
1                    2.4                    486                  0.019
2                    2.6                    508                  0.019
3                    2.6                    509                  0.018
4                    2.5                    496                  0.018
5                    2.5                    468                  0.017
6                    2.5                    490                  0.018
  ManufacturingProcess37 ManufacturingProcess38 ManufacturingProcess39
1                    0.5                      3                    7.2
2                    2.0                      2                    7.2
3                    0.7                      2                    7.2
4                    1.2                      2                    7.2
5                    0.2                      2                    7.3
6                    0.4                      2                    7.2
  ManufacturingProcess40 ManufacturingProcess41 ManufacturingProcess42
1                     NA                     NA                   11.6
2                    0.1                   0.15                   11.1
3                    0.0                   0.00                   12.0
4                    0.0                   0.00                   10.6
5                    0.0                   0.00                   11.0
6                    0.0                   0.00                   11.5
  ManufacturingProcess43 ManufacturingProcess44 ManufacturingProcess45
1                    3.0                    1.8                    2.4
2                    0.9                    1.9                    2.2
3                    1.0                    1.8                    2.3
4                    1.1                    1.8                    2.1
5                    1.1                    1.7                    2.1
6                    2.2                    1.8                    2.0
dim(ChemicalManufacturingProcess)
[1] 176  58
str(ChemicalManufacturingProcess)
'data.frame':   176 obs. of  58 variables:
 $ Yield                 : num  38 42.4 42 41.4 42.5 ...
 $ BiologicalMaterial01  : num  6.25 8.01 8.01 8.01 7.47 6.12 7.48 6.94 6.94 6.94 ...
 $ BiologicalMaterial02  : num  49.6 61 61 61 63.3 ...
 $ BiologicalMaterial03  : num  57 67.5 67.5 67.5 72.2 ...
 $ BiologicalMaterial04  : num  12.7 14.7 14.7 14.7 14 ...
 $ BiologicalMaterial05  : num  19.5 19.4 19.4 19.4 17.9 ...
 $ BiologicalMaterial06  : num  43.7 53.1 53.1 53.1 54.7 ...
 $ BiologicalMaterial07  : num  100 100 100 100 100 100 100 100 100 100 ...
 $ BiologicalMaterial08  : num  16.7 19 19 19 18.2 ...
 $ BiologicalMaterial09  : num  11.4 12.6 12.6 12.6 12.8 ...
 $ BiologicalMaterial10  : num  3.46 3.46 3.46 3.46 3.05 3.78 3.04 3.85 3.85 3.85 ...
 $ BiologicalMaterial11  : num  138 154 154 154 148 ...
 $ BiologicalMaterial12  : num  18.8 21.1 21.1 21.1 21.1 ...
 $ ManufacturingProcess01: num  NA 0 0 0 10.7 12 11.5 12 12 12 ...
 $ ManufacturingProcess02: num  NA 0 0 0 0 0 0 0 0 0 ...
 $ ManufacturingProcess03: num  NA NA NA NA NA NA 1.56 1.55 1.56 1.55 ...
 $ ManufacturingProcess04: num  NA 917 912 911 918 924 933 929 928 938 ...
 $ ManufacturingProcess05: num  NA 1032 1004 1015 1028 ...
 $ ManufacturingProcess06: num  NA 210 207 213 206 ...
 $ ManufacturingProcess07: num  NA 177 178 177 178 178 177 178 177 177 ...
 $ ManufacturingProcess08: num  NA 178 178 177 178 178 178 178 177 177 ...
 $ ManufacturingProcess09: num  43 46.6 45.1 44.9 45 ...
 $ ManufacturingProcess10: num  NA NA NA NA NA NA 11.6 10.2 9.7 10.1 ...
 $ ManufacturingProcess11: num  NA NA NA NA NA NA 11.5 11.3 11.1 10.2 ...
 $ ManufacturingProcess12: num  NA 0 0 0 0 0 0 0 0 0 ...
 $ ManufacturingProcess13: num  35.5 34 34.8 34.8 34.6 34 32.4 33.6 33.9 34.3 ...
 $ ManufacturingProcess14: num  4898 4869 4878 4897 4992 ...
 $ ManufacturingProcess15: num  6108 6095 6087 6102 6233 ...
 $ ManufacturingProcess16: num  4682 4617 4617 4635 4733 ...
 $ ManufacturingProcess17: num  35.5 34 34.8 34.8 33.9 33.4 33.8 33.6 33.9 35.3 ...
 $ ManufacturingProcess18: num  4865 4867 4877 4872 4886 ...
 $ ManufacturingProcess19: num  6049 6097 6078 6073 6102 ...
 $ ManufacturingProcess20: num  4665 4621 4621 4611 4659 ...
 $ ManufacturingProcess21: num  0 0 0 0 -0.7 -0.6 1.4 0 0 1 ...
 $ ManufacturingProcess22: num  NA 3 4 5 8 9 1 2 3 4 ...
 $ ManufacturingProcess23: num  NA 0 1 2 4 1 1 2 3 1 ...
 $ ManufacturingProcess24: num  NA 3 4 5 18 1 1 2 3 4 ...
 $ ManufacturingProcess25: num  4873 4869 4897 4892 4930 ...
 $ ManufacturingProcess26: num  6074 6107 6116 6111 6151 ...
 $ ManufacturingProcess27: num  4685 4630 4637 4630 4684 ...
 $ ManufacturingProcess28: num  10.7 11.2 11.1 11.1 11.3 11.4 11.2 11.1 11.3 11.4 ...
 $ ManufacturingProcess29: num  21 21.4 21.3 21.3 21.6 21.7 21.2 21.2 21.5 21.7 ...
 $ ManufacturingProcess30: num  9.9 9.9 9.4 9.4 9 10.1 11.2 10.9 10.5 9.8 ...
 $ ManufacturingProcess31: num  69.1 68.7 69.3 69.3 69.4 68.2 67.6 67.9 68 68.5 ...
 $ ManufacturingProcess32: num  156 169 173 171 171 173 159 161 160 164 ...
 $ ManufacturingProcess33: num  66 66 66 68 70 70 65 65 65 66 ...
 $ ManufacturingProcess34: num  2.4 2.6 2.6 2.5 2.5 2.5 2.5 2.5 2.5 2.5 ...
 $ ManufacturingProcess35: num  486 508 509 496 468 490 475 478 491 488 ...
 $ ManufacturingProcess36: num  0.019 0.019 0.018 0.018 0.017 0.018 0.019 0.019 0.019 0.019 ...
 $ ManufacturingProcess37: num  0.5 2 0.7 1.2 0.2 0.4 0.8 1 1.2 1.8 ...
 $ ManufacturingProcess38: num  3 2 2 2 2 2 2 2 3 3 ...
 $ ManufacturingProcess39: num  7.2 7.2 7.2 7.2 7.3 7.2 7.3 7.3 7.4 7.1 ...
 $ ManufacturingProcess40: num  NA 0.1 0 0 0 0 0 0 0 0 ...
 $ ManufacturingProcess41: num  NA 0.15 0 0 0 0 0 0 0 0 ...
 $ ManufacturingProcess42: num  11.6 11.1 12 10.6 11 11.5 11.7 11.4 11.4 11.3 ...
 $ ManufacturingProcess43: num  3 0.9 1 1.1 1.1 2.2 0.7 0.8 0.9 0.8 ...
 $ ManufacturingProcess44: num  1.8 1.9 1.8 1.8 1.7 1.8 2 2 1.9 1.9 ...
 $ ManufacturingProcess45: num  2.4 2.2 2.3 2.1 2.1 2 2.2 2.2 2.1 2.4 ...
summary(ChemicalManufacturingProcess)
     Yield       BiologicalMaterial01 BiologicalMaterial02 BiologicalMaterial03
 Min.   :35.25   Min.   :4.580        Min.   :46.87        Min.   :56.97       
 1st Qu.:38.75   1st Qu.:5.978        1st Qu.:52.68        1st Qu.:64.98       
 Median :39.97   Median :6.305        Median :55.09        Median :67.22       
 Mean   :40.18   Mean   :6.411        Mean   :55.69        Mean   :67.70       
 3rd Qu.:41.48   3rd Qu.:6.870        3rd Qu.:58.74        3rd Qu.:70.43       
 Max.   :46.34   Max.   :8.810        Max.   :64.75        Max.   :78.25       
                                                                               
 BiologicalMaterial04 BiologicalMaterial05 BiologicalMaterial06
 Min.   : 9.38        Min.   :13.24        Min.   :40.60       
 1st Qu.:11.24        1st Qu.:17.23        1st Qu.:46.05       
 Median :12.10        Median :18.49        Median :48.46       
 Mean   :12.35        Mean   :18.60        Mean   :48.91       
 3rd Qu.:13.22        3rd Qu.:19.90        3rd Qu.:51.34       
 Max.   :23.09        Max.   :24.85        Max.   :59.38       
                                                               
 BiologicalMaterial07 BiologicalMaterial08 BiologicalMaterial09
 Min.   :100.0        Min.   :15.88        Min.   :11.44       
 1st Qu.:100.0        1st Qu.:17.06        1st Qu.:12.60       
 Median :100.0        Median :17.51        Median :12.84       
 Mean   :100.0        Mean   :17.49        Mean   :12.85       
 3rd Qu.:100.0        3rd Qu.:17.88        3rd Qu.:13.13       
 Max.   :100.8        Max.   :19.14        Max.   :14.08       
                                                               
 BiologicalMaterial10 BiologicalMaterial11 BiologicalMaterial12
 Min.   :1.770        Min.   :135.8        Min.   :18.35       
 1st Qu.:2.460        1st Qu.:143.8        1st Qu.:19.73       
 Median :2.710        Median :146.1        Median :20.12       
 Mean   :2.801        Mean   :147.0        Mean   :20.20       
 3rd Qu.:2.990        3rd Qu.:149.6        3rd Qu.:20.75       
 Max.   :6.870        Max.   :158.7        Max.   :22.21       
                                                               
 ManufacturingProcess01 ManufacturingProcess02 ManufacturingProcess03
 Min.   : 0.00          Min.   : 0.00          Min.   :1.47          
 1st Qu.:10.80          1st Qu.:19.30          1st Qu.:1.53          
 Median :11.40          Median :21.00          Median :1.54          
 Mean   :11.21          Mean   :16.68          Mean   :1.54          
 3rd Qu.:12.15          3rd Qu.:21.50          3rd Qu.:1.55          
 Max.   :14.10          Max.   :22.50          Max.   :1.60          
 NA's   :1              NA's   :3              NA's   :15            
 ManufacturingProcess04 ManufacturingProcess05 ManufacturingProcess06
 Min.   :911.0          Min.   : 923.0         Min.   :203.0         
 1st Qu.:928.0          1st Qu.: 986.8         1st Qu.:205.7         
 Median :934.0          Median : 999.2         Median :206.8         
 Mean   :931.9          Mean   :1001.7         Mean   :207.4         
 3rd Qu.:936.0          3rd Qu.:1008.9         3rd Qu.:208.7         
 Max.   :946.0          Max.   :1175.3         Max.   :227.4         
 NA's   :1              NA's   :1              NA's   :2             
 ManufacturingProcess07 ManufacturingProcess08 ManufacturingProcess09
 Min.   :177.0          Min.   :177.0          Min.   :38.89         
 1st Qu.:177.0          1st Qu.:177.0          1st Qu.:44.89         
 Median :177.0          Median :178.0          Median :45.73         
 Mean   :177.5          Mean   :177.6          Mean   :45.66         
 3rd Qu.:178.0          3rd Qu.:178.0          3rd Qu.:46.52         
 Max.   :178.0          Max.   :178.0          Max.   :49.36         
 NA's   :1              NA's   :1                                    
 ManufacturingProcess10 ManufacturingProcess11 ManufacturingProcess12
 Min.   : 7.500         Min.   : 7.500         Min.   :   0.0        
 1st Qu.: 8.700         1st Qu.: 9.000         1st Qu.:   0.0        
 Median : 9.100         Median : 9.400         Median :   0.0        
 Mean   : 9.179         Mean   : 9.386         Mean   : 857.8        
 3rd Qu.: 9.550         3rd Qu.: 9.900         3rd Qu.:   0.0        
 Max.   :11.600         Max.   :11.500         Max.   :4549.0        
 NA's   :9              NA's   :10             NA's   :1             
 ManufacturingProcess13 ManufacturingProcess14 ManufacturingProcess15
 Min.   :32.10          Min.   :4701           Min.   :5904          
 1st Qu.:33.90          1st Qu.:4828           1st Qu.:6010          
 Median :34.60          Median :4856           Median :6032          
 Mean   :34.51          Mean   :4854           Mean   :6039          
 3rd Qu.:35.20          3rd Qu.:4882           3rd Qu.:6061          
 Max.   :38.60          Max.   :5055           Max.   :6233          
                        NA's   :1                                    
 ManufacturingProcess16 ManufacturingProcess17 ManufacturingProcess18
 Min.   :   0           Min.   :31.30          Min.   :   0          
 1st Qu.:4561           1st Qu.:33.50          1st Qu.:4813          
 Median :4588           Median :34.40          Median :4835          
 Mean   :4566           Mean   :34.34          Mean   :4810          
 3rd Qu.:4619           3rd Qu.:35.10          3rd Qu.:4862          
 Max.   :4852           Max.   :40.00          Max.   :4971          
                                                                     
 ManufacturingProcess19 ManufacturingProcess20 ManufacturingProcess21
 Min.   :5890           Min.   :   0           Min.   :-1.8000       
 1st Qu.:6001           1st Qu.:4553           1st Qu.:-0.6000       
 Median :6022           Median :4582           Median :-0.3000       
 Mean   :6028           Mean   :4556           Mean   :-0.1642       
 3rd Qu.:6050           3rd Qu.:4610           3rd Qu.: 0.0000       
 Max.   :6146           Max.   :4759           Max.   : 3.6000       
                                                                     
 ManufacturingProcess22 ManufacturingProcess23 ManufacturingProcess24
 Min.   : 0.000         Min.   :0.000          Min.   : 0.000        
 1st Qu.: 3.000         1st Qu.:2.000          1st Qu.: 4.000        
 Median : 5.000         Median :3.000          Median : 8.000        
 Mean   : 5.406         Mean   :3.017          Mean   : 8.834        
 3rd Qu.: 8.000         3rd Qu.:4.000          3rd Qu.:14.000        
 Max.   :12.000         Max.   :6.000          Max.   :23.000        
 NA's   :1              NA's   :1              NA's   :1             
 ManufacturingProcess25 ManufacturingProcess26 ManufacturingProcess27
 Min.   :   0           Min.   :   0           Min.   :   0          
 1st Qu.:4832           1st Qu.:6020           1st Qu.:4560          
 Median :4855           Median :6047           Median :4587          
 Mean   :4828           Mean   :6016           Mean   :4563          
 3rd Qu.:4877           3rd Qu.:6070           3rd Qu.:4609          
 Max.   :4990           Max.   :6161           Max.   :4710          
 NA's   :5              NA's   :5              NA's   :5             
 ManufacturingProcess28 ManufacturingProcess29 ManufacturingProcess30
 Min.   : 0.000         Min.   : 0.00          Min.   : 0.000        
 1st Qu.: 0.000         1st Qu.:19.70          1st Qu.: 8.800        
 Median :10.400         Median :19.90          Median : 9.100        
 Mean   : 6.592         Mean   :20.01          Mean   : 9.161        
 3rd Qu.:10.750         3rd Qu.:20.40          3rd Qu.: 9.700        
 Max.   :11.500         Max.   :22.00          Max.   :11.200        
 NA's   :5              NA's   :5              NA's   :5             
 ManufacturingProcess31 ManufacturingProcess32 ManufacturingProcess33
 Min.   : 0.00          Min.   :143.0          Min.   :56.00         
 1st Qu.:70.10          1st Qu.:155.0          1st Qu.:62.00         
 Median :70.80          Median :158.0          Median :64.00         
 Mean   :70.18          Mean   :158.5          Mean   :63.54         
 3rd Qu.:71.40          3rd Qu.:162.0          3rd Qu.:65.00         
 Max.   :72.50          Max.   :173.0          Max.   :70.00         
 NA's   :5                                     NA's   :5             
 ManufacturingProcess34 ManufacturingProcess35 ManufacturingProcess36
 Min.   :2.300          Min.   :463.0          Min.   :0.01700       
 1st Qu.:2.500          1st Qu.:490.0          1st Qu.:0.01900       
 Median :2.500          Median :495.0          Median :0.02000       
 Mean   :2.494          Mean   :495.6          Mean   :0.01957       
 3rd Qu.:2.500          3rd Qu.:501.5          3rd Qu.:0.02000       
 Max.   :2.600          Max.   :522.0          Max.   :0.02200       
 NA's   :5              NA's   :5              NA's   :5             
 ManufacturingProcess37 ManufacturingProcess38 ManufacturingProcess39
 Min.   :0.000          Min.   :0.000          Min.   :0.000         
 1st Qu.:0.700          1st Qu.:2.000          1st Qu.:7.100         
 Median :1.000          Median :3.000          Median :7.200         
 Mean   :1.014          Mean   :2.534          Mean   :6.851         
 3rd Qu.:1.300          3rd Qu.:3.000          3rd Qu.:7.300         
 Max.   :2.300          Max.   :3.000          Max.   :7.500         
                                                                     
 ManufacturingProcess40 ManufacturingProcess41 ManufacturingProcess42
 Min.   :0.00000        Min.   :0.00000        Min.   : 0.00         
 1st Qu.:0.00000        1st Qu.:0.00000        1st Qu.:11.40         
 Median :0.00000        Median :0.00000        Median :11.60         
 Mean   :0.01771        Mean   :0.02371        Mean   :11.21         
 3rd Qu.:0.00000        3rd Qu.:0.00000        3rd Qu.:11.70         
 Max.   :0.10000        Max.   :0.20000        Max.   :12.10         
 NA's   :1              NA's   :1                                    
 ManufacturingProcess43 ManufacturingProcess44 ManufacturingProcess45
 Min.   : 0.0000        Min.   :0.000          Min.   :0.000         
 1st Qu.: 0.6000        1st Qu.:1.800          1st Qu.:2.100         
 Median : 0.8000        Median :1.900          Median :2.200         
 Mean   : 0.9119        Mean   :1.805          Mean   :2.138         
 3rd Qu.: 1.0250        3rd Qu.:1.900          3rd Qu.:2.300         
 Max.   :11.0000        Max.   :2.100          Max.   :2.600         
                                                                     
# Missing values
sum(is.na(ChemicalManufacturingProcess))
[1] 106
colSums(is.na(ChemicalManufacturingProcess))
                 Yield   BiologicalMaterial01   BiologicalMaterial02 
                     0                      0                      0 
  BiologicalMaterial03   BiologicalMaterial04   BiologicalMaterial05 
                     0                      0                      0 
  BiologicalMaterial06   BiologicalMaterial07   BiologicalMaterial08 
                     0                      0                      0 
  BiologicalMaterial09   BiologicalMaterial10   BiologicalMaterial11 
                     0                      0                      0 
  BiologicalMaterial12 ManufacturingProcess01 ManufacturingProcess02 
                     0                      1                      3 
ManufacturingProcess03 ManufacturingProcess04 ManufacturingProcess05 
                    15                      1                      1 
ManufacturingProcess06 ManufacturingProcess07 ManufacturingProcess08 
                     2                      1                      1 
ManufacturingProcess09 ManufacturingProcess10 ManufacturingProcess11 
                     0                      9                     10 
ManufacturingProcess12 ManufacturingProcess13 ManufacturingProcess14 
                     1                      0                      1 
ManufacturingProcess15 ManufacturingProcess16 ManufacturingProcess17 
                     0                      0                      0 
ManufacturingProcess18 ManufacturingProcess19 ManufacturingProcess20 
                     0                      0                      0 
ManufacturingProcess21 ManufacturingProcess22 ManufacturingProcess23 
                     0                      1                      1 
ManufacturingProcess24 ManufacturingProcess25 ManufacturingProcess26 
                     1                      5                      5 
ManufacturingProcess27 ManufacturingProcess28 ManufacturingProcess29 
                     5                      5                      5 
ManufacturingProcess30 ManufacturingProcess31 ManufacturingProcess32 
                     5                      5                      0 
ManufacturingProcess33 ManufacturingProcess34 ManufacturingProcess35 
                     5                      5                      5 
ManufacturingProcess36 ManufacturingProcess37 ManufacturingProcess38 
                     5                      0                      0 
ManufacturingProcess39 ManufacturingProcess40 ManufacturingProcess41 
                     0                      1                      1 
ManufacturingProcess42 ManufacturingProcess43 ManufacturingProcess44 
                     0                      0                      0 
ManufacturingProcess45 
                     0 
# Remove observations with missing response
model_data <- ChemicalManufacturingProcess |>
  filter(!is.na(Yield))

set.seed(42)

train_index <- createDataPartition(
  model_data$Yield,
  p = 0.80,
  list = FALSE
)

train_data <- model_data[train_index, , drop = FALSE]
test_data <- model_data[-train_index, , drop = FALSE]

# Preprocessing
preprocess_model <- preProcess(
  train_data |> select(-Yield),
  method = c("medianImpute", "center", "scale")
)

x_train <- predict(
  preprocess_model,
  train_data |> select(-Yield)
)

x_test <- predict(
  preprocess_model,
  test_data |> select(-Yield)
)

train_processed <- bind_cols(
  tibble(Yield = train_data$Yield),
  as_tibble(x_train)
)

test_processed <- bind_cols(
  tibble(Yield = test_data$Yield),
  as_tibble(x_test)
)

# Check results
dim(train_processed)
[1] 144  58
dim(test_processed)
[1] 32 58
sum(is.na(train_processed))
[1] 0
sum(is.na(test_processed))
[1] 0
library(caret)
library(rpart)
library(randomForest)
library(gbm)
library(Cubist)
library(party)

# Same resampling method
set.seed(200)

ctrl <- trainControl(
  method = "cv",
  number = 10,
  savePredictions = "final"
)

# Regression tree
set.seed(200)
cart_model <- train(
  Yield ~ .,
  data = train_processed,
  method = "rpart",
  trControl = ctrl,
  tuneLength = 10
)
Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,
: There were missing values in resampled performance measures.
# Random forest
set.seed(200)
rf_model <- train(
  Yield ~ .,
  data = train_processed,
  method = "rf",
  trControl = ctrl,
  tuneLength = 5,
  ntree = 1000
)

# Boosted trees
set.seed(200)
gbm_model <- train(
  Yield ~ .,
  data = train_processed,
  method = "gbm",
  trControl = ctrl,
  tuneLength = 5,
  verbose = FALSE
)

# Cubist
# Separate predictors and response
x_train <- train_processed |>
  select(-Yield) |>
  as.data.frame()

y_train <- train_processed$Yield

# Verify
class(x_train)
[1] "data.frame"
class(y_train)
[1] "numeric"
# Fit Cubist
set.seed(200)

cubist_model <- train(
  x = x_train,
  y = y_train,
  method = "cubist",
  trControl = ctrl,
  tuneLength = 5
)

cubist_model
Cubist 

144 samples
 57 predictor

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 130, 129, 129, 130, 131, 131, ... 
Resampling results across tuning parameters:

  committees  neighbors  RMSE      Rsquared   MAE      
   1          0          1.735790  0.4413753  1.2179269
   1          5          1.582588  0.5499179  1.0668288
   1          9          1.645473  0.5028669  1.1169141
  10          0          1.158436  0.5886756  0.9400021
  10          5          1.040224  0.6697810  0.8314036
  10          9          1.074680  0.6437898  0.8579826
  20          0          1.118389  0.6175767  0.9021613
  20          5          1.007256  0.6915112  0.8018895
  20          9          1.040483  0.6673778  0.8241642

RMSE was used to select the optimal model using the smallest value.
The final values used for the model were committees = 20 and neighbors = 5.
# Conditional inference tree
set.seed(200)
ctree_model <- train(
  Yield ~ .,
  data = train_processed,
  method = "ctree",
  trControl = ctrl,
  tuneLength = 5
)
results <- resamples(
  list(
    CART = cart_model,
    RF = rf_model,
    GBM = gbm_model,
    Cubist = cubist_model,
    CTree = ctree_model
  )
)

summary(results)

Call:
summary.resamples(object = results)

Models: CART, RF, GBM, Cubist, CTree 
Number of resamples: 10 

MAE 
            Min.   1st Qu.    Median      Mean   3rd Qu.     Max. NA's
CART   0.7193020 1.0666906 1.2200787 1.1778999 1.3310017 1.539486    0
RF     0.4786234 0.6672862 0.9401144 0.8561532 1.0284428 1.206007    0
GBM    0.5853965 0.6971122 0.8527824 0.8274906 0.9333039 1.076002    0
Cubist 0.4708544 0.6577201 0.8635086 0.8018895 0.9226732 1.044647    0
CTree  0.6155495 0.9893697 1.1895246 1.1881499 1.4020487 1.793866    0

RMSE 
            Min.   1st Qu.   Median     Mean  3rd Qu.     Max. NA's
CART   0.9112750 1.3128026 1.537560 1.480402 1.668496 1.856052    0
RF     0.5845038 0.8910123 1.198280 1.109493 1.270006 1.725238    0
GBM    0.8204959 0.9177231 1.076479 1.068098 1.236501 1.321217    0
Cubist 0.6041534 0.8300363 1.058250 1.007256 1.184712 1.284616    0
CTree  0.7226186 1.1882544 1.451357 1.447584 1.605547 2.413296    0

Rsquared 
             Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
CART   0.07009737 0.3848740 0.4285589 0.4036575 0.5011280 0.7336325    0
RF     0.26589999 0.5562021 0.6798057 0.6430271 0.7910120 0.9163950    0
GBM    0.40396688 0.6362956 0.6883401 0.6532606 0.7209765 0.8342853    0
Cubist 0.41685790 0.6194127 0.7496965 0.6915112 0.8101867 0.8533850    0
CTree  0.07691661 0.1957385 0.4067542 0.3948990 0.5605886 0.7682798    0
bwplot(results, metric = "RMSE")

library(dplyr)

models <- list(
  CART = cart_model,
  RF = rf_model,
  GBM = gbm_model,
  Cubist = cubist_model,
  CTree = ctree_model
)

test_results <- lapply(names(models), function(name) {
  
  predictions <- predict(models[[name]], newdata = test_processed)
  
  data.frame(
    Model = name,
    RMSE = RMSE(predictions, test_processed$Yield),
    Rsquared = R2(predictions, test_processed$Yield),
    MAE = MAE(predictions, test_processed$Yield)
  )
}) |>
  bind_rows() |>
  arrange(RMSE)

test_results
   Model      RMSE  Rsquared       MAE
1 Cubist 0.9540283 0.7949420 0.7306373
2     RF 1.1563687 0.7497949 0.8483462
3    GBM 1.2413309 0.6482784 0.9532315
4   CART 1.5027045 0.4561721 1.1069728
5  CTree 1.5874339 0.3709582 1.1280027

Interpretation:

The Cubist regression model showed the best overall test performance among the five tree-based models, with the lowest RMSE of 0.9540, MAE of 0.7306, and highest R² of 0.7949. Random Forest was the second-best model on the test set, followed by GBM, CART, and CTree. The cross-validation plot also showed that Cubist and GBM had relatively low prediction errors. Overall, Cubist appears to be the optimal tree-based model for predicting manufacturing Yield.

b. Identify the most important predictors

cubist_importance <- varImp(cubist_model, scale = FALSE)

cubist_importance
cubist variable importance

  only 20 most important variables shown (out of 57)

                       Overall
ManufacturingProcess32    52.5
ManufacturingProcess09    35.0
ManufacturingProcess17    30.0
BiologicalMaterial06      19.0
ManufacturingProcess04    18.0
ManufacturingProcess33    16.5
ManufacturingProcess29    16.0
ManufacturingProcess45    13.0
ManufacturingProcess39    12.5
ManufacturingProcess27    12.0
BiologicalMaterial03      12.0
ManufacturingProcess13    10.5
BiologicalMaterial12      10.0
BiologicalMaterial02       9.5
ManufacturingProcess25     8.5
ManufacturingProcess05     6.5
ManufacturingProcess01     6.5
ManufacturingProcess11     6.5
ManufacturingProcess31     5.5
ManufacturingProcess37     5.5
plot(cubist_importance, top = 10)

top10_tree <- cubist_importance$importance |>
  tibble::rownames_to_column("Predictor") |>
  arrange(desc(Overall)) |>
  slice_head(n = 10)

top10_tree
                Predictor Overall
1  ManufacturingProcess32    52.5
2  ManufacturingProcess09    35.0
3  ManufacturingProcess17    30.0
4    BiologicalMaterial06    19.0
5  ManufacturingProcess04    18.0
6  ManufacturingProcess33    16.5
7  ManufacturingProcess29    16.0
8  ManufacturingProcess45    13.0
9  ManufacturingProcess39    12.5
10   BiologicalMaterial03    12.0

Interpretation:

The Cubist model identified ManufacturingProcess32, ManufacturingProcess09, and ManufacturingProcess17 as the three most important predictors, with importance scores of 52.5, 35.0, and 30.0, respectively. Among the top 10 predictors, eight were manufacturing process variables and two were biological material variables. This suggests that manufacturing process variables play a major role in predicting Yield in the Cubist model. However, biological material variables also contribute to the prediction.

library(dplyr)

# Import results from previous exercises
top10_linear <- read.csv("ridge_top10.csv")
top10_nonlinear <- read.csv("svm_top10.csv")

# Cubist vs Ridge
common_linear <- intersect(
  top10_tree$Predictor,
  top10_linear$Predictor
)

# Cubist vs SVM Radial
common_nonlinear <- intersect(
  top10_tree$Predictor,
  top10_nonlinear$Variable
)

common_linear
[1] "ManufacturingProcess32" "ManufacturingProcess09" "ManufacturingProcess17"
[4] "ManufacturingProcess33" "BiologicalMaterial03"  
common_nonlinear
[1] "ManufacturingProcess32" "ManufacturingProcess09" "ManufacturingProcess17"
[4] "BiologicalMaterial03"  

c. Plot the optimal

#install.packages('rpart.plot')
library(rpart.plot)
Warning: package 'rpart.plot' was built under R version 4.5.2
rpart.plot(
  cart_model$finalModel,
  #cubist_model$finalModel,
  type = 2,
  extra = 101,
  fallen.leaves = TRUE,
  main = "Regression Tree for Manufacturing Yield"
)

library(ggplot2)

node_data <- data.frame(
  Yield = train_processed$Yield,
  Node = factor(cart_model$finalModel$where)
)

ggplot(node_data, aes(x = Node, y = Yield)) +
  geom_boxplot(fill = "steelblue", alpha = 0.7) +
  labs(
    title = "Distribution of Yield in Terminal Nodes",
    x = "Terminal Node",
    y = "Manufacturing Yield"
  ) +
  theme_minimal()

Interpretation:

Yes, the regression tree provides additional information about the relationships between biological material, manufacturing process predictors, and Yield. ManufacturingProcess32 is the first splitting variable, suggesting that it plays an important role in predicting Yield. Other process variables, including ManufacturingProcess17, ManufacturingProcess13, and ManufacturingProcess03, also contribute to the tree, while BiologicalMaterial03 and BiologicalMaterial05 indicate that biological characteristics are important as well.

The terminal-node boxplot shows that Yield differs across groups of observations. Node 5 has the lowest median Yield of approximately 37.3, while Node 15 has the highest median Yield of approximately 43.0. This suggests that certain combinations of biological and manufacturing process conditions are associated with higher manufacturing yields.

Overall, the regression tree provides additional insight by showing how different predictors interact to separate observations into groups with different Yield levels. However, these relationships are associations and do not necessarily indicate causation.