HHernandez_DATA624_HW09

The importance of V1 decreased. Due to the correlation, the importance to the model was distributed between V1 and duplicate1 since, during tree model creation, any of them can be picked at random and any of them provide a good data split.

3. Use the cforest function in the party package to fit a random forest model using conditional inference trees. The party package function varimp can calculate predictor importance. The conditional argument of that function toggles between the traditional importance measure and the modified version described in Strobl et al. (2007). Do these importances show the same pattern as the traditional random forest model?

library(party)
cfModel <- cforest(y ~ ., data=simulated)
cfImp <- varimp(cfModel)
cfImp[order(-cfImp)]

##            V4            V2    duplicate1            V1            V5 
##  7.6223892727  6.0579730772  5.0941897280  4.6171158805  1.7161194047 
##            V7            V9            V3            V6           V10 
##  0.0465374951  0.0046062409  0.0003116115 -0.0289427183 -0.0310326410 
##            V8 
## -0.0380965511

cond_cfImp <- varimp(cfModel, conditional=T)
cond_cfImp[order(-cond_cfImp)]

##           V4           V2   duplicate1           V1           V5 
##  6.175211816  4.806626676  2.039771619  1.894092913  1.028543061 
##           V6           V3           V9           V7          V10 
##  0.021733840  0.020998486  0.003766844 -0.006026025 -0.015318278 
##           V8 
## -0.022310557

Importances differ in couple of ways, for top 5 features, the rank order was the same but the conditional values were lower than traditional, the most drastic reduction was on the highly correlated variables. For the reamining 5, the order was changed completely but the magnitude was not.

4. Repeat this process with different tree models, such as boosted trees and Cubist. Does the same pattern occur?

#install.packages("gbm")
library(gbm)
gbmModel <- gbm(y ~ ., data=simulated, distribution='gaussian')
summary(gbmModel)

##                   var    rel.inf
## V4                 V4 27.8345107
## V2                 V2 23.2038908
## V1                 V1 17.6456066
## duplicate1 duplicate1 12.3248580
## V5                 V5 10.5288759
## V3                 V3  7.9846250
## V7                 V7  0.2611591
## V6                 V6  0.2164740
## V8                 V8  0.0000000
## V9                 V9  0.0000000
## V10               V10  0.0000000

library(Cubist)
cubistModel <- cubist(x=simulated[,-(ncol(simulated)-1)], y=simulated$y, committees=100)
varImp(cubistModel)

##            Overall
## V3            43.5
## V1            52.5
## V2            59.5
## duplicate1    27.5
## V4            46.0
## V8             4.0
## V5            27.0
## V6            10.0
## V10            1.0
## V7             0.0
## V9             0.0

For Boosted Trees model, the pattern for the top 5 variables was very similar to the Conditional Random Forest. The relative importance between the correlated variables favors V1 over the new one, this is because in boosting trees, one tree improves over the previous which helps reducing collinearity effects.

In Cubist model, the pattern changed completely for the top 5 variables, V3 was ranked top vs V4 in all the previous models. Also, the relative importance between the correlated variables considerable favors V1 even above other variables like V2

As expected, V1 and V4 are the variables the tree gives the highest importance (they were used to generate the target). Confirming tree selection bias, V4 showed highest, as it has the most distinct values (compared to V1). Another thing to highlight is that the tree still used the other 2 variables that apparently should not provide any useful information to the tree.

Gradient boosting involves creating and adding trees to the model sequentially.New trees are created to correct the residual errors in the predictions from the existing sequence of trees. The effect is that the model can quickly fit, then overfit the training dataset.

Learning rate is a technique to slow down the learning in the gradient boosting model by applying a weighting factor for the corrections by new trees when added to the model. Smaller learning rates generally require more trees to be added to the model. The more trees in the model, the more likely to have more predictors used across trees which will increase their overall importance. The right-hand plot learning rate of 0.9 has the opposite effect which focuses on the most relevant predictors within fewer number of trees.

Bagging fraction is the fraction of the training set observations randomly selected to propose the next tree in the expansion. Every tree may potentially be built using very different dataset. Large bagging fractions, help build subsequent trees that leverage very similar datasets (splitting similarly). This means that larger bagging fraction increases the dependent/correlated structure in the boosting trees. The right-hand plot bagging fraction focuses on the most relevant predictors as well.

2. Which model do you think would be more predictive of other samples?

Learning rate and bagging fraction are important parameters to control the overfitting of a boosting model. Smaller rate and fraction leads to better generalization and better bias-variance trade-off which in turn is supposed to provide better test/validation model performance. Left-hand model (0.1 rate and fraction) will be more predictive

3. How would increasing interaction depth affect the slope of predictor importance for either model in Fig. 8.24?

Interaction depth controls the maximum nodes per tree - number of splits it has to perform on a tree. As each split increases the total number of nodes by 3 and number of terminal nodes by 2, the total number of nodes in the tree will be 3∗N+1 and the number of terminal nodes 2∗N+. Higher interaction depth spreads out the importance more, since each tree now can grow deeper, and has more chance for other features to be involved in the splitting process

Boosting model appears to be the most optimal amongst all the tree algorithms across all the performance metrics (MAE, RMSE & Rsquared). Closely followed by Bagging

For the Test set, the Boosting model confirms to be the most optimal amongst all the algorithms across all the performance metrics (RMSE, Rsquared & MAE).

2. Which predictors are most important in the optimal tree-based regression model? Do either the biological or process variables dominate the list? How do the top 10 important predictors compare to the top 10 predictors from the optimal linear and nonlinear models?

gbmImp <- varImp(gbmModel)
gbmImp 

## gbm variable importance
## 
##   only 20 most important variables shown (out of 47)
## 
##                        Overall
## ManufacturingProcess32 100.000
## ManufacturingProcess17  22.987
## BiologicalMaterial06    18.743
## ManufacturingProcess06  17.645
## ManufacturingProcess13  12.974
## ManufacturingProcess11  11.390
## ManufacturingProcess30  10.446
## BiologicalMaterial01     8.057
## ManufacturingProcess09   7.777
## ManufacturingProcess25   7.196
## BiologicalMaterial08     7.188
## ManufacturingProcess22   7.010
## BiologicalMaterial05     6.820
## ManufacturingProcess01   6.638
## BiologicalMaterial11     6.634
## BiologicalMaterial09     6.571
## ManufacturingProcess21   6.186
## ManufacturingProcess39   5.759
## ManufacturingProcess28   5.330
## ManufacturingProcess24   4.621

plot(gbmImp, top=10)

# Linear Model (PLS) - For Reference

ctrl <- trainControl(method = "boot", number = 25)
pls_tune <- train(x = transTrainPredictors, y = trainYield,
                 method = "pls",
                 tuneLength = 15,
                 trControl = ctrl)
plsImp <- varImp(pls_tune, scale = FALSE)

# Nonlinear Model (SVM) - For Reference

svmModel <- train(x = transTrainPredictors,
                  y = trainYield,
                  method = "svmRadial",
                  preProc = c("center", "scale"),
                  tuneLength = 10)
svmImp <- varImp(svmModel)

plot(svmImp, top=10)

plot(plsImp, top=10)

The most important predictors for the tree-based model (Boosting) are primarily ManufacturingProcess predictors (8 out of 10). Interestingly, one of the 2 BiologicalMaterial predictors is the third most relevant for the model. Regarding the linear model (PLS), same proportion of ManufacturingProcess predictors in the top 10 (7 out of 10), but in this model, is more clearly defined the predominant influence over the BiologicalMaterial ones. Compared to the linear (PLS) and Nonlinear (SVM), Boosting trees agrees almost in all the top 5 predictors as well as order of importance, they both consider BiologicalMaterial06 as a relevant variable. In contrast, Linear PLS model prioritizes ManufacturingProcess variables completely (top 6)

3. Plot the optimal single tree with the distribution of yield in the terminal nodes. Does this view of the data provide additional knowledge about the biological or process predictors and their relationship with yield?

rpartFM <- rpartModel$finalModel
plot(rpartFM)
text(rpartFM, use.n=TRUE, all=TRUE, cex=.8)

trainData <- data.frame(transTrainPredictors, trainYield)
less <- subset(trainData, ManufacturingProcess32 < 0.02642 & ManufacturingProcess06 < 0.248)
more <- subset(trainData, ManufacturingProcess32 >= 0.02642 & ManufacturingProcess06 > 0.248)

mean(less$trainYield)

## [1] 39.04113

mean(more$trainYield)

## [1] 42.44591

The plot shows that after the first split using ManufacturingProcess32, the next two splits that lead to leaf nodes consider BiologicalMaterial06 on one path and ManufacturingProcess06 on the other one as the most important predictors. Maximum yield comes from the full ManufacturingProcess-based branch and is calculated as the mean of the values falling on each leaf node according to the path split values.