Gradient Boosted Trees
We consider the same dataset and variable importance problem using gradient boosted trees. The gbm library and variable importance measures are run within caret training procedure. The gradient boosting approach relies on a weak learner in an additive manner that minimizes the given loss function.
set.seed(1027)
gbmControlFull <- trainControl(method = "repeatedcv",
number = 10, # for debug mode, set this to a low number (1)
repeats = 10 ,
selectionFunction = "best",
verboseIter = FALSE)
gbmControlLight <- trainControl(method = "repeatedcv",
number = 5, # for debug mode, set this to a low number (1)
repeats = 1 ,
selectionFunction = "best",
verboseIter = FALSE)
if(tuningFull == TRUE)
{
gbmControl = gbmControlFull
gbmTune = expand.grid( n.trees = seq(250, 1000, by = 250),
shrinkage = c( 0.02, 0.05, 0.1) ,
interaction.depth = c( 3, 7) ,
n.minobsinnode = 15
)
} else {
gbmControl = gbmControlLight
gbmTune = expand.grid( n.trees = c( 2000),
shrinkage = c( 0.02, 0.05 , 0.1 ) ,
interaction.depth = c( 3, 7) ,
n.minobsinnode = 4
)
}
(gbmTune = caret::train( x = sdata_trainX_pp,
y = sdata_train_pp$PH ,
method = "gbm",
tuneGrid = gbmTune ,
verbose = FALSE,
metric = "RMSE" ,
trControl = gbmControl ) )## Stochastic Gradient Boosting
##
## 2055 samples
## 32 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times)
## Summary of sample sizes: 1850, 1849, 1849, 1851, 1850, 1849, ...
## Resampling results across tuning parameters:
##
## shrinkage interaction.depth n.trees RMSE Rsquared MAE
## 0.02 3 250 0.1434388 0.3444707 0.1130682
## 0.02 3 500 0.1399058 0.3781026 0.1093679
## 0.02 3 750 0.1381834 0.3948353 0.1075680
## 0.02 3 1000 0.1371452 0.4052631 0.1065308
## 0.02 7 250 0.1375659 0.3998528 0.1072464
## 0.02 7 500 0.1339780 0.4329578 0.1037270
## 0.02 7 750 0.1326790 0.4454112 0.1025093
## 0.02 7 1000 0.1320687 0.4515818 0.1019326
## 0.05 3 250 0.1391815 0.3845878 0.1085746
## 0.05 3 500 0.1370316 0.4073504 0.1063473
## 0.05 3 750 0.1364003 0.4163621 0.1056836
## 0.05 3 1000 0.1360908 0.4213742 0.1053774
## 0.05 7 250 0.1337953 0.4330017 0.1032590
## 0.05 7 500 0.1324980 0.4471243 0.1018901
## 0.05 7 750 0.1319808 0.4530628 0.1013538
## 0.05 7 1000 0.1317524 0.4561859 0.1010493
## 0.10 3 250 0.1379920 0.3994445 0.1072680
## 0.10 3 500 0.1371891 0.4134839 0.1064164
## 0.10 3 750 0.1368470 0.4199454 0.1059928
## 0.10 3 1000 0.1366708 0.4242737 0.1056828
## 0.10 7 250 0.1343453 0.4341820 0.1034168
## 0.10 7 500 0.1335705 0.4438374 0.1027183
## 0.10 7 750 0.1332796 0.4472765 0.1023144
## 0.10 7 1000 0.1332705 0.4482689 0.1022585
##
## Tuning parameter 'n.minobsinnode' was held constant at a value of 15
## RMSE was used to select the optimal model using the smallest value.
## The final values used for the model were n.trees = 1000, interaction.depth =
## 7, shrinkage = 0.05 and n.minobsinnode = 15.As the model output indicates the greater tree depth outperforms the lower counts and the lower shrinkage value produces a lower (better) RMSE value. The shrinkage, or learning rate, works better with smaller values, yet require longer processing time.
## Saving 7 x 5 in imageThe plot of the tuned model clearly shows the better performance with the increased max tree depth but also indicates the lower shrinkage value doesn’t guarantee a lower RMSE. For the max tree depth of 3, the lowest shrinkage value does not produce the lowest RMSE for that tree depth.
## Saving 7 x 5 in imageThe variable importance plot shows the 100% importance of the MnfFlow followed by BrandCode as the only other variable with an importance above 50%. Usagecont, OxygenFiller, and Temperature round out the top five variables based on importance with scores greater than 25%.
Overall, the final Gradient Boosted model results in an RMSE of approximately 0.13 and an \(R^2\) less than 0.5.
