3.1 tuneLength Function

tuneLength is to tell the algorithm to try different default values for the main parameter.

More details are here

http://www.rpubs.com/Mentors_Ubiqum/tunegrid_tunelength

set.seed let the model re-produce the same results.

For example, mtry tuning parameter is 6,5,1,4,7, the tuning parameters won’t change next time the model is run.

If set.seed is not set. the tuning parameters will change again when next time the model is run.

set.seed(2456)
Model.mice <- train(Survived ~., data = titanic.train[,-1],
                method = "ranger",
                trControl = myControl,
                tuneLength = 5
                )
Model.mice

## Random Forest 
## 
## 891 samples
##   8 predictor
##   2 classes: '0', '1' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 3 times) 
## Summary of sample sizes: 802, 802, 802, 802, 802, 802, ... 
## Resampling results across tuning parameters:
## 
##   min.node.size  mtry  splitrule   Accuracy   Kappa    
##    3             6     gini        0.8103567  0.5910529
##    5             5     extratrees  0.8182053  0.6042057
##    9             1     gini        0.8088711  0.5813119
##   12             4     extratrees  0.8174771  0.6000868
##   19             7     gini        0.8163284  0.6012792
## 
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were mtry = 5, splitrule =
##  extratrees and min.node.size = 5.

In this example, the value for each tuning parameter

1. min.node.size
2. mtry
3. splitrule are randomly selected, it is defined by the search function in the trainControlcall.

The model has been tuned 5 times, which is defined by the tuneLength function in the traincall.

The results show mtry = 5, splitrule = extratrees and min.node.size = 5 have the highest accuracy.

plot(Model.mice)

3.2 TuneGrid

Alternative, we can specify the values for tuning parameters using tuneGrid function in thetrain call.

the training data set has 10 columns. the 1st column is passengerID, not a predictor, the last column is outcome. The number of predictors is 8.

In below example, let’s setup mtry to be 1:8 in the tuneGrid function. method = ranger has 3 parameters. in below example, the ranger is changed to rf in the method function as rf only has one parameter, which is mtry

myControl.tunegird <- trainControl(method ="repeatedcv", 
                          number = 10,
                          repeats = 3,
                          savePredictions = "final",
                          verboseIter = FALSE)

In this example, tuneGrid is specified by myGrid, mtry is from 1 to 8 predictors

set.seed(2600)

myGrid <- expand.grid(.mtry = c(1:8))

Model.mice.tunegrid <- train(Survived ~., data = titanic.train[,-1],
                method = "rf",
                trControl = myControl.tunegird,
                tuneGrid = myGrid
                )
Model.mice.tunegrid

## Random Forest 
## 
## 891 samples
##   8 predictor
##   2 classes: '0', '1' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 3 times) 
## Summary of sample sizes: 803, 802, 802, 802, 802, 801, ... 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##   1     0.8091247  0.5816191
##   2     0.8196288  0.6063718
##   3     0.8267367  0.6202330
##   4     0.8248639  0.6157731
##   5     0.8222338  0.6121237
##   6     0.8151133  0.5995484
##   7     0.8109935  0.5894494
##   8     0.8072606  0.5821678
## 
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was mtry = 3.

when mtry =3, the model has the highest accuracy, therefore, mtry = 3 is selected.

randomForest, the rule of thumb,

1. for classification,mtry is the square root of the number of predictor variables (rounded down).

2. for regression models, it is the number of predictor variables divided by 3 (rounded down)

in this example, the square root of the number of predictor variables (rounded down) is

floor(sqrt(ncol(titanic.mice)))

## [1] 3

more on randomForest can be found here

http://code.env.duke.edu/projects/mget/export/HEAD/MGET/Trunk/PythonPackage/dist/TracOnlineDocumentation/Documentation/ArcGISReference/RandomForestModel.FitToArcGISTable.html

plot(Model.mice.tunegrid)

3.3 ROC metric

In the above example, the accuracy is used as the metric function is not defined in the train call.

Classification can also be evaluated by ROC.

More on how to choose metric is

https://www.machinelearningplus.com/machine-learning/evaluation-metrics-classification-models-r/

myControl.ROC <- trainControl(method ="repeatedcv", 
                          number = 10,
                          repeats = 3,
                          savePredictions = "final",
                          summaryFunction = twoClassSummary,
                          classProbs = TRUE,
                          verboseIter = FALSE)

table(titanic.train$Survived)

## 
##   0   1 
## 549 342

levels(titanic.train$Survived) <- make.names(levels(factor(titanic.train$Survived)))

table(titanic.train$Survived)

## 
##  X0  X1 
## 549 342

set.seed(3001)
myGrid <- expand.grid(.mtry = c(1:8))

Model.mice.ROC <- train(Survived ~., data = titanic.train[,-1],
                method = "rf",
                trControl = myControl.ROC,
                metric = "ROC",
                tuneGrid = myGrid
                )
Model.mice.ROC

## Random Forest 
## 
## 891 samples
##   8 predictor
##   2 classes: 'X0', 'X1' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 3 times) 
## Summary of sample sizes: 802, 802, 802, 802, 802, 802, ... 
## Resampling results across tuning parameters:
## 
##   mtry  ROC        Sens       Spec     
##   1     0.8562551  0.9016049  0.6580672
##   2     0.8585748  0.9028283  0.6835294
##   3     0.8601011  0.9143771  0.6815126
##   4     0.8601161  0.9113692  0.6766387
##   5     0.8588008  0.8962065  0.6872549
##   6     0.8528111  0.8864422  0.6900840
##   7     0.8468462  0.8834343  0.6822129
##   8     0.8426973  0.8804040  0.6792437
## 
## ROC was used to select the optimal model using the largest value.
## The final value used for the model was mtry = 4.

plot(Model.mice.ROC)