Predicting the Loan Default (Includes Preprocessing Stage) - A machine learning approach for a Classification Problem

library(lubridate)

## 
## Attaching package: 'lubridate'

## The following object is masked from 'package:base':
## 
##     date

library(ggplot2)
library(dplyr)

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:lubridate':
## 
##     intersect, setdiff, union

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

library(stringr)
library(caret)

## Loading required package: lattice

library(rpart)
library(ROSE)

## Loaded ROSE 0.0-3

library(ROCR)

## Loading required package: gplots

## 
## Attaching package: 'gplots'

## The following object is masked from 'package:stats':
## 
##     lowess

library(MASS)

## 
## Attaching package: 'MASS'

## The following object is masked from 'package:dplyr':
## 
##     select

library(ipred)
library(plyr)

## -------------------------------------------------------------------------

## You have loaded plyr after dplyr - this is likely to cause problems.
## If you need functions from both plyr and dplyr, please load plyr first, then dplyr:
## library(plyr); library(dplyr)

## -------------------------------------------------------------------------

## 
## Attaching package: 'plyr'

## The following objects are masked from 'package:dplyr':
## 
##     arrange, count, desc, failwith, id, mutate, rename, summarise,
##     summarize

## The following object is masked from 'package:lubridate':
## 
##     here

library(rpart.plot)

data<-read.csv('Data.csv', header=T, sep=",")

as.data.frame(names(data))

##                    names(data)
## 1                  Customer_ID
## 2          Status_Checking_Acc
## 3           Duration_in_Months
## 4               Credit_History
## 5        Purposre_Credit_Taken
## 6                Credit_Amount
## 7                  Savings_Acc
## 8  Years_At_Present_Employment
## 9               Inst_Rt_Income
## 10       Marital_Status_Gender
## 11    Other_Debtors_Guarantors
## 12         Current_Address_Yrs
## 13                    Property
## 14                         Age
## 15            Other_Inst_Plans
## 16                     Housing
## 17                      Num_CC
## 18                         Job
## 19                  Dependents
## 20                   Telephone
## 21              Foreign_Worker
## 22          Default_On_Payment
## 23                       Count

Reading the Data and transform the Default into a factor. Importing the stargazer library to get the Descriptive Statistics

as.data.frame(rapply(data,function(x)length(unique(x))))

##                             rapply(data, function(x) length(unique(x)))
## Customer_ID                                                        5000
## Status_Checking_Acc                                                   4
## Duration_in_Months                                                   33
## Credit_History                                                        5
## Purposre_Credit_Taken                                                10
## Credit_Amount                                                       921
## Savings_Acc                                                           5
## Years_At_Present_Employment                                           5
## Inst_Rt_Income                                                        4
## Marital_Status_Gender                                                 4
## Other_Debtors_Guarantors                                              3
## Current_Address_Yrs                                                   4
## Property                                                              4
## Age                                                                  53
## Other_Inst_Plans                                                      3
## Housing                                                               3
## Num_CC                                                                4
## Job                                                                   4
## Dependents                                                            2
## Telephone                                                             2
## Foreign_Worker                                                        2
## Default_On_Payment                                                    2
## Count                                                                 1

data$Default_On_Payment=as.factor(data$Default_On_Payment)
levels(data$Default_On_Payment)<- c("not.defaulted", "defaulted")

library(stargazer)

## 
## Please cite as:

##  Hlavac, Marek (2015). stargazer: Well-Formatted Regression and Summary Statistics Tables.

##  R package version 5.2. http://CRAN.R-project.org/package=stargazer

stargazer(data, type="text", title="Descriptive statistics", digits=1)

## 
## Descriptive statistics
## ============================================================
## Statistic             N     Mean    St. Dev.   Min     Max  
## ------------------------------------------------------------
## Customer_ID         5,000 102,500.5 1,443.5  100,001 105,000
## Duration_in_Months  5,000   20.9      12.1      4      72   
## Credit_Amount       5,000  3,271.3  2,821.6    250   18,424 
## Inst_Rt_Income      5,000    3.0      1.1       1       4   
## Current_Address_Yrs 5,000    2.8      1.1       1       4   
## Age                 5,000   35.5      11.4     19      75   
## Num_CC              5,000    1.4      0.6       1       4   
## Dependents          5,000    1.2      0.4       1       2   
## Count               5,000    1.0      0.0       1       1   
## ------------------------------------------------------------

Preprocessing the data

Look for duplicates id

nrow(data)

## [1] 5000

length(unique(data$member_Customer_ID))

## [1] 0

Remove columns that have unique values

col_uv = sapply(data, function(x) length(unique(x)))
cat("Constant feature count:", length(col_uv[col_uv==1]))

## Constant feature count: 1

names(col_uv[col_uv==1])

## [1] "Count"

data = data[, !names(data) %in% names(col_uv[col_uv==1])]

Percentage of missing values per case

perc.miss.case<-sapply(data, function(col) sum(is.na(col))/nrow(data))
perc.miss.case

##                 Customer_ID         Status_Checking_Acc 
##                           0                           0 
##          Duration_in_Months              Credit_History 
##                           0                           0 
##       Purposre_Credit_Taken               Credit_Amount 
##                           0                           0 
##                 Savings_Acc Years_At_Present_Employment 
##                           0                           0 
##              Inst_Rt_Income       Marital_Status_Gender 
##                           0                           0 
##    Other_Debtors_Guarantors         Current_Address_Yrs 
##                           0                           0 
##                    Property                         Age 
##                           0                           0 
##            Other_Inst_Plans                     Housing 
##                           0                           0 
##                      Num_CC                         Job 
##                           0                           0 
##                  Dependents                   Telephone 
##                           0                           0 
##              Foreign_Worker          Default_On_Payment 
##                           0                           0

data$Customer_ID<- NULL

class = sapply(data, class)
table(class)

## class
##  factor integer 
##      14       7

Recoding Variables from low to high scale. I want to capture this scale. This recoding is in accordance with the description of the variables.

# Recoding Variables 
# Status_Checking_Acc
oldvaluesSCA <- c("A11", "A12", "A13", "A14")
newvaluesSCA <- factor(c("low", "medium", "high", "none" ))
data$Status_Checking_Acc <- newvaluesSCA[ match(data$Status_Checking_Acc, oldvaluesSCA)]
data$Status_Checking_Acc=as.factor(data$Status_Checking_Acc) 
#Credit History

oldvaluesCH <- c("A30", "A31", "A32", "A33", "A34")
newvaluesCH <- factor(c("no/paid", "paid/bank", "paid/existing", "delayed/past", "existing/other" ))
data$Credit_History <- newvaluesCH[ match(data$Credit_History, oldvaluesCH)]
data$Credit_History=as.factor(data$Credit_History) 
#Purposre_Credit_Taken

oldvaluesPCT <- c("A40", "A41", "A42", "A43", "A44", "A45", "A46", "A410", "A48", "A49")
newvaluesPCT <- factor(c("new/car", "used/car", "furniture/equipment", "radio/tv", "domestic/appliances", "repair", "education", "vacation", "retraining", "bussiness", "other"))
data$Purposre_Credit_Taken <- newvaluesPCT[ match(data$Purposre_Credit_Taken, oldvaluesPCT)]
data$Purposre_Credit_Taken=as.factor(data$Purposre_Credit_Taken)
# Savings Account

oldvaluesSA <- c("A61", "A62", "A63", "A64", "A65")
newvaluesSA <- factor(c("low", "minimal", "medium", "high", "none" ))
data$Savings_Acc <- newvaluesSA[ match(data$Savings_Acc, oldvaluesSA)]
data$Savings_Acc=as.factor(data$Savings_Acc)
# Years at present employer

oldvaluesYPE <- c("A71", "A72", "A73", "A74", "A75")
newvaluesYPE <- factor(c("unemploed", "less/year", "1to4", "4to7", "more/7" ))
data$Years_At_Present_Employment <- newvaluesYPE[ match(data$Years_At_Present_Employment, oldvaluesYPE)]
data$Years_At_Present_Employment=as.factor(data$Years_At_Present_Employment)
# Personal Status

oldvaluesPS <- c("A91", "A92", "A93", "A94") 
newvaluesPS <- factor(c("male/divorce/sep", "female/divorced/sep/married", "male/single", "male/married/widowed"))
data$Marital_Status_Gender <- newvaluesPS[ match(data$Marital_Status_Gender, oldvaluesPS)]
data$Marital_Status_Gender=as.factor(data$Marital_Status_Gender)
# Other Debtors Guarantors

oldvaluesODG <- c("A101", "A102", "A103") 
newvaluesODG <- factor(c("none", "co-applicant", "guarantor"))
data$Other_Debtors_Guarantors <- newvaluesODG[ match(data$Other_Debtors_Guarantors, oldvaluesODG)]
data$Other_Debtors_Guarantors=as.factor(data$Other_Debtors_Guarantors)
# Property
oldvaluesP <- c("A121", "A122", "A123", "A124") 
newvaluesP <- factor(c("real/estate", "build/soc/lifeinsurance", "car/other", "no/property"))
data$Property <- newvaluesP[ match(data$Property, oldvaluesP)]
data$Property=as.factor(data$Property)
#Other Inst Plans

oldvaluesOIP<- c("A141", "A142", "A143") 
newvaluesOIP <- factor(c("bank", "stores", "none"))
data$Other_Inst_Plans <- newvaluesOIP[ match(data$Other_Inst_Plans, oldvaluesOIP)]
data$Other_Inst_Plans=as.factor(data$Other_Inst_Plans)
# Housing

oldvaluesH <- c("A151", "A152", "A153") 
newvaluesH <- factor(c("rent", "owned", "free"))
data$Housing<- newvaluesH[ match(data$Housing, oldvaluesH)]
data$Other_Inst_Plans=as.factor(data$Other_Inst_Plans)
# Jobs
oldvaluesJ <- c("A171", "A172", "A173", "A174") 
newvaluesJ <- factor(c("unemployed/unskilled/non-resident", "unskilled/resident", "skilled/employee", "management/highly/qualified"))
data$Job <- newvaluesJ[ match(data$Job, oldvaluesJ)]
data$Job=as.factor(data$Job)
# Telephone
oldvaluesT <- c("A191", "A192") 
newvaluesT<- factor(c("no", "yes"))
data$Telephone<- newvaluesT[ match(data$Telephone, oldvaluesT)]
data$Telephone=as.factor(data$Telephone)
# Foreign Worker

oldvaluesFW <- c("A201", "A202") 
newvaluesFW<- factor(c("yes", "no"))
data$Foreign_Worker<- newvaluesFW[ match(data$Foreign_Worker, oldvaluesFW)]
data$Foreign_Worker=as.factor(data$Foreign_Worker)

Look again for missing values since maybe through recoding the NA has sneak into the data. Yet, I do not believe that I double check. Additionally, it is observed that we have interger and factor variables.

str(data)

## 'data.frame':    5000 obs. of  21 variables:
##  $ Status_Checking_Acc        : Factor w/ 4 levels "high","low","medium",..: 2 3 4 2 2 4 4 3 4 3 ...
##  $ Duration_in_Months         : int  6 48 12 42 24 36 24 36 12 30 ...
##  $ Credit_History             : Factor w/ 5 levels "delayed/past",..: 2 5 2 5 1 5 5 5 5 2 ...
##  $ Purposre_Credit_Taken      : Factor w/ 11 levels "bussiness","domestic/appliances",..: 7 7 3 4 5 3 4 10 7 5 ...
##  $ Credit_Amount              : int  1169 5951 2096 7882 4870 9055 2835 6948 3059 5234 ...
##  $ Savings_Acc                : Factor w/ 5 levels "high","low","medium",..: 5 2 2 2 2 5 3 2 1 2 ...
##  $ Years_At_Present_Employment: Factor w/ 5 levels "1to4","4to7",..: 4 1 2 2 1 1 4 1 2 5 ...
##  $ Inst_Rt_Income             : int  4 2 2 2 3 2 3 2 2 4 ...
##  $ Marital_Status_Gender      : Factor w/ 4 levels "female/divorced/sep/married",..: 4 1 4 4 4 4 4 4 2 3 ...
##  $ Other_Debtors_Guarantors   : Factor w/ 3 levels "co-applicant",..: 3 3 3 2 3 3 3 3 3 3 ...
##  $ Current_Address_Yrs        : int  4 2 3 4 4 4 4 2 4 2 ...
##  $ Property                   : Factor w/ 4 levels "build/soc/lifeinsurance",..: 4 4 4 1 3 3 1 2 4 2 ...
##  $ Age                        : int  67 22 49 45 53 35 53 35 61 28 ...
##  $ Other_Inst_Plans           : Factor w/ 3 levels "bank","none",..: 2 2 2 2 2 2 2 2 2 2 ...
##  $ Housing                    : Factor w/ 3 levels "free","owned",..: 2 2 2 1 1 1 2 3 2 2 ...
##  $ Num_CC                     : int  2 1 1 1 2 1 1 1 1 2 ...
##  $ Job                        : Factor w/ 4 levels "management/highly/qualified",..: 2 2 4 2 2 4 2 1 4 1 ...
##  $ Dependents                 : int  1 1 2 2 2 2 1 1 1 1 ...
##  $ Telephone                  : Factor w/ 2 levels "no","yes": 2 1 1 1 1 2 1 2 1 1 ...
##  $ Foreign_Worker             : Factor w/ 2 levels "no","yes": 2 2 2 2 2 2 2 2 2 2 ...
##  $ Default_On_Payment         : Factor w/ 2 levels "not.defaulted",..: 1 1 1 1 2 1 1 1 1 2 ...

perc.miss.case<-sapply(data, function(col) sum(is.na(col))/nrow(data))
perc.miss.case

##         Status_Checking_Acc          Duration_in_Months 
##                           0                           0 
##              Credit_History       Purposre_Credit_Taken 
##                           0                           0 
##               Credit_Amount                 Savings_Acc 
##                           0                           0 
## Years_At_Present_Employment              Inst_Rt_Income 
##                           0                           0 
##       Marital_Status_Gender    Other_Debtors_Guarantors 
##                           0                           0 
##         Current_Address_Yrs                    Property 
##                           0                           0 
##                         Age            Other_Inst_Plans 
##                           0                           0 
##                     Housing                      Num_CC 
##                           0                           0 
##                         Job                  Dependents 
##                           0                           0 
##                   Telephone              Foreign_Worker 
##                           0                           0 
##          Default_On_Payment 
##                           0

Training and test sets created stratified random sampling.

library(caret)
set.seed(22)
split1 <- createDataPartition(data$Default_On_Payment, p = .50)[[1]]
training <- data[split1,]
testing  <- data[-split1,]

Checking for class imbalances. It is observed that we have a class imbalance, that we have 30 % versus 70 %. We have to apply some functions to balance the classes, otherwise the we gonna get bad performances.

prop.table(table(training$Default_On_Payment))

## 
## not.defaulted     defaulted 
##     0.7009196     0.2990804

I have chosen to apply a smote function. There are more functions with the purpose of balancing the classes. e.g. rose, down, up. Yet, smote give me the best performance as I have tried it on this dataset. For now I am gonna apply the smote function for the sake of jumping to the next section.

library(DMwR)

## Loading required package: grid

## 
## Attaching package: 'DMwR'

## The following object is masked from 'package:plyr':
## 
##     join

set.seed(2)
smote_train <- SMOTE(Default_On_Payment~., data = training, perc.over = 100, perc.under =200)

table(smote_train$Default_On_Payment) # we see now that we have the classes balanced. 

## 
## not.defaulted     defaulted 
##          1496          1496

Finally, the last stage of preprocessing is centering and scalling.

Centering and scaling.Determine a predictor set without highly sparse and unbalanced distribution. One important observation - all the transformations are applied only on the training datasets.

preProcValues <- preProcess(smote_train[, -21],
                            method = c("center", "scale", "YeoJohnson", "nzv"))

# Storing the transformed values into a new dataset called transformed. 
transformed <- predict(preProcValues, newdata=training)

II. Applying Machine Learning. A classification issue.

After preprocessing we get into the actual datascience task. Applying machine learning in order to get some patterns. Nevertheless we create some functions to get different performances. Accuracy, Sensitivity, Specificity, Kappa.

# Obtain different perfomances measures, two wrapper functions
# For Accuracy, Kappa, the area under the ROC curve, 
# sensitivity and specificity
fiveStats <- function (...)c(twoClassSummary(...),
                             defaultSummary(...))

# Everything but the area under the ROC curv
fourStats <- function(data, lev=levels(data$obs), model =NULL)
{
  
  accKapp <- postResample(data[, "pred"], data[, "obs"])
  out<- c(accKapp,
          sensitivity(data[,"pred"], data[,"obs"], lev[1]),
          specificity(data[,"pred"], data[,"obs"], lev[2]))
  names(out)[3:4] <- c("Sens", "Spec")
  out
}

Creating a data frame to collect model’s performance. This is done for estethic purposes not that is necessary.

table_perf = data.frame(model=character(0),
                        auc=numeric(0),
                        accuracy=numeric(0),
                        sensitivity=numeric(0),
                        specificity=numeric(0),
                        kappa=numeric(0),
                        stringsAsFactors = FALSE)

A control function is developed

ctrl <- trainControl(method = "cv", 
                     number = 10, 
                     repeats = 4,
                     classProbs = TRUE,
                     summaryFunction = fiveStats,
                     verboseIter = TRUE,
                     allowParallel = TRUE)

Random Forest

I choose the same seed for the next models I have developed bellow.

set.seed(22)

rf.Fit <-train(Default_On_Payment~., data=transformed,
               method = "rf",
               trControl =ctrl,
               ntree = 1500,
               tuneLength= 5,
               metric="ROC")

## Loading required package: randomForest

## randomForest 4.6-12

## Type rfNews() to see new features/changes/bug fixes.

## 
## Attaching package: 'randomForest'

## The following object is masked from 'package:dplyr':
## 
##     combine

## The following object is masked from 'package:ggplot2':
## 
##     margin

## + Fold01: mtry= 2 
## - Fold01: mtry= 2 
## + Fold01: mtry=13 
## - Fold01: mtry=13 
## + Fold01: mtry=25 
## - Fold01: mtry=25 
## + Fold01: mtry=37 
## - Fold01: mtry=37 
## + Fold01: mtry=49 
## - Fold01: mtry=49 
## + Fold02: mtry= 2 
## - Fold02: mtry= 2 
## + Fold02: mtry=13 
## - Fold02: mtry=13 
## + Fold02: mtry=25 
## - Fold02: mtry=25 
## + Fold02: mtry=37 
## - Fold02: mtry=37 
## + Fold02: mtry=49 
## - Fold02: mtry=49 
## + Fold03: mtry= 2 
## - Fold03: mtry= 2 
## + Fold03: mtry=13 
## - Fold03: mtry=13 
## + Fold03: mtry=25 
## - Fold03: mtry=25 
## + Fold03: mtry=37 
## - Fold03: mtry=37 
## + Fold03: mtry=49 
## - Fold03: mtry=49 
## + Fold04: mtry= 2 
## - Fold04: mtry= 2 
## + Fold04: mtry=13 
## - Fold04: mtry=13 
## + Fold04: mtry=25 
## - Fold04: mtry=25 
## + Fold04: mtry=37 
## - Fold04: mtry=37 
## + Fold04: mtry=49 
## - Fold04: mtry=49 
## + Fold05: mtry= 2 
## - Fold05: mtry= 2 
## + Fold05: mtry=13 
## - Fold05: mtry=13 
## + Fold05: mtry=25 
## - Fold05: mtry=25 
## + Fold05: mtry=37 
## - Fold05: mtry=37 
## + Fold05: mtry=49 
## - Fold05: mtry=49 
## + Fold06: mtry= 2 
## - Fold06: mtry= 2 
## + Fold06: mtry=13 
## - Fold06: mtry=13 
## + Fold06: mtry=25 
## - Fold06: mtry=25 
## + Fold06: mtry=37 
## - Fold06: mtry=37 
## + Fold06: mtry=49 
## - Fold06: mtry=49 
## + Fold07: mtry= 2 
## - Fold07: mtry= 2 
## + Fold07: mtry=13 
## - Fold07: mtry=13 
## + Fold07: mtry=25 
## - Fold07: mtry=25 
## + Fold07: mtry=37 
## - Fold07: mtry=37 
## + Fold07: mtry=49 
## - Fold07: mtry=49 
## + Fold08: mtry= 2 
## - Fold08: mtry= 2 
## + Fold08: mtry=13 
## - Fold08: mtry=13 
## + Fold08: mtry=25 
## - Fold08: mtry=25 
## + Fold08: mtry=37 
## - Fold08: mtry=37 
## + Fold08: mtry=49 
## - Fold08: mtry=49 
## + Fold09: mtry= 2 
## - Fold09: mtry= 2 
## + Fold09: mtry=13 
## - Fold09: mtry=13 
## + Fold09: mtry=25 
## - Fold09: mtry=25 
## + Fold09: mtry=37 
## - Fold09: mtry=37 
## + Fold09: mtry=49 
## - Fold09: mtry=49 
## + Fold10: mtry= 2 
## - Fold10: mtry= 2 
## + Fold10: mtry=13 
## - Fold10: mtry=13 
## + Fold10: mtry=25 
## - Fold10: mtry=25 
## + Fold10: mtry=37 
## - Fold10: mtry=37 
## + Fold10: mtry=49 
## - Fold10: mtry=49 
## Aggregating results
## Selecting tuning parameters
## Fitting mtry = 13 on full training set

rf.Fit

## Random Forest 
## 
## 2501 samples
##   20 predictor
##    2 classes: 'not.defaulted', 'defaulted' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 2252, 2251, 2251, 2251, 2251, 2250, ... 
## Resampling results across tuning parameters:
## 
##   mtry  ROC        Sens       Spec       Accuracy   Kappa    
##    2    0.9798237  0.9982857  0.5241081  0.8564553  0.6027554
##   13    0.9978379  0.9914318  0.9544505  0.9803951  0.9528488
##   25    0.9973407  0.9902890  0.9598198  0.9811951  0.9549345
##   37    0.9970937  0.9897175  0.9598198  0.9807951  0.9539933
##   49    0.9964526  0.9868701  0.9624865  0.9795999  0.9512452
## 
## ROC was used to select the optimal model using  the largest value.
## The final value used for the model was mtry = 13.

Variable Importance gives us the most important variables, the first 10, as I have chosen, in predicting the loan default. We see that Credit Amount, Duration in Months and Age, etc, are the predictors in predicting the loan default. In other words, these 10 variables are to be taken into account in assessing an individual on whether he will laon default.

rf.varImp <- varImp(rf.Fit)

plot(rf.varImp, 10)

Testing the predictions on test dataset. As it can be observed we got AUC is 81% with accuracy of 61 % which reveals a weak performance.

evalResults.RF <- predict(rf.Fit, testing, type="prob")[,1]
predict_rf = ifelse(evalResults.RF<0.5, "defaulted", "not.defaulted")
cmRF<- confusionMatrix(predict_rf, testing$Default_On_Payment, positive="defaulted")

## Warning in confusionMatrix.default(predict_rf, testing
## $Default_On_Payment, : Levels are not in the same order for reference and
## data. Refactoring data to match.

evalResults <- data.frame(Default_On_Payment=testing$Default_On_Payment)

library(pROC)

## Type 'citation("pROC")' for a citation.

## 
## Attaching package: 'pROC'

## The following objects are masked from 'package:stats':
## 
##     cov, smooth, var

rf.pROC <- pROC::roc(evalResults$Default_On_Payment, evalResults.RF)
auc_curve=auc(rf.pROC)
auc_curve

## Area under the curve: 0.8133

table_perf[1,] = c("Random Forest",
                   round(auc_curve,3),
                   as.numeric(round(cmRF$overall["Accuracy"],3)),
                   as.numeric(round(cmRF$byClass["Sensitivity"],3)),
                   as.numeric(round(cmRF$byClass["Specificity"],3)),
                   as.numeric(round(cmRF$overall["Kappa"],3)))
table_perf

##           model   auc accuracy sensitivity specificity kappa
## 1 Random Forest 0.813    0.625        0.94        0.49 0.324

Tuning the Random Forest model as the performances were weak.

set.seed(22)
mtryValues <- c(1,3,5,7,10)

rf.Fittuned <-train(Default_On_Payment~., data=transformed,
                    method = "rf",
                    trControl =ctrl, 
                    ntree = 1500,
                    tuneGrid = data.frame(.mtry = mtryValues),
                    tuneLength= 5, 
                    metric="ROC")

## + Fold01: mtry= 1 
## - Fold01: mtry= 1 
## + Fold01: mtry= 3 
## - Fold01: mtry= 3 
## + Fold01: mtry= 5 
## - Fold01: mtry= 5 
## + Fold01: mtry= 7 
## - Fold01: mtry= 7 
## + Fold01: mtry=10 
## - Fold01: mtry=10 
## + Fold02: mtry= 1 
## - Fold02: mtry= 1 
## + Fold02: mtry= 3 
## - Fold02: mtry= 3 
## + Fold02: mtry= 5 
## - Fold02: mtry= 5 
## + Fold02: mtry= 7 
## - Fold02: mtry= 7 
## + Fold02: mtry=10 
## - Fold02: mtry=10 
## + Fold03: mtry= 1 
## - Fold03: mtry= 1 
## + Fold03: mtry= 3 
## - Fold03: mtry= 3 
## + Fold03: mtry= 5 
## - Fold03: mtry= 5 
## + Fold03: mtry= 7 
## - Fold03: mtry= 7 
## + Fold03: mtry=10 
## - Fold03: mtry=10 
## + Fold04: mtry= 1 
## - Fold04: mtry= 1 
## + Fold04: mtry= 3 
## - Fold04: mtry= 3 
## + Fold04: mtry= 5 
## - Fold04: mtry= 5 
## + Fold04: mtry= 7 
## - Fold04: mtry= 7 
## + Fold04: mtry=10 
## - Fold04: mtry=10 
## + Fold05: mtry= 1 
## - Fold05: mtry= 1 
## + Fold05: mtry= 3 
## - Fold05: mtry= 3 
## + Fold05: mtry= 5 
## - Fold05: mtry= 5 
## + Fold05: mtry= 7 
## - Fold05: mtry= 7 
## + Fold05: mtry=10 
## - Fold05: mtry=10 
## + Fold06: mtry= 1 
## - Fold06: mtry= 1 
## + Fold06: mtry= 3 
## - Fold06: mtry= 3 
## + Fold06: mtry= 5 
## - Fold06: mtry= 5 
## + Fold06: mtry= 7 
## - Fold06: mtry= 7 
## + Fold06: mtry=10 
## - Fold06: mtry=10 
## + Fold07: mtry= 1 
## - Fold07: mtry= 1 
## + Fold07: mtry= 3 
## - Fold07: mtry= 3 
## + Fold07: mtry= 5 
## - Fold07: mtry= 5 
## + Fold07: mtry= 7 
## - Fold07: mtry= 7 
## + Fold07: mtry=10 
## - Fold07: mtry=10 
## + Fold08: mtry= 1 
## - Fold08: mtry= 1 
## + Fold08: mtry= 3 
## - Fold08: mtry= 3 
## + Fold08: mtry= 5 
## - Fold08: mtry= 5 
## + Fold08: mtry= 7 
## - Fold08: mtry= 7 
## + Fold08: mtry=10 
## - Fold08: mtry=10 
## + Fold09: mtry= 1 
## - Fold09: mtry= 1 
## + Fold09: mtry= 3 
## - Fold09: mtry= 3 
## + Fold09: mtry= 5 
## - Fold09: mtry= 5 
## + Fold09: mtry= 7 
## - Fold09: mtry= 7 
## + Fold09: mtry=10 
## - Fold09: mtry=10 
## + Fold10: mtry= 1 
## - Fold10: mtry= 1 
## + Fold10: mtry= 3 
## - Fold10: mtry= 3 
## + Fold10: mtry= 5 
## - Fold10: mtry= 5 
## + Fold10: mtry= 7 
## - Fold10: mtry= 7 
## + Fold10: mtry=10 
## - Fold10: mtry=10 
## Aggregating results
## Selecting tuning parameters
## Fitting mtry = 10 on full training set

rf.Fittuned

## Random Forest 
## 
## 2501 samples
##   20 predictor
##    2 classes: 'not.defaulted', 'defaulted' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 2252, 2251, 2251, 2251, 2251, 2250, ... 
## Resampling results across tuning parameters:
## 
##   mtry  ROC        Sens       Spec       Accuracy   Kappa    
##    1    0.9092557  1.0000000  0.0000000  0.7009202  0.0000000
##    3    0.9944001  0.9942890  0.8302162  0.9452190  0.8627295
##    5    0.9976268  0.9948604  0.9451171  0.9800015  0.9515881
##    7    0.9979373  0.9937175  0.9504505  0.9807983  0.9536086
##   10    0.9979717  0.9914351  0.9531171  0.9799967  0.9518435
## 
## ROC was used to select the optimal model using  the largest value.
## The final value used for the model was mtry = 10.

Check the variables that are having impact on the model. Other words, I check what variable is having an impact on loan default. As I have already applied this function previosly, I am looking to see whether the same variables are impacting on the tuned model. Yes, the same variables are coming as important in whether an individual will default or not.

rf.varImpt <- varImp(rf.Fittuned)
plot(rf.varImpt, 10)

Test the results and calculating the ROC in order to get the Area Under Curve

evalResults.RFt <- predict(rf.Fittuned, testing, type="prob")[,1]
predict_rft = ifelse(evalResults.RFt<0.5, "defaulted", "not.defaulted")
cmRFt<- confusionMatrix(predict_rft, testing$Default_On_Payment, positive="defaulted")

## Warning in confusionMatrix.default(predict_rft, testing
## $Default_On_Payment, : Levels are not in the same order for reference and
## data. Refactoring data to match.

evalResults <- data.frame(Default_On_Payment=testing$Default_On_Payment)

library(pROC)
rf.pROCt <- pROC::roc(evalResults$Default_On_Payment, evalResults.RFt)
auc_curve=auc(rf.pROCt)
auc_curve

## Area under the curve: 0.8358

Storing the performances and checking the results. Was tuning the model a better idea? We can see that tuning the model has brought better result for all measurements.

table_perf[2,] = c("Random Forest Tuned",
                   round(auc_curve,3),
                   as.numeric(round(cmRFt$overall["Accuracy"],3)),
                   as.numeric(round(cmRFt$byClass["Sensitivity"],3)),
                   as.numeric(round(cmRFt$byClass["Specificity"],3)),
                   as.numeric(round(cmRFt$overall["Kappa"],3)))
table_perf

##                 model   auc accuracy sensitivity specificity kappa
## 1       Random Forest 0.813    0.625        0.94        0.49 0.324
## 2 Random Forest Tuned 0.836    0.655       0.945       0.531 0.367

Gradient Boosted

I apply Gradient Boosted and tuned it straight away just as I did at Random Forest.

C5.0.Grid <-expand.grid(interaction.depth = c(1,5,9),
                        n.trees = (1:30)*50,
                        shrinkage=0.1,
                        n.minosinnode =20)
set.seed(22)
C5.0.Fit <- train(Default_On_Payment~., 
                  data= transformed, 
                  method= "C5.0", 
                  tuneLength=30,
                  Grid =C5.0.Grid, 
                  metric="ROC",
                  trControl = ctrl)

## Loading required package: C50

## + Fold01: model=tree, winnow=FALSE, trials=100 
## - Fold01: model=tree, winnow=FALSE, trials=100 
## + Fold01: model=tree, winnow= TRUE, trials=100 
## - Fold01: model=tree, winnow= TRUE, trials=100 
## + Fold01: model=rules, winnow=FALSE, trials=100 
## - Fold01: model=rules, winnow=FALSE, trials=100 
## + Fold01: model=rules, winnow= TRUE, trials=100 
## - Fold01: model=rules, winnow= TRUE, trials=100 
## + Fold02: model=tree, winnow=FALSE, trials=100 
## - Fold02: model=tree, winnow=FALSE, trials=100 
## + Fold02: model=tree, winnow= TRUE, trials=100 
## - Fold02: model=tree, winnow= TRUE, trials=100 
## + Fold02: model=rules, winnow=FALSE, trials=100 
## - Fold02: model=rules, winnow=FALSE, trials=100 
## + Fold02: model=rules, winnow= TRUE, trials=100 
## - Fold02: model=rules, winnow= TRUE, trials=100 
## + Fold03: model=tree, winnow=FALSE, trials=100 
## - Fold03: model=tree, winnow=FALSE, trials=100 
## + Fold03: model=tree, winnow= TRUE, trials=100 
## - Fold03: model=tree, winnow= TRUE, trials=100 
## + Fold03: model=rules, winnow=FALSE, trials=100 
## - Fold03: model=rules, winnow=FALSE, trials=100 
## + Fold03: model=rules, winnow= TRUE, trials=100 
## - Fold03: model=rules, winnow= TRUE, trials=100 
## + Fold04: model=tree, winnow=FALSE, trials=100 
## - Fold04: model=tree, winnow=FALSE, trials=100 
## + Fold04: model=tree, winnow= TRUE, trials=100 
## - Fold04: model=tree, winnow= TRUE, trials=100 
## + Fold04: model=rules, winnow=FALSE, trials=100 
## - Fold04: model=rules, winnow=FALSE, trials=100 
## + Fold04: model=rules, winnow= TRUE, trials=100 
## - Fold04: model=rules, winnow= TRUE, trials=100 
## + Fold05: model=tree, winnow=FALSE, trials=100 
## - Fold05: model=tree, winnow=FALSE, trials=100 
## + Fold05: model=tree, winnow= TRUE, trials=100 
## - Fold05: model=tree, winnow= TRUE, trials=100 
## + Fold05: model=rules, winnow=FALSE, trials=100 
## - Fold05: model=rules, winnow=FALSE, trials=100 
## + Fold05: model=rules, winnow= TRUE, trials=100 
## - Fold05: model=rules, winnow= TRUE, trials=100 
## + Fold06: model=tree, winnow=FALSE, trials=100 
## - Fold06: model=tree, winnow=FALSE, trials=100 
## + Fold06: model=tree, winnow= TRUE, trials=100 
## - Fold06: model=tree, winnow= TRUE, trials=100 
## + Fold06: model=rules, winnow=FALSE, trials=100 
## - Fold06: model=rules, winnow=FALSE, trials=100 
## + Fold06: model=rules, winnow= TRUE, trials=100 
## - Fold06: model=rules, winnow= TRUE, trials=100 
## + Fold07: model=tree, winnow=FALSE, trials=100 
## - Fold07: model=tree, winnow=FALSE, trials=100 
## + Fold07: model=tree, winnow= TRUE, trials=100 
## - Fold07: model=tree, winnow= TRUE, trials=100 
## + Fold07: model=rules, winnow=FALSE, trials=100 
## - Fold07: model=rules, winnow=FALSE, trials=100 
## + Fold07: model=rules, winnow= TRUE, trials=100 
## - Fold07: model=rules, winnow= TRUE, trials=100 
## + Fold08: model=tree, winnow=FALSE, trials=100 
## - Fold08: model=tree, winnow=FALSE, trials=100 
## + Fold08: model=tree, winnow= TRUE, trials=100 
## - Fold08: model=tree, winnow= TRUE, trials=100 
## + Fold08: model=rules, winnow=FALSE, trials=100 
## - Fold08: model=rules, winnow=FALSE, trials=100 
## + Fold08: model=rules, winnow= TRUE, trials=100 
## - Fold08: model=rules, winnow= TRUE, trials=100 
## + Fold09: model=tree, winnow=FALSE, trials=100 
## - Fold09: model=tree, winnow=FALSE, trials=100 
## + Fold09: model=tree, winnow= TRUE, trials=100 
## - Fold09: model=tree, winnow= TRUE, trials=100 
## + Fold09: model=rules, winnow=FALSE, trials=100 
## - Fold09: model=rules, winnow=FALSE, trials=100 
## + Fold09: model=rules, winnow= TRUE, trials=100 
## - Fold09: model=rules, winnow= TRUE, trials=100 
## + Fold10: model=tree, winnow=FALSE, trials=100 
## - Fold10: model=tree, winnow=FALSE, trials=100 
## + Fold10: model=tree, winnow= TRUE, trials=100 
## - Fold10: model=tree, winnow= TRUE, trials=100 
## + Fold10: model=rules, winnow=FALSE, trials=100 
## - Fold10: model=rules, winnow=FALSE, trials=100 
## + Fold10: model=rules, winnow= TRUE, trials=100 
## - Fold10: model=rules, winnow= TRUE, trials=100 
## Aggregating results
## Selecting tuning parameters
## Fitting trials = 100, model = rules, winnow = FALSE on full training set

C5.0.Fit

## C5.0 
## 
## 2501 samples
##   20 predictor
##    2 classes: 'not.defaulted', 'defaulted' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 2252, 2251, 2251, 2251, 2251, 2250, ... 
## Resampling results across tuning parameters:
## 
##   model  winnow  trials  ROC        Sens       Spec       Accuracy 
##   rules  FALSE     1     0.9037030  0.9372338  0.8168829  0.9012364
##   rules  FALSE    10     0.9922442  0.9834481  0.9665225  0.9784031
##   rules  FALSE    20     0.9963795  0.9851558  0.9691892  0.9803999
##   rules  FALSE    30     0.9965868  0.9868734  0.9678739  0.9811983
##   rules  FALSE    40     0.9966844  0.9868701  0.9692252  0.9816015
##   rules  FALSE    50     0.9968128  0.9874416  0.9692252  0.9820031
##   rules  FALSE    60     0.9969336  0.9868701  0.9692252  0.9816015
##   rules  FALSE    70     0.9969095  0.9868701  0.9665586  0.9808015
##   rules  FALSE    80     0.9967720  0.9868701  0.9638559  0.9799983
##   rules  FALSE    90     0.9970621  0.9874416  0.9638559  0.9803999
##   rules  FALSE   100     0.9971843  0.9868701  0.9651892  0.9803983
##   rules   TRUE     1     0.8901632  0.9446786  0.8048468  0.9028540
##   rules   TRUE    10     0.9921599  0.9845942  0.9625586  0.9780063
##   rules   TRUE    20     0.9954819  0.9857305  0.9678739  0.9804015
##   rules   TRUE    30     0.9961717  0.9868766  0.9679099  0.9812047
##   rules   TRUE    40     0.9957803  0.9863084  0.9732793  0.9824047
##   rules   TRUE    50     0.9965184  0.9863052  0.9692432  0.9812015
##   rules   TRUE    60     0.9966695  0.9868734  0.9678919  0.9811999
##   rules   TRUE    70     0.9967013  0.9880130  0.9665225  0.9815967
##   rules   TRUE    80     0.9969523  0.9868701  0.9678559  0.9811951
##   rules   TRUE    90     0.9968452  0.9874416  0.9665045  0.9811951
##   rules   TRUE   100     0.9969681  0.9874416  0.9665225  0.9811967
##   tree   FALSE     1     0.9449244  0.9332435  0.7994955  0.8932396
##   tree   FALSE    10     0.9940218  0.9885844  0.9544865  0.9783935
##   tree   FALSE    20     0.9950953  0.9885877  0.9518018  0.9775983
##   tree   FALSE    30     0.9959870  0.9897273  0.9558018  0.9795967
##   tree   FALSE    40     0.9960810  0.9902955  0.9571532  0.9803983
##   tree   FALSE    50     0.9962704  0.9902922  0.9598378  0.9811983
##   tree   FALSE    60     0.9966688  0.9885844  0.9612072  0.9804047
##   tree   FALSE    70     0.9967302  0.9891526  0.9612072  0.9808031
##   tree   FALSE    80     0.9968743  0.9902955  0.9625586  0.9820031
##   tree   FALSE    90     0.9968204  0.9897240  0.9612072  0.9812031
##   tree   FALSE   100     0.9967423  0.9902955  0.9625586  0.9820031
##   tree    TRUE     1     0.9388215  0.9446851  0.8115315  0.9048620
##   tree    TRUE    10     0.9932946  0.9851591  0.9545225  0.9759983
##   tree    TRUE    20     0.9954474  0.9880097  0.9612072  0.9799983
##   tree    TRUE    30     0.9967905  0.9902955  0.9625225  0.9819999
##   tree    TRUE    40     0.9968686  0.9908636  0.9624865  0.9823951
##   tree    TRUE    50     0.9964540  0.9908636  0.9611532  0.9819951
##   tree    TRUE    60     0.9964079  0.9897240  0.9598018  0.9807935
##   tree    TRUE    70     0.9964465  0.9897240  0.9611532  0.9811951
##   tree    TRUE    80     0.9961713  0.9902955  0.9625225  0.9819999
##   tree    TRUE    90     0.9963999  0.9902955  0.9625225  0.9819999
##   tree    TRUE   100     0.9965376  0.9891526  0.9638559  0.9815983
##   Kappa    
##   0.7610429
##   0.9486241
##   0.9533082
##   0.9551759
##   0.9561359
##   0.9570996
##   0.9561495
##   0.9542080
##   0.9522408
##   0.9531906
##   0.9531965
##   0.7634879
##   0.9474698
##   0.9532943
##   0.9551353
##   0.9581282
##   0.9552261
##   0.9551793
##   0.9560477
##   0.9551547
##   0.9550891
##   0.9551062
##   0.7411864
##   0.9480619
##   0.9461199
##   0.9510083
##   0.9529150
##   0.9549390
##   0.9530794
##   0.9540486
##   0.9569116
##   0.9549971
##   0.9569116
##   0.7687778
##   0.9426168
##   0.9523114
##   0.9568977
##   0.9578095
##   0.9568238
##   0.9539231
##   0.9549267
##   0.9568617
##   0.9568617
##   0.9559767
## 
## ROC was used to select the optimal model using  the largest value.
## The final values used for the model were trials = 100, model = rules
##  and winnow = FALSE.

Looking the the variables that show importance in whether an individual will or not default. Here the Age, Income, Purpose, Existing Credit History, Duration in Months, Credit Amount, etc have all equal importance in whether a person will default or not.Actually, all variables have almost 100 % importance.

varImpC5.O <- varImp(C5.0.Fit)

plot(varImpC5.O, 20)

evalResults.C5.0 <- predict(C5.0.Fit, testing, type="prob")[,1]
predict.C5.0 = ifelse(evalResults.C5.0<0.5, "defaulted", "not.defaulted")
cm.C5.0<- confusionMatrix(predict.C5.0, testing$Default_On_Payment, positive="defaulted")

## Warning in confusionMatrix.default(predict.C5.0, testing
## $Default_On_Payment, : Levels are not in the same order for reference and
## data. Refactoring data to match.

evalResults<- data.frame(Default_On_Payment=testing$Default_On_Payment)

library(pROC)
C5.O.pROC <- pROC::roc(evalResults$Default_On_Payment, evalResults.C5.0)
auc_curve=auc(C5.O.pROC)
auc_curve

## Area under the curve: 0.8239

Checking the performances obtained in the Cubist model against the other models developed above. Random Forest(tuned) does well compared to Cubist model

table_perf[3,] = c("C5.0 - Cubist",
                   round(auc_curve,3),
                   as.numeric(round(cm.C5.0$overall["Accuracy"],3)),
                   as.numeric(round(cm.C5.0$byClass["Sensitivity"],3)),
                   as.numeric(round(cm.C5.0$byClass["Specificity"],3)),
                   as.numeric(round(cm.C5.0$overall["Kappa"],3)))
table_perf

##                 model   auc accuracy sensitivity specificity kappa
## 1       Random Forest 0.813    0.625        0.94        0.49 0.324
## 2 Random Forest Tuned 0.836    0.655       0.945       0.531 0.367
## 3       C5.0 - Cubist 0.824    0.543       0.976       0.359 0.235

xgbBoosting

At this phase I am not looking anymore for the varImp since I have already looked for it in the pref

set.seed(22)
xgbGrid <- expand.grid(
  eta = 0.12,
  max_depth = 10,
  nrounds = 400,
  gamma = 0,               #default=0
  colsample_bytree = 1,    #default=1
  min_child_weight = 1     #default=1
)

XGBoost.Fit <- train(Default_On_Payment ~., 
                     data= transformed, 
                     method= "xgbTree", 
                     metric="ROC",
                     #tuneGrid = xgbGrid, does not work ...
                     trControl = ctrl, 
                     nthread = 3)

## Loading required package: xgboost

## 
## Attaching package: 'xgboost'

## The following object is masked from 'package:dplyr':
## 
##     slice

## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold01: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold02: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold03: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold04: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold05: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold06: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold07: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold08: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold09: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.3, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=1, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=2, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.6, min_child_weight=1, subsample=1.00, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.50, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=0.75, nrounds=150 
## + Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## - Fold10: eta=0.4, max_depth=3, gamma=0, colsample_bytree=0.8, min_child_weight=1, subsample=1.00, nrounds=150 
## Aggregating results
## Selecting tuning parameters
## Fitting nrounds = 150, max_depth = 3, eta = 0.4, gamma = 0, colsample_bytree = 0.8, min_child_weight = 1, subsample = 0.75 on full training set

XGBoost.Fit 

## eXtreme Gradient Boosting 
## 
## 2501 samples
##   20 predictor
##    2 classes: 'not.defaulted', 'defaulted' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 2252, 2251, 2251, 2251, 2251, 2250, ... 
## Resampling results across tuning parameters:
## 
##   eta  max_depth  colsample_bytree  subsample  nrounds  ROC      
##   0.3  1          0.6               0.50        50      0.8162735
##   0.3  1          0.6               0.50       100      0.8255710
##   0.3  1          0.6               0.50       150      0.8313237
##   0.3  1          0.6               0.75        50      0.8142198
##   0.3  1          0.6               0.75       100      0.8253597
##   0.3  1          0.6               0.75       150      0.8323904
##   0.3  1          0.6               1.00        50      0.8126662
##   0.3  1          0.6               1.00       100      0.8229252
##   0.3  1          0.6               1.00       150      0.8288032
##   0.3  1          0.8               0.50        50      0.8220723
##   0.3  1          0.8               0.50       100      0.8329816
##   0.3  1          0.8               0.50       150      0.8368980
##   0.3  1          0.8               0.75        50      0.8207133
##   0.3  1          0.8               0.75       100      0.8320828
##   0.3  1          0.8               0.75       150      0.8392645
##   0.3  1          0.8               1.00        50      0.8177333
##   0.3  1          0.8               1.00       100      0.8303128
##   0.3  1          0.8               1.00       150      0.8363783
##   0.3  2          0.6               0.50        50      0.8597175
##   0.3  2          0.6               0.50       100      0.8846756
##   0.3  2          0.6               0.50       150      0.8937315
##   0.3  2          0.6               0.75        50      0.8575276
##   0.3  2          0.6               0.75       100      0.8866862
##   0.3  2          0.6               0.75       150      0.9023021
##   0.3  2          0.6               1.00        50      0.8607914
##   0.3  2          0.6               1.00       100      0.8844233
##   0.3  2          0.6               1.00       150      0.9017572
##   0.3  2          0.8               0.50        50      0.8733231
##   0.3  2          0.8               0.50       100      0.9008841
##   0.3  2          0.8               0.50       150      0.9126863
##   0.3  2          0.8               0.75        50      0.8792442
##   0.3  2          0.8               0.75       100      0.9028741
##   0.3  2          0.8               0.75       150      0.9222327
##   0.3  2          0.8               1.00        50      0.8762755
##   0.3  2          0.8               1.00       100      0.9044330
##   0.3  2          0.8               1.00       150      0.9229366
##   0.3  3          0.6               0.50        50      0.8975407
##   0.3  3          0.6               0.50       100      0.9289865
##   0.3  3          0.6               0.50       150      0.9506948
##   0.3  3          0.6               0.75        50      0.9073413
##   0.3  3          0.6               0.75       100      0.9408423
##   0.3  3          0.6               0.75       150      0.9565309
##   0.3  3          0.6               1.00        50      0.9042032
##   0.3  3          0.6               1.00       100      0.9382318
##   0.3  3          0.6               1.00       150      0.9557961
##   0.3  3          0.8               0.50        50      0.9162504
##   0.3  3          0.8               0.50       100      0.9505528
##   0.3  3          0.8               0.50       150      0.9712812
##   0.3  3          0.8               0.75        50      0.9213983
##   0.3  3          0.8               0.75       100      0.9588099
##   0.3  3          0.8               0.75       150      0.9774452
##   0.3  3          0.8               1.00        50      0.9226837
##   0.3  3          0.8               1.00       100      0.9572918
##   0.3  3          0.8               1.00       150      0.9738224
##   0.4  1          0.6               0.50        50      0.8181156
##   0.4  1          0.6               0.50       100      0.8305062
##   0.4  1          0.6               0.50       150      0.8380444
##   0.4  1          0.6               0.75        50      0.8152383
##   0.4  1          0.6               0.75       100      0.8302985
##   0.4  1          0.6               0.75       150      0.8365599
##   0.4  1          0.6               1.00        50      0.8172747
##   0.4  1          0.6               1.00       100      0.8282699
##   0.4  1          0.6               1.00       150      0.8342253
##   0.4  1          0.8               0.50        50      0.8232586
##   0.4  1          0.8               0.50       100      0.8375255
##   0.4  1          0.8               0.50       150      0.8435258
##   0.4  1          0.8               0.75        50      0.8253171
##   0.4  1          0.8               0.75       100      0.8344996
##   0.4  1          0.8               0.75       150      0.8396134
##   0.4  1          0.8               1.00        50      0.8236766
##   0.4  1          0.8               1.00       100      0.8340386
##   0.4  1          0.8               1.00       150      0.8399601
##   0.4  2          0.6               0.50        50      0.8595011
##   0.4  2          0.6               0.50       100      0.8881532
##   0.4  2          0.6               0.50       150      0.9015981
##   0.4  2          0.6               0.75        50      0.8706540
##   0.4  2          0.6               0.75       100      0.8968622
##   0.4  2          0.6               0.75       150      0.9116721
##   0.4  2          0.6               1.00        50      0.8693652
##   0.4  2          0.6               1.00       100      0.8975353
##   0.4  2          0.6               1.00       150      0.9133924
##   0.4  2          0.8               0.50        50      0.8778073
##   0.4  2          0.8               0.50       100      0.9083313
##   0.4  2          0.8               0.50       150      0.9277480
##   0.4  2          0.8               0.75        50      0.8874756
##   0.4  2          0.8               0.75       100      0.9183952
##   0.4  2          0.8               0.75       150      0.9342203
##   0.4  2          0.8               1.00        50      0.8874614
##   0.4  2          0.8               1.00       100      0.9173615
##   0.4  2          0.8               1.00       150      0.9358262
##   0.4  3          0.6               0.50        50      0.9078750
##   0.4  3          0.6               0.50       100      0.9383488
##   0.4  3          0.6               0.50       150      0.9568401
##   0.4  3          0.6               0.75        50      0.9137737
##   0.4  3          0.6               0.75       100      0.9451539
##   0.4  3          0.6               0.75       150      0.9656784
##   0.4  3          0.6               1.00        50      0.9175095
##   0.4  3          0.6               1.00       100      0.9496776
##   0.4  3          0.6               1.00       150      0.9661978
##   0.4  3          0.8               0.50        50      0.9252495
##   0.4  3          0.8               0.50       100      0.9602025
##   0.4  3          0.8               0.50       150      0.9767378
##   0.4  3          0.8               0.75        50      0.9389844
##   0.4  3          0.8               0.75       100      0.9708239
##   0.4  3          0.8               0.75       150      0.9838921
##   0.4  3          0.8               1.00        50      0.9422685
##   0.4  3          0.8               1.00       100      0.9712028
##   0.4  3          0.8               1.00       150      0.9823343
##   Sens       Spec       Accuracy   Kappa    
##   0.8984610  0.4625946  0.7680933  0.3944539
##   0.8967338  0.5281802  0.7864919  0.4542845
##   0.8910097  0.5321441  0.7836807  0.4504249
##   0.9104286  0.4198198  0.7637093  0.3694137
##   0.8938766  0.5268288  0.7840982  0.4496760
##   0.8950227  0.5415495  0.7893015  0.4648920
##   0.9167110  0.4010811  0.7624996  0.3594728
##   0.8978799  0.4893874  0.7757029  0.4196130
##   0.8961623  0.5201441  0.7837014  0.4465077
##   0.9144091  0.4519459  0.7761030  0.4056785
##   0.8915942  0.5374414  0.7857014  0.4565591
##   0.8915909  0.5495495  0.7892999  0.4670371
##   0.9115649  0.4305405  0.7676981  0.3814853
##   0.8990162  0.5200721  0.7856998  0.4500289
##   0.8961558  0.5494775  0.7924934  0.4733328
##   0.9224123  0.4144865  0.7704917  0.3804449
##   0.9041591  0.5026847  0.7840934  0.4407978
##   0.9007370  0.5214595  0.7873046  0.4537760
##   0.9081396  0.5709189  0.8072856  0.5101116
##   0.9138214  0.6176577  0.8252824  0.5601460
##   0.9246688  0.6497477  0.8424585  0.6041450
##   0.9058506  0.5616396  0.8028999  0.4985179
##   0.9161136  0.6377297  0.8328761  0.5810806
##   0.9252403  0.7072613  0.8600618  0.6539806
##   0.9155519  0.5601802  0.8092855  0.5109596
##   0.9235422  0.6097117  0.8296792  0.5675427
##   0.9286688  0.6738919  0.8524682  0.6311861
##   0.9144156  0.5829369  0.8152808  0.5296124
##   0.9286688  0.6725225  0.8520554  0.6295567
##   0.9263864  0.7072252  0.8608570  0.6559881
##   0.9183929  0.5883604  0.8196776  0.5408679
##   0.9218214  0.6711892  0.8468586  0.6179538
##   0.9332273  0.7260721  0.8712555  0.6814389
##   0.9223994  0.5722883  0.8176712  0.5318751
##   0.9258279  0.6645225  0.8476762  0.6185201
##   0.9366591  0.6979099  0.8652506  0.6631007
##   0.9241104  0.6617297  0.8456633  0.6137343
##   0.9417922  0.7486847  0.8840364  0.7138636
##   0.9509253  0.7900901  0.9028252  0.7613059
##   0.9309610  0.6897477  0.8588554  0.6486838
##   0.9452370  0.7860901  0.8976444  0.7496953
##   0.9537857  0.8301982  0.9168365  0.7980766
##   0.9275227  0.6683784  0.8500554  0.6248812
##   0.9434968  0.7566486  0.8876348  0.7231942
##   0.9572078  0.8034414  0.9112348  0.7821896
##   0.9332370  0.6925405  0.8612474  0.6525724
##   0.9543506  0.7740721  0.9004284  0.7532976
##   0.9697532  0.8462703  0.9328269  0.8355553
##   0.9349513  0.7259820  0.8724475  0.6832830
##   0.9560552  0.8035135  0.9104301  0.7796177
##   0.9731786  0.8502162  0.9364142  0.8442036
##   0.9343766  0.7005766  0.8644475  0.6619282
##   0.9589286  0.8061622  0.9132396  0.7865994
##   0.9726169  0.8515856  0.9364222  0.8443200
##   0.8984416  0.5000000  0.7792982  0.4307873
##   0.8893117  0.5361261  0.7836903  0.4523083
##   0.8938669  0.5548468  0.7924855  0.4754824
##   0.8995779  0.4692973  0.7708965  0.4027251
##   0.8864513  0.5414414  0.7832903  0.4531335
##   0.8915779  0.5575315  0.7916807  0.4749083
##   0.9024610  0.4399279  0.7641109  0.3776794
##   0.8984481  0.5161261  0.7841030  0.4456457
##   0.8944545  0.5334955  0.7864998  0.4566920
##   0.9012922  0.5066306  0.7832774  0.4406194
##   0.8858734  0.5574414  0.7876839  0.4665084
##   0.8847403  0.5735676  0.7916871  0.4800446
##   0.9018604  0.4747027  0.7740950  0.4112481
##   0.8955909  0.5281081  0.7856950  0.4525971
##   0.8904448  0.5602162  0.7916887  0.4753464
##   0.9115682  0.4492072  0.7732950  0.4001165
##   0.8955974  0.5134054  0.7812982  0.4387590
##   0.8978734  0.5321622  0.7884934  0.4594807
##   0.9041494  0.5962523  0.8120919  0.5271777
##   0.9218442  0.6592432  0.8432729  0.6078379
##   0.9269805  0.6898919  0.8560602  0.6409561
##   0.9126981  0.6096757  0.8220776  0.5514320
##   0.9252370  0.6711532  0.8492617  0.6238576
##   0.9235227  0.7099099  0.8596554  0.6541113
##   0.9092760  0.5829550  0.8116840  0.5230426
##   0.9275455  0.6805225  0.8536746  0.6357567
##   0.9292500  0.7086126  0.8632666  0.6615571
##   0.9161071  0.6203423  0.8276601  0.5661506
##   0.9292468  0.6952973  0.8592586  0.6496962
##   0.9400909  0.7353874  0.8788555  0.6996905
##   0.9155617  0.6177658  0.8264920  0.5628026
##   0.9298182  0.7006847  0.8612650  0.6555956
##   0.9355357  0.7568108  0.8820635  0.7104308
##   0.9235422  0.6190450  0.8324744  0.5754507
##   0.9383766  0.7113333  0.8704555  0.6773109
##   0.9480844  0.7554234  0.8904476  0.7286512
##   0.9252403  0.6885225  0.8544570  0.6384458
##   0.9372338  0.7768108  0.8892476  0.7296228
##   0.9583539  0.8169009  0.9160477  0.7943156
##   0.9332370  0.7179820  0.8688555  0.6757565
##   0.9440747  0.7914054  0.8984396  0.7520907
##   0.9566266  0.8462342  0.9236269  0.8150026
##   0.9366656  0.7071712  0.8680442  0.6710535
##   0.9480714  0.7887207  0.9004364  0.7562176
##   0.9634805  0.8368829  0.9256205  0.8183841
##   0.9321006  0.7487748  0.8772555  0.6984771
##   0.9657630  0.8275676  0.9244333  0.8141294
##   0.9760292  0.8769369  0.9464174  0.8694917
##   0.9457987  0.7420721  0.8848555  0.7139845
##   0.9640552  0.8342342  0.9252349  0.8173782
##   0.9828766  0.8983063  0.9576190  0.8968602
##   0.9492078  0.7607568  0.8928380  0.7348107
##   0.9686104  0.8342162  0.9284237  0.8242955
##   0.9823084  0.8943063  0.9560158  0.8928779
## 
## Tuning parameter 'gamma' was held constant at a value of 0
## 
## Tuning parameter 'min_child_weight' was held constant at a value of 1
## ROC was used to select the optimal model using  the largest value.
## The final values used for the model were nrounds = 150, max_depth = 3,
##  eta = 0.4, gamma = 0, colsample_bytree = 0.8, min_child_weight = 1
##  and subsample = 0.75.

Testing the results on the test data.

evalResults.XGB <- predict(XGBoost.Fit, testing, type="prob")[,1]
predict.xgb = ifelse(evalResults.XGB <0.5, "defaulted", "not.defaulted")
cm.XGB<- confusionMatrix(predict.xgb, testing$Default_On_Payment, positive="defaulted")

## Warning in confusionMatrix.default(predict.xgb, testing
## $Default_On_Payment, : Levels are not in the same order for reference and
## data. Refactoring data to match.

evalResults <- data.frame(Default_On_Payment=testing$Default_On_Payment)

library(pROC)
xgb.pROC <- pROC::roc(evalResults$Default_On_Payment, evalResults.XGB)
auc_curve=auc(xgb.pROC)
auc_curve

## Area under the curve: 0.7258

Taking into account xgBoost model as well, and compared to all the other, Random Forest again performs better.

table_perf[4,] = c("XGBOOST I",
                   round(auc_curve,3),
                   as.numeric(round(cm.XGB$overall["Accuracy"],3)),
                   as.numeric(round(cm.XGB$byClass["Sensitivity"],3)),
                   as.numeric(round(cm.XGB$byClass["Specificity"],3)),
                   as.numeric(round(cm.XGB$overall["Kappa"],3)))
table_perf

##                 model   auc accuracy sensitivity specificity kappa
## 1       Random Forest 0.813    0.625        0.94        0.49 0.324
## 2 Random Forest Tuned 0.836    0.655       0.945       0.531 0.367
## 3       C5.0 - Cubist 0.824    0.543       0.976       0.359 0.235
## 4           XGBOOST I 0.726    0.582       0.827       0.477 0.234

The problem I seek to solve is to determine the individuals who do default yet I do not want to get someone who may default into the category of no default.Then I look for the best thresholds (applying the function “best”) and see whether specificity has improved. Yet, this is at the cost of Sensitivity.

In this example, I choose to plot the default thresholds which is .5 and the best thresholds for Random Forest as this has brought the best model amongst the ones I have developed. We can see that the .3 threshold improved my Sensitivity. Interesting is to see that Random Forest, with the .5 threshold has indentified well the no default, yet not really well those who defaulted. Yet, the best threshold has balanced the gap between sensitivity and specificity.

plot(rf.pROC,legacy.axes = TRUE, print.thres= .5)
plot(rf.pROC,legacy.axes = TRUE, print.thres="best",col = "orange", add=TRUE)

In Conclusion, among the four models developed, after Preprocessing has been achieved, the best model was given by Random Forest. Yet, a decision making with the data provided cannot be taken. Performances weren’t great. Accuracy of the model hardly passed 65 %. Therefore, adding more data is needed. Using the best threshold, we also looked for the individuals who default as well as those who do not default. We apply this function since we do not want those who default wrongly categorised (as no default). With varImp function we have found those variables which seem to be the most important in predictions of the loan default.