R Markdown

Performance estimation helps evaluate based on metrics of predictive performance from an unseen data distribution. In comparing to the SVM (Support Vector Machine) and RF (Random Forest) Models which estimates would provide a better approach. For the redWine data we will treat it as an Classification Model.

The goal of performance estimation is to obtain a reliable estimate of the predicted prediction error of a model on the unknown data distribution. To derive a model that has reliable estimate we must use unseen data that can be used as test set.

Another goal of performace estimation is to run and repeat the testing several times to collect a variety of scores to average the error rate and also the standard error as an estimate of the true population.

Loading Data

load("redWine.Rdata")

# Set the Library functions
library(performanceEstimation)
library(randomForest)
## randomForest 4.6-12
## Type rfNews() to see new features/changes/bug fixes.
library(e1071)
library(lattice)
library(DMwR)
## Loading required package: grid

Hold Out Method

70% Training and 30% Test Set

trPrec<-0.7 # 70% training set
sp<-sample(1:nrow(redWine),as.integer(trPrec*nrow(redWine)))

redWine$quality<-as.factor(redWine$quality) # Convert to Classification Model

train<-redWine[sp,] # Training Set
test<-redWine[-sp,] # Test Set


modelSVM <- svm(quality ~ ., train) # default radial kernel
modelRF <- randomForest(quality ~ ., train, ntree=750,importance=T)

pred_svm<-predict(modelSVM, test, type='class')
pred_RF<-predict(modelRF, test,type='class')

classificationMetrics(test$quality,pred_svm)
##       acc       err    microF    macroF  macroRec macroPrec 
## 0.5958333 0.4041667 0.5958333       NaN 0.2701593       NaN
classificationMetrics(test$quality,pred_RF)
##       acc       err    microF    macroF  macroRec macroPrec 
## 0.6604167 0.3395833 0.6604167       NaN 0.3059113       NaN

The Error rate for Radial Kernel for Support Vector Machine has a rate of 39.2% which is a higher rate compared to 31.6% for Random Forest

Hold-Out set with Iterations

res <- performanceEstimation(
  PredTask(quality ~ .,redWine),
    workflowVariants(learner=c("svm","randomForest")),
  EstimationTask(metrics="err",method=Holdout(nReps=10,hldSz=0.3))  )
## 
## 
## ##### PERFORMANCE ESTIMATION USING  HOLD OUT  #####
## 
## ** PREDICTIVE TASK :: redWine.quality
## 
## ++ MODEL/WORKFLOW :: svm 
## Task for estimating  err  using
##  10 x 70 % / 30 % Holdout
##   Run with seed =  1234 
## Iteration :  1  2  3  4  5  6  7  8  9  10
## 
## 
## ++ MODEL/WORKFLOW :: randomForest 
## Task for estimating  err  using
##  10 x 70 % / 30 % Holdout
##   Run with seed =  1234 
## Iteration :  1  2  3  4  5  6  7  8  9  10

Selection of Model

summary(res)
## 
## == Summary of a  Hold Out Performance Estimation Experiment ==
## 
## Task for estimating  err  using
##  10 x 70 % / 30 % Holdout
##   Run with seed =  1234 
## 
## * Predictive Tasks ::  redWine.quality
## * Workflows  ::  svm, randomForest 
## 
## -> Task:  redWine.quality
##   *Workflow: svm 
##                err
## avg     0.36534447
## std     0.01530967
## med     0.36638831
## iqr     0.01774530
## min     0.34237996
## max     0.38622129
## invalid 0.00000000
## 
##   *Workflow: randomForest 
##                 err
## avg     0.318162839
## std     0.011186378
## med     0.315240084
## iqr     0.007306889
## min     0.304801670
## max     0.340292276
## invalid 0.000000000
rankWorkflows(res,2)
## $redWine.quality
## $redWine.quality$err
##       Workflow  Estimate
## 1 randomForest 0.3181628
## 2          svm 0.3653445
topPerformers(res) # Choose Top Performing model
## $redWine.quality
##         Workflow Estimate
## err randomForest    0.318

Plot the Cross Validation (Error Rate) Mean Squared Error Comparison

plot(res)

The average of 10 iterations for the Hold-Out Set of 30% been Test set shows the error rate to be lower at 36.5% for SVM and same error rate for 31.8% for Random Forest.

K-Fold Cross Validation

res <- performanceEstimation(
  PredTask(quality ~ .,redWine),
    workflowVariants(learner=c("svm","randomForest")),
  EstimationTask(metrics="err",method=CV(nReps=1,nFolds=10))  )
## 
## 
## ##### PERFORMANCE ESTIMATION USING  CROSS VALIDATION  #####
## 
## ** PREDICTIVE TASK :: redWine.quality
## 
## ++ MODEL/WORKFLOW :: svm 
## Task for estimating  err  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********
## 
## 
## ++ MODEL/WORKFLOW :: randomForest 
## Task for estimating  err  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********

Using 10-Fold Cross Validation with a set of n-observations it will randomly split into 10 non-overlapping groups as an validation set. It will iterate through each of the 10 Folds and averaging the test error rate out.

Selection of Model

summary(res)
## 
## == Summary of a  Cross Validation Performance Estimation Experiment ==
## 
## Task for estimating  err  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## 
## * Predictive Tasks ::  redWine.quality
## * Workflows  ::  svm, randomForest 
## 
## -> Task:  redWine.quality
##   *Workflow: svm 
##                err
## avg     0.37232704
## std     0.03325347
## med     0.36163522
## iqr     0.03144654
## min     0.33333333
## max     0.44654088
## invalid 0.00000000
## 
##   *Workflow: randomForest 
##                err
## avg     0.27987421
## std     0.03857101
## med     0.26729560
## iqr     0.01886792
## min     0.24528302
## max     0.36477987
## invalid 0.00000000
rankWorkflows(res,2)
## $redWine.quality
## $redWine.quality$err
##       Workflow  Estimate
## 1 randomForest 0.2798742
## 2          svm 0.3723270
topPerformers(res) # Choose Top Performing model
## $redWine.quality
##         Workflow Estimate
## err randomForest     0.28

Plot the Cross Validation (Error Rate) Model Comparison

plot(res)

BootStrapping

res <- performanceEstimation(
  PredTask(quality ~ .,redWine),
    workflowVariants(learner=c("svm","randomForest")),
  EstimationTask(metrics="err",method=Bootstrap(nReps=100)) )
## 
## 
## ##### PERFORMANCE ESTIMATION USING  BOOTSTRAP  #####
## 
## ** PREDICTIVE TASK :: redWine.quality
## 
## ++ MODEL/WORKFLOW :: svm 
## Task for estimating  err  using
## 100  repetitions of  e0  Bootstrap experiment
##   Run with seed =  1234 
## Iteration :  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99  100
## 
## 
## ++ MODEL/WORKFLOW :: randomForest 
## Task for estimating  err  using
## 100  repetitions of  e0  Bootstrap experiment
##   Run with seed =  1234 
## Iteration :  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99  100

The bootstrapping method is training a model on a random sample set of size n with replacement. So several observations would be used as training dataset for the number of iterations.