Accuracy

• Percentage of correctly classified instances out of all instances.
• It is more useful on a binary classification than multi-class classification problems

Kappa

• Classification accuracy that it is normalized at the baseline of random chance on your dataset.
• It is a more useful measure to use on problems that have an imbalance in the classes (e.g. 70-30 split for classes 0 and 1 and you can achieve 70% accuracy by predicting all instances are for class 0).

ROC

• Area Under the ROC Curve or AUC
• Represents the Model’s ability to discriminate between +ve and -ve classes
• An area of 1.0 represents a model that made all predictions perfectly
• An area of 0.5 represents a model that is as good as random
• ROC can be brokern down into Sensivity and Specificity
• Formula for Sensitivity or True Positive Rate (TPR) is TPR = TP / (TP + FN)
• Formula for Specificity or True Negative Rate (SPC) is SPC = TN / (FP + TN)

LogLoss

• Is used to evaluate binary classification but is more common for multi-class classification algorithms
• It evaluates the probably estimated by the algorithms

Default performance metrics by response types.

**************************************************************************
Response Default Metrics
**************************************************************************

Factor Brier Score, Accuracy, Cohen’s Kappa

Binary Factor Brier Score, Accuracy, Cohen’s Kappa, Area Under ROC

Curve, Sensitivity, Specificity

Numeric Vector or Matrix Root Mean Squared Error, R2, Mean Absolute Error

Survival Means Concordance Index

Survival Probabilities Area Under ROC Curve, Brier Score, Accuracy

Survival Events Accuracy

**************************************************************************

Box Plot

Bar Plot

Featureplot ellipse

Featureplot density

summarize accuracy of models

Summary results

Call:
summary.resamples(object = results)

Models: lda, cart, knn, svm, rf 
Number of resamples: 10 

Accuracy 
          Min.   1st Qu.    Median      Mean 3rd Qu. Max. NA's
lda  0.9166667 0.9375000 1.0000000 0.9750000       1    1    0
cart 0.7500000 0.9166667 0.9583333 0.9333333       1    1    0
knn  0.9166667 0.9166667 1.0000000 0.9666667       1    1    0
svm  0.8333333 0.9166667 1.0000000 0.9500000       1    1    0
rf   0.8333333 0.9166667 0.9583333 0.9500000       1    1    0

Kappa 
      Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
lda  0.875 0.90625 1.0000 0.9625       1    1    0
cart 0.625 0.87500 0.9375 0.9000       1    1    0
knn  0.875 0.87500 1.0000 0.9500       1    1    0
svm  0.750 0.87500 1.0000 0.9250       1    1    0
rf   0.750 0.87500 0.9375 0.9250       1    1    0

Summarize the best model

Linear Discriminant Analysis 

120 samples
  4 predictor
  3 classes: 'Iris-setosa', 'Iris-versicolor', 'Iris-virginica' 

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 108, 108, 108, 108, 108, 108, ... 
Resampling results:

  Accuracy  Kappa 
  0.975     0.9625

Confusion Matrix and Statistics

                 Reference
Prediction        Iris-setosa Iris-versicolor Iris-virginica
  Iris-setosa              10               0              0
  Iris-versicolor           0              10              0
  Iris-virginica            0               0             10

Overall Statistics
                                     
               Accuracy : 1          
                 95% CI : (0.8843, 1)
    No Information Rate : 0.3333     
    P-Value [Acc > NIR] : 4.857e-15  
                                     
                  Kappa : 1          
 Mcnemar's Test P-Value : NA         

Statistics by Class:

                     Class: Iris-setosa Class: Iris-versicolor
Sensitivity                      1.0000                 1.0000
Specificity                      1.0000                 1.0000
Pos Pred Value                   1.0000                 1.0000
Neg Pred Value                   1.0000                 1.0000
Prevalence                       0.3333                 0.3333
Detection Rate                   0.3333                 0.3333
Detection Prevalence             0.3333                 0.3333
Balanced Accuracy                1.0000                 1.0000
                     Class: Iris-virginica
Sensitivity                         1.0000
Specificity                         1.0000
Pos Pred Value                      1.0000
Neg Pred Value                      1.0000
Prevalence                          0.3333
Detection Rate                      0.3333
Detection Prevalence                0.3333
Balanced Accuracy                   1.0000

compare accuracy of models