KNN modeling with Confusion Matrix Evaluation

Summary

This notebook describe the following:

Upload the necessary packages and dataset.
Do an EDA (Explanatory Data Analysis) on the dataset
Spit the data between Training Dataset and Test Data using Simple Proportion and Random Sampling.
Memorize the datset and run the KNN model on unseen data.
Evaluate the overall model with a Confusion Matrix.
Evaluate the model for each class label with a Confusion Matrix:
- Setosa
- Versicolor
- Verginica
Summarize the model performance by defining the following measures:
- True Positive (TP)
- True Negative (TN)
- False Positive (FP)
- False Negative (FN)
- Accuracy
- Specificity
Optimizing K

Upload the necessary packages and dataset

Packages:
- Class (for the KNN algorithm)
- Caret (for the confusion matrix)
Dataset: the iris dataset

library(class)
library(caret)
library(ggplot2)
dataset = iris

Explanatory Data Analysis

Structure.data = str(dataset)

'data.frame':   150 obs. of  5 variables:
 $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
 $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
 $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
 $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
 $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

Prepare the data

#define the proportion that split training data and test data
Train_data_prop = .7

#split the Training dataset
Train_sample = sample(1:nrow(iris), Train_data_prop*nrow(iris),replace = TRUE)
Train_data  = iris[Train_sample,1:4]
Train_class =  iris[Train_sample,5]

#split the Test dataset
Test_data =  iris[-Train_sample,1:4]
Test_class =  iris[-Train_sample,5]

Memorize the dataset and run the KNN model on unseen data

#Create and train the KNN model
Kneighbors = 3
Model  = knn(
              train = Train_data, # Training Predictor variables or features
              test = Test_data,   # Test Predictor variables or features
              cl = Train_class,   # Training data labels 
              k = Kneighbors      # Defined k 
              )

Summarize the model performance

  table(data.frame("Actual" = Test_class, "Predicted" = Model))

            Predicted
Actual       setosa versicolor virginica
  setosa         29          0         0
  versicolor      0         24         1
  virginica       0          3        19

Evaluate the overall model with a Confusion Matrix

  options(scipen=999)
  ConfusionMatrix = confusionMatrix(
                                      data = Model,
                                      reference = Test_class #Factor of labels to be true results
                                    )
  ConfusionMatrix

Confusion Matrix and Statistics

            Reference
Prediction   setosa versicolor virginica
  setosa         29          0         0
  versicolor      0         24         3
  virginica       0          1        19

Overall Statistics
                                               
               Accuracy : 0.9474               
                 95% CI : (0.8707, 0.9855)     
    No Information Rate : 0.3816               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.9204               
 Mcnemar's Test P-Value : NA                   

Statistics by Class:

                     Class: setosa Class: versicolor Class: virginica
Sensitivity                 1.0000            0.9600           0.8636
Specificity                 1.0000            0.9412           0.9815
Pos Pred Value              1.0000            0.8889           0.9500
Neg Pred Value              1.0000            0.9796           0.9464
Prevalence                  0.3816            0.3289           0.2895
Detection Rate              0.3816            0.3158           0.2500
Detection Prevalence        0.3816            0.3553           0.2632
Balanced Accuracy           1.0000            0.9506           0.9226

Basic Metrics

  TruePositive = sum(diag(as.matrix(ConfusionMatrix$table)))
  TrueNegative = sum(as.matrix(ConfusionMatrix$table)) - TruePositive
  TotalObservation = sum(as.matrix(ConfusionMatrix$table))
  Accuracy = TruePositive/TotalObservation

Accuracy

Definition: The ability of the classifier to select all cases that need to be selected and reject all cases that need to be rejected.

Accuracy = 0.9473684

Optimizing K

Kmin = 1
Kmax = length(Test_class)
DataFrame = as.data.frame(cbind(K = c(Kmin:Kmax),Accuracy = rep(0,Kmax)))
i = Kmin
while (i<= Kmax)
{
  knn.temp = knn(
                 train = Train_data,
                 test = Test_data,
                 cl = Train_class,
                 k = i
                )
  ConfusionMatrix.temp = confusionMatrix(data = knn.temp,reference = Test_class)
  
  DataFrame[i,2] = ConfusionMatrix.temp$overall[[1]]
  i = i+1
  next
}
 Accuracy.plot = ggplot(data = DataFrame, aes(x=K,y=Accuracy)) + geom_line()
 Accuracy.plot

LS0tDQp0aXRsZTogIktOTiBtb2RlbGluZyB3aXRoIENvbmZ1c2lvbiBNYXRyaXggRXZhbHVhdGlvbiINCm91dHB1dDogaHRtbF9ub3RlYm9vaw0KLS0tDQoNCiMjIFN1bW1hcnkNCg0KVGhpcyBub3RlYm9vayBkZXNjcmliZSB0aGUgZm9sbG93aW5nOg0KDQoqIFVwbG9hZCB0aGUgbmVjZXNzYXJ5IHBhY2thZ2VzIGFuZCBkYXRhc2V0Lg0KKiBEbyBhbiBFREEgKEV4cGxhbmF0b3J5IERhdGEgQW5hbHlzaXMpIG9uIHRoZSBkYXRhc2V0DQoqIFNwaXQgdGhlIGRhdGEgYmV0d2VlbiBUcmFpbmluZyBEYXRhc2V0IGFuZCBUZXN0IERhdGEgdXNpbmcgU2ltcGxlIFByb3BvcnRpb24gYW5kIFJhbmRvbSBTYW1wbGluZy4NCiogTWVtb3JpemUgdGhlIGRhdHNldCBhbmQgcnVuIHRoZSBLTk4gbW9kZWwgb24gdW5zZWVuIGRhdGEuDQoqIEV2YWx1YXRlIHRoZSBvdmVyYWxsIG1vZGVsIHdpdGggYSBDb25mdXNpb24gTWF0cml4Lg0KKiBFdmFsdWF0ZSB0aGUgbW9kZWwgZm9yIGVhY2ggY2xhc3MgbGFiZWwgd2l0aCBhIENvbmZ1c2lvbiBNYXRyaXg6DQoNCiAgLSBTZXRvc2ENCiAgLSBWZXJzaWNvbG9yDQogIC0gVmVyZ2luaWNhICANCg0KKiBTdW1tYXJpemUgdGhlIG1vZGVsIHBlcmZvcm1hbmNlIGJ5IGRlZmluaW5nIHRoZSBmb2xsb3dpbmcgbWVhc3VyZXM6DQoNCiAgLSBUcnVlIFBvc2l0aXZlIChUUCkNCiAgLSBUcnVlIE5lZ2F0aXZlIChUTikNCiAgLSBGYWxzZSBQb3NpdGl2ZSAoRlApDQogIC0gRmFsc2UgTmVnYXRpdmUgKEZOKQ0KDQogIC0gQWNjdXJhY3kNCiAgLSBTcGVjaWZpY2l0eQ0KICANCiogT3B0aW1pemluZyBLDQoNCiMjIFVwbG9hZCB0aGUgbmVjZXNzYXJ5IHBhY2thZ2VzIGFuZCBkYXRhc2V0DQoNCiogUGFja2FnZXM6IA0KDQogIC0gQ2xhc3MgKGZvciB0aGUgS05OIGFsZ29yaXRobSkNCiAgLSBDYXJldCAoZm9yIHRoZSBjb25mdXNpb24gbWF0cml4KQ0KICANCiogRGF0YXNldDogdGhlIGlyaXMgZGF0YXNldA0KDQpgYGB7cn0NCmxpYnJhcnkoY2xhc3MpDQpsaWJyYXJ5KGNhcmV0KQ0KbGlicmFyeShnZ3Bsb3QyKQ0KZGF0YXNldCA9IGlyaXMNCmBgYA0KDQojIyBFeHBsYW5hdG9yeSBEYXRhIEFuYWx5c2lzDQoNCmBgYHtyfQ0KU3RydWN0dXJlLmRhdGEgPSBzdHIoZGF0YXNldCkNCmBgYA0KDQojIyBQcmVwYXJlIHRoZSBkYXRhDQoNCmBgYHtyfQ0KI2RlZmluZSB0aGUgcHJvcG9ydGlvbiB0aGF0IHNwbGl0IHRyYWluaW5nIGRhdGEgYW5kIHRlc3QgZGF0YQ0KVHJhaW5fZGF0YV9wcm9wID0gLjcNCg0KI3NwbGl0IHRoZSBUcmFpbmluZyBkYXRhc2V0DQpUcmFpbl9zYW1wbGUgPSBzYW1wbGUoMTpucm93KGlyaXMpLCBUcmFpbl9kYXRhX3Byb3AqbnJvdyhpcmlzKSxyZXBsYWNlID0gVFJVRSkNClRyYWluX2RhdGEgID0gaXJpc1tUcmFpbl9zYW1wbGUsMTo0XQ0KVHJhaW5fY2xhc3MgPSAgaXJpc1tUcmFpbl9zYW1wbGUsNV0NCg0KI3NwbGl0IHRoZSBUZXN0IGRhdGFzZXQNClRlc3RfZGF0YSA9ICBpcmlzWy1UcmFpbl9zYW1wbGUsMTo0XQ0KVGVzdF9jbGFzcyA9ICBpcmlzWy1UcmFpbl9zYW1wbGUsNV0NCmBgYA0KDQojIyBNZW1vcml6ZSB0aGUgZGF0YXNldCBhbmQgcnVuIHRoZSBLTk4gbW9kZWwgb24gdW5zZWVuIGRhdGENCg0KYGBge3J9DQojQ3JlYXRlIGFuZCB0cmFpbiB0aGUgS05OIG1vZGVsDQpLbmVpZ2hib3JzID0gMw0KTW9kZWwgID0ga25uKA0KICAgICAgICAgICAgICB0cmFpbiA9IFRyYWluX2RhdGEsICMgVHJhaW5pbmcgUHJlZGljdG9yIHZhcmlhYmxlcyBvciBmZWF0dXJlcw0KICAgICAgICAgICAgICB0ZXN0ID0gVGVzdF9kYXRhLCAgICMgVGVzdCBQcmVkaWN0b3IgdmFyaWFibGVzIG9yIGZlYXR1cmVzDQogICAgICAgICAgICAgIGNsID0gVHJhaW5fY2xhc3MsICAgIyBUcmFpbmluZyBkYXRhIGxhYmVscyANCiAgICAgICAgICAgICAgayA9IEtuZWlnaGJvcnMgICAgICAjIERlZmluZWQgayANCiAgICAgICAgICAgICAgKQ0KYGBgDQoNCiMjIFN1bW1hcml6ZSB0aGUgbW9kZWwgcGVyZm9ybWFuY2UNCg0KYGBge3J9DQogIHRhYmxlKGRhdGEuZnJhbWUoIkFjdHVhbCIgPSBUZXN0X2NsYXNzLCAiUHJlZGljdGVkIiA9IE1vZGVsKSkNCmBgYA0KDQojIyBFdmFsdWF0ZSB0aGUgb3ZlcmFsbCBtb2RlbCB3aXRoIGEgQ29uZnVzaW9uIE1hdHJpeA0KDQpgYGB7cn0NCiAgb3B0aW9ucyhzY2lwZW49OTk5KQ0KICBDb25mdXNpb25NYXRyaXggPSBjb25mdXNpb25NYXRyaXgoDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBNb2RlbCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVmZXJlbmNlID0gVGVzdF9jbGFzcyAjRmFjdG9yIG9mIGxhYmVscyB0byBiZSB0cnVlIHJlc3VsdHMNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICkNCiAgQ29uZnVzaW9uTWF0cml4DQpgYGANCg0KIyMjIEJhc2ljIE1ldHJpY3MNCg0KYGBge3J9DQogIFRydWVQb3NpdGl2ZSA9IHN1bShkaWFnKGFzLm1hdHJpeChDb25mdXNpb25NYXRyaXgkdGFibGUpKSkNCiAgVHJ1ZU5lZ2F0aXZlID0gc3VtKGFzLm1hdHJpeChDb25mdXNpb25NYXRyaXgkdGFibGUpKSAtIFRydWVQb3NpdGl2ZQ0KICBUb3RhbE9ic2VydmF0aW9uID0gc3VtKGFzLm1hdHJpeChDb25mdXNpb25NYXRyaXgkdGFibGUpKQ0KICBBY2N1cmFjeSA9IFRydWVQb3NpdGl2ZS9Ub3RhbE9ic2VydmF0aW9uDQpgYGANCg0KIyMjIyMgKipBY2N1cmFjeSoqDQoNCioqRGVmaW5pdGlvbioqOiBUaGUgYWJpbGl0eSBvZiB0aGUgY2xhc3NpZmllciB0byBzZWxlY3QgYWxsIGNhc2VzIHRoYXQgbmVlZCB0byBiZSBzZWxlY3RlZCBhbmQgcmVqZWN0IGFsbCBjYXNlcyB0aGF0IG5lZWQgdG8gYmUgcmVqZWN0ZWQuDQoNCipBY2N1cmFjeSogPSBgciBBY2N1cmFjeWANCg0KIyMjIE9wdGltaXppbmcgSw0KDQpgYGB7cn0NCkttaW4gPSAxDQpLbWF4ID0gbGVuZ3RoKFRlc3RfY2xhc3MpDQpEYXRhRnJhbWUgPSBhcy5kYXRhLmZyYW1lKGNiaW5kKEsgPSBjKEttaW46S21heCksQWNjdXJhY3kgPSByZXAoMCxLbWF4KSkpDQppID0gS21pbg0Kd2hpbGUgKGk8PSBLbWF4KQ0Kew0KICBrbm4udGVtcCA9IGtubigNCiAgICAgICAgICAgICAgICAgdHJhaW4gPSBUcmFpbl9kYXRhLA0KICAgICAgICAgICAgICAgICB0ZXN0ID0gVGVzdF9kYXRhLA0KICAgICAgICAgICAgICAgICBjbCA9IFRyYWluX2NsYXNzLA0KICAgICAgICAgICAgICAgICBrID0gaQ0KICAgICAgICAgICAgICAgICkNCiAgQ29uZnVzaW9uTWF0cml4LnRlbXAgPSBjb25mdXNpb25NYXRyaXgoZGF0YSA9IGtubi50ZW1wLHJlZmVyZW5jZSA9IFRlc3RfY2xhc3MpDQogIA0KICBEYXRhRnJhbWVbaSwyXSA9IENvbmZ1c2lvbk1hdHJpeC50ZW1wJG92ZXJhbGxbWzFdXQ0KICBpID0gaSsxDQogIG5leHQNCn0NCiBBY2N1cmFjeS5wbG90ID0gZ2dwbG90KGRhdGEgPSBEYXRhRnJhbWUsIGFlcyh4PUsseT1BY2N1cmFjeSkpICsgZ2VvbV9saW5lKCkNCiBBY2N1cmFjeS5wbG90DQpgYGANCg0KDQo=