Classification Metrics

nr <- sum(penguin_df$sex == "female")

nullErrorRate = 1 - (nr / length(penguin_df$sex))

ggplot(data = penguin_df, aes(x = sex )) + 
  geom_bar(fill = "steelblue") 

Knowing the null error rate gives us a baseline understanding of whether our approach is accurate. If guessing the hypothesis every time is correct more than 95% of the time, then our model may appear to be accurate, but would never actually predict anything.

pThresh2 <- threshold_change(penguin_df, .2)
pThresh5 <- threshold_change(penguin_df, .5)
pThresh8 <- threshold_change(penguin_df, .8)

penguins_CM(pThresh2)

Confusion Matrix and Statistics

          Reference
Prediction female male
    female     37    6
    male        2   48
                                          
               Accuracy : 0.914           
                 95% CI : (0.8375, 0.9621)
    No Information Rate : 0.5806          
    P-Value [Acc > NIR] : 9.542e-13       
                                          
                  Kappa : 0.8258          
                                          
 Mcnemar's Test P-Value : 0.2888          
                                          
            Sensitivity : 0.9487          
            Specificity : 0.8889          
         Pos Pred Value : 0.8605          
         Neg Pred Value : 0.9600          
              Precision : 0.8605          
                 Recall : 0.9487          
                     F1 : 0.9024          
             Prevalence : 0.4194          
         Detection Rate : 0.3978          
   Detection Prevalence : 0.4624          
      Balanced Accuracy : 0.9188          
                                          
       'Positive' Class : female          
                                          

penguins_CM(pThresh5)

Confusion Matrix and Statistics

          Reference
Prediction female male
    female     36    3
    male        3   51
                                         
               Accuracy : 0.9355         
                 95% CI : (0.8648, 0.976)
    No Information Rate : 0.5806         
    P-Value [Acc > NIR] : 1.319e-14      
                                         
                  Kappa : 0.8675         
                                         
 Mcnemar's Test P-Value : 1              
                                         
            Sensitivity : 0.9231         
            Specificity : 0.9444         
         Pos Pred Value : 0.9231         
         Neg Pred Value : 0.9444         
              Precision : 0.9231         
                 Recall : 0.9231         
                     F1 : 0.9231         
             Prevalence : 0.4194         
         Detection Rate : 0.3871         
   Detection Prevalence : 0.4194         
      Balanced Accuracy : 0.9338         
                                         
       'Positive' Class : female         
                                         

penguins_CM(pThresh8)

Confusion Matrix and Statistics

          Reference
Prediction female male
    female     36    3
    male        3   51
                                         
               Accuracy : 0.9355         
                 95% CI : (0.8648, 0.976)
    No Information Rate : 0.5806         
    P-Value [Acc > NIR] : 1.319e-14      
                                         
                  Kappa : 0.8675         
                                         
 Mcnemar's Test P-Value : 1              
                                         
            Sensitivity : 0.9231         
            Specificity : 0.9444         
         Pos Pred Value : 0.9231         
         Neg Pred Value : 0.9444         
              Precision : 0.9231         
                 Recall : 0.9231         
                     F1 : 0.9231         
             Prevalence : 0.4194         
         Detection Rate : 0.3871         
   Detection Prevalence : 0.4194         
      Balanced Accuracy : 0.9338         
                                         
       'Positive' Class : female         
                                         

accuracy is in the table, Pos Pred value is the precision, recall is sensitivity, and F1 is explicit in the output.