Penguins_Classification_Metrics_CCB

Penguin Predictions Approach

In order to practice classification metrics, I will first look over the data and then run a series of codes to test it’s accuracy.

As I work with the data to calculate the null rate and experiment with different probability thresholds.

Null Rate Error

From the penguins predictions, we can see that there are 54 males and 39 females. Classifying the majority of the penguins male would mean we would have a null rate error of 41.9%. Males are the majority in this dataset.

library(tidyverse) 

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

penguin_predictions <- read_csv("penguin_predictions.csv")  

Rows: 93 Columns: 3
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): .pred_class, sex
dbl (1): .pred_female

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

glimpse(penguin_predictions)

Rows: 93
Columns: 3
$ .pred_female <dbl> 0.99217462, 0.95423945, 0.98473504, 0.18702056, 0.9947012…
$ .pred_class  <chr> "female", "female", "female", "male", "female", "female",…
$ sex          <chr> "female", "female", "female", "female", "female", "female…

table(penguin_predictions$sex)

female   male 
    39     54 

gender_counts <-table(penguin_predictions$sex)  
null_error_rate <- 1 - max(gender_counts)/sum(gender_counts)  
null_error_rate

[1] 0.4193548

Confusion Matrix

We can measure the thresholds. The tables below count the true positives, false positives, true negatives, and false negatives produced using the 0.2, 0.5, and 0.8 threshold.

penguin_predictions <- penguin_predictions %>%   
  mutate(     
    pred_2 = if_else(.pred_female >= 0.2, "female", "male"),     
    pred_5 = if_else(.pred_female >= 0.5, "female", "male"),     
    pred_8 = if_else(.pred_female >= 0.8, "female", "male")  
    ) 
penguin_predictions 

# A tibble: 93 × 6
   .pred_female .pred_class sex    pred_2 pred_5 pred_8
          <dbl> <chr>       <chr>  <chr>  <chr>  <chr> 
 1        0.992 female      female female female female
 2        0.954 female      female female female female
 3        0.985 female      female female female female
 4        0.187 male        female male   male   male  
 5        0.995 female      female female female female
 6        1.000 female      female female female female
 7        0.959 female      female female female female
 8        1.000 female      female female female female
 9        1.000 female      female female female female
10        0.339 male        female female male   male  
# ℹ 83 more rows

table(   
  Predicted = penguin_predictions$pred_2,   
  Actual = penguin_predictions$sex ) 

         Actual
Predicted female male
   female     37    6
   male        2   48

table(   
  Predicted = penguin_predictions$pred_5,   
  Actual = penguin_predictions$sex )

         Actual
Predicted female male
   female     36    3
   male        3   51

table(   Predicted = penguin_predictions$pred_8,   
         Actual = penguin_predictions$sex )

         Actual
Predicted female male
   female     36    2
   male        3   52

Explanation

For each threshold, the predicted sex will be female. If 20% chance the penguin is female, then call it female otherwise it is male. In the first code chunk, we see 32 females as TP, 6 actual female categorized as male meaning FP, 2 actual male categorized as female meaning FN, and 48 actual male categorized as male as TN.

Performance Metrics

Accuracy = How often was the prediction correct overall? Precision = Of the penguins predicted to be female, how many actually were female?
Recall = Of the penguins who were actually female, how many did the model identify?
F1 score = A combined measure of precision and recall.