Assignment 2. Part b. Evaluating Classification Model Performance Using Penguin Data

Evaluating Classification Model Performance Using Penguin Data

Introduction/Approach

For this assignment part 2b, I will use the penguin_predictions.csv data to check how well the model works.

First, I will look at the sex column and count each class. Then, I will make a bar chart to show the class distribution, and will calculate the null error rate to get a baseline for the model.

Furthermore, I will use .pred_female to make predictions. I will also test three thresholds: 0.2, 0.5, and 0.8.

For each threshold, I will create a confusion matrix and will find True Positives, False Positives, True Negatives, and False Negatives.

Moreover, I will calculate accuracy, precision, recall, and F1 score for each threshold and compare the results.

Finally, I will explain how changing the threshold changes the model results and provide examples of when a low or high threshold can be useful.

GitHub links:

https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv

https://github.com/acatlin/data/blob/master/Performance%20Metrics%20for%20Classification%20problems%20in%20Machine%20Learning.pdf

Code Base

Lets upload the data set first from provided source and upload the packages to work with data.

library(dplyr)

Warning: package 'dplyr' was built under R version 4.5.2

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

library(ggplot2)

Warning: package 'ggplot2' was built under R version 4.5.2

penguins <- read.csv("https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv")

head(penguins)

  .pred_female .pred_class    sex
1    0.9921746      female female
2    0.9542394      female female
3    0.9847350      female female
4    0.1870206        male female
5    0.9947012      female female
6    0.9999891      female female

1. Lets explore the data and calculate null error rate

# Exporing of data
names(penguins)

[1] ".pred_female" ".pred_class"  "sex"         

unique(penguins$.pred_class)

[1] "female" "male"  

# Lets count each class
table(penguins$sex)

female   male 
    39     54 

unique(penguins$sex)

[1] "female" "male"  

#Lets create Class distribution plot
ggplot(penguins, aes(x = factor(sex))) + 
  geom_bar(fill = 'lightblue') + 
  labs(title = "Class Distribution", x = "sex", y = "Count")

#Lets calculate the Null Error Rate
class_counts <- table(penguins$sex)
# The class distribution shows the number of female and male penguins in the data. I decided to check this to see if one class of penguins is larger than another. 
null_error_rate <- 1 - max(class_counts)/sum(class_counts)
print(null_error_rate)

[1] 0.4193548

# Therefore, the null error rate is 41.94 % approximately. Thus, if I always predicted the most common penguin class, I would be wrong 41.94% of the time. I use this as a baseline to compare with classification model. 
# Lets show null error rate as a percentage
null_error_rate * 100

[1] 41.93548

2. Function for Thresholds

calculate_metrics <- function(data, threshold) {
# Lets predict class
  predicted <- ifelse(data$.pred_female > threshold, 'female','male')
#Lets see actual class
actual <- data$sex

# Confusion matrix values 
TP <- sum(predicted =="female" & actual == "female", na.rm = TRUE)
FP <- sum(predicted == "female" & actual =="male", na.rm  = TRUE)
TN <- sum(predicted== "male" & actual == "male",na.rm = TRUE )
FN <- sum(predicted == "male" & actual =="female", na.rm = TRUE)

# In this analysis, female is the positive class. A true positive or TP means that the model predicted female and the actual category was female penguins. In its turn, a false positive or FP means that model predicted female but the actual class was male penguins. True negative or TN means that model predicted male and the actual class of penguins was male. Laslty, a false negative or FN means that model predicted male but the actual class was female. 

#Lets calculate metrics using provide formulas for accurary, precision and recall

accuracy <- (TP+TN) / (TP + FP + TN + FN)
precision <- TP / (TP + FP)
recall <- TP / (TP + FN)
f1 <- 2 * (precision * recall)/(precision + recall)

# Lets create confusion matrix

confusion_matrix <- table(
  Predicted = predicted,
  Actual=actual
)

print(paste('Threshold:', threshold))
print(confusion_matrix)

data.frame(
  Threshold = threshold,
  TP = TP,
  FP = FP,
  TN = TN,
  FN = FN,
  Accuracy = accuracy,
  Precision = precision,
  Recall = recall,
  F1 =f1
)
}

3.Lets test the Threshold 0.2, Threshold 0.5, and Threshold 0.8

result_02 <- calculate_metrics(penguins, 0.2)

[1] "Threshold: 0.2"
         Actual
Predicted female male
   female     37    6
   male        2   48

result_05 <- calculate_metrics(penguins, 0.5)

[1] "Threshold: 0.5"
         Actual
Predicted female male
   female     36    3
   male        3   51

result_08 <- calculate_metrics(penguins, 0.8)

[1] "Threshold: 0.8"
         Actual
Predicted female male
   female     36    2
   male        3   52

4. Lets compare results

   results <- bind_rows(
     result_02,
     result_05,
     result_08
   )
   
   results

     Threshold TP FP TN FN  Accuracy Precision    Recall        F1
   1       0.2 37  6 48  2 0.9139785 0.8604651 0.9487179 0.9024390
   2       0.5 36  3 51  3 0.9354839 0.9230769 0.9230769 0.9230769
   3       0.8 36  2 52  3 0.9462366 0.9473684 0.9230769 0.9350649

Conclusion

Overall, I tested thresholds 0.2, 0.5, and 0.8. The results changed when I was changing the threshold. At threshold 0.2, recall was the highest, model found more of positive predicted. At the same time, it also had larger number of false positives as well. In its turn, with 0.8 threshold, the model had fewer false positives and had the highest accuracy, precision, and F1 score. Therefore, may be concluded that changing the threshold affects the performance of the model.

Thus, lower threshold more positives predicted , and lower threshold can be useful when it is important not to miss positive cases. In its turn, 0.8 threshold ca be beneficial when is required to reduce false positives.

Baseline Comparison: The null error rate is 41.94%, meaning the baseline accuracy is 58.06%. This means that always predicting the most common penguin class would be right 58.06% of the time. The model performs much better than this baseline. For instance, with 0.8 threshold, the model will be accurate 94.62%. Therefore, the classification model does better than simply predicting the most common class.

References

Google Deep Mind. (2025). Gemini 3 Flash [Large language model]. [https://gemini.google.com.](https://gemini.google.com/) Accessed September 13th, 2026.

Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for data science (2nd ed.). O’Reilly Media. https://r4ds.hadley.nz/