Approach

For this assignment, I plan to use the penguin predictions dataset to learn how the probability threshold of a binary classification model affects its predictions and performance. I will first examine the actual class distribution and calculate the null error rate to understand the baseline performance of the model.

I will then compare probability thresholds of 0.2, 0.5, and 0.8. For each threshold, I plan to create a confusion matrix containing true positives, false positives, true negatives, and false negatives. I will use these results to calculate accuracy, precision, recall, and F1 score.

Finally, I will compare the results across the three thresholds to understand the tradeoff between predicting more or fewer positive cases. Some challenges I expect are understanding the confusion matrix values, calculating the performance metrics correctly, and explaining why a lower or higher threshold may be better depending on the real-world situation.

penguins <- read.csv(
  "https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv"
)

head(penguins)
##   .pred_female .pred_class    sex
## 1    0.9921746      female female
## 2    0.9542394      female female
## 3    0.9847350      female female
## 4    0.1870206        male female
## 5    0.9947012      female female
## 6    0.9999891      female female
names(penguins)
## [1] ".pred_female" ".pred_class"  "sex"
str(penguins)
## 'data.frame':    93 obs. of  3 variables:
##  $ .pred_female: num  0.992 0.954 0.985 0.187 0.995 ...
##  $ .pred_class : chr  "female" "female" "female" "male" ...
##  $ sex         : chr  "female" "female" "female" "female" ...

Null Error Rate

The null error rate represents the error rate I would get if I ignored the model and predicted every penguin as the most common class. First, I will examine the distribution of the actual sex values.

table(penguins$sex)
## 
## female   male 
##     39     54
null_error_rate <- 39 / 93

null_error_rate
## [1] 0.4193548

Null Error Rate Using Class Counts

class_counts <- table(penguins$sex)

null_error_rate <- min(class_counts) / sum(class_counts)

null_error_rate
## [1] 0.4193548

I found that the dataset contains 39 female and 54 male penguins. Since male is the majority class, I used male as the baseline prediction. If I predicted every penguin as male, I would incorrectly classify all 39 female penguins. I calculated the null error rate as approximately 0.4194, or 41.94%. This gives me a baseline to compare the classification model against.

library(ggplot2)

ggplot(penguins, aes(x = sex, fill = sex)) +
  geom_bar() +
  labs(
    title = "Distribution of Actual Penguin Sex",
    x = "Sex",
    y = "Count"
  ) +
  theme_minimal() +
  theme(legend.position = "none")

Confusion Matrices

Threshold 0.2

I will first use a probability threshold of 0.2. If the predicted probability of female is greater than 0.2, the penguin will be classified as female. Otherwise, it will be classified as male.

penguins$pred_02 <- ifelse(
  penguins$.pred_female > 0.2,
  "female",
  "male"
)

table(penguins$pred_02)
## 
## female   male 
##     43     50
confusion_02 <- table(
  Predicted = penguins$pred_02,
  Actual = penguins$sex
)

confusion_02
##          Actual
## Predicted female male
##    female     37    6
##    male        2   48
TP_02 <- sum(penguins$pred_02 == "female" & penguins$sex == "female")
FP_02 <- sum(penguins$pred_02 == "female" & penguins$sex == "male")
TN_02 <- sum(penguins$pred_02 == "male" & penguins$sex == "male")
FN_02 <- sum(penguins$pred_02 == "male" & penguins$sex == "female")

TP_02
## [1] 37
FP_02
## [1] 6
TN_02
## [1] 48
FN_02
## [1] 2
accuracy_02 <- (TP_02 + TN_02) / (TP_02 + FP_02 + TN_02 + FN_02)

precision_02 <- TP_02 / (TP_02 + FP_02)

recall_02 <- TP_02 / (TP_02 + FN_02)

f1_02 <- 2 * (precision_02 * recall_02) / (precision_02 + recall_02)

accuracy_02
## [1] 0.9139785
precision_02
## [1] 0.8604651
recall_02
## [1] 0.9487179
f1_02
## [1] 0.902439

At the 0.2 threshold, I calculated an accuracy of 91.40%, precision of 86.05%, recall of 94.87%, and an F1 score of 90.24%. I noticed that this threshold had the highest recall, meaning it identified most of the actual female penguins.

Threshold 0.5

Next, I will use a probability threshold of 0.5. If the predicted probability of female is greater than 0.5, the penguin will be classified as female. Otherwise, it will be classified as male.

penguins$pred_05 <- ifelse(
  penguins$.pred_female > 0.5,
  "female",
  "male"
)

confusion_05 <- table(
  Predicted = penguins$pred_05,
  Actual = penguins$sex
)

confusion_05
##          Actual
## Predicted female male
##    female     36    3
##    male        3   51
TP_05 <- sum(penguins$pred_05 == "female" & penguins$sex == "female")
FP_05 <- sum(penguins$pred_05 == "female" & penguins$sex == "male")
TN_05 <- sum(penguins$pred_05 == "male" & penguins$sex == "male")
FN_05 <- sum(penguins$pred_05 == "male" & penguins$sex == "female")

TP_05
## [1] 36
FP_05
## [1] 3
TN_05
## [1] 51
FN_05
## [1] 3
accuracy_05 <- (TP_05 + TN_05) / (TP_05 + FP_05 + TN_05 + FN_05)

precision_05 <- TP_05 / (TP_05 + FP_05)

recall_05 <- TP_05 / (TP_05 + FN_05)

f1_05 <- 2 * (precision_05 * recall_05) / (precision_05 + recall_05)

accuracy_05
## [1] 0.9354839
precision_05
## [1] 0.9230769
recall_05
## [1] 0.9230769
f1_05
## [1] 0.9230769

At the 0.5 threshold, I calculated an accuracy of 93.55%. Precision, recall, and the F1 score were all approximately 92.31%. Compared with the 0.2 threshold, I found that precision improved while recall decreased slightly.

Threshold 0.8

Finally, I will use a probability threshold of 0.8. If the predicted probability of female is greater than 0.8, the penguin will be classified as female. Otherwise, it will be classified as male.

penguins$pred_08 <- ifelse(
  penguins$.pred_female > 0.8,
  "female",
  "male"
)

confusion_08 <- table(
  Predicted = penguins$pred_08,
  Actual = penguins$sex
)

confusion_08
##          Actual
## Predicted female male
##    female     36    2
##    male        3   52
TP_08 <- sum(penguins$pred_08 == "female" & penguins$sex == "female")
FP_08 <- sum(penguins$pred_08 == "female" & penguins$sex == "male")
TN_08 <- sum(penguins$pred_08 == "male" & penguins$sex == "male")
FN_08 <- sum(penguins$pred_08 == "male" & penguins$sex == "female")

TP_08
## [1] 36
FP_08
## [1] 2
TN_08
## [1] 52
FN_08
## [1] 3
accuracy_08 <- (TP_08 + TN_08) / (TP_08 + FP_08 + TN_08 + FN_08)

precision_08 <- TP_08 / (TP_08 + FP_08)

recall_08 <- TP_08 / (TP_08 + FN_08)

f1_08 <- 2 * (precision_08 * recall_08) / (precision_08 + recall_08)

accuracy_08
## [1] 0.9462366
precision_08
## [1] 0.9473684
recall_08
## [1] 0.9230769
f1_08
## [1] 0.9350649

At the 0.8 threshold, I calculated an accuracy of 94.62%, precision of 94.74%, recall of 92.31%, and an F1 score of 93.51%. I found that this threshold had the highest accuracy, precision, and F1 score of the three thresholds.

Performance Comparison

performance_table <- data.frame(
  Threshold = c(0.2, 0.5, 0.8),
  Accuracy = c(accuracy_02, accuracy_05, accuracy_08),
  Precision = c(precision_02, precision_05, precision_08),
  Recall = c(recall_02, recall_05, recall_08),
  F1 = c(f1_02, f1_05, f1_08)
)

performance_table
##   Threshold  Accuracy Precision    Recall        F1
## 1       0.2 0.9139785 0.8604651 0.9487179 0.9024390
## 2       0.5 0.9354839 0.9230769 0.9230769 0.9230769
## 3       0.8 0.9462366 0.9473684 0.9230769 0.9350649
performance_table[, 2:5] <- round(
  performance_table[, 2:5],
  4
)

performance_table
##   Threshold Accuracy Precision Recall     F1
## 1       0.2   0.9140    0.8605 0.9487 0.9024
## 2       0.5   0.9355    0.9231 0.9231 0.9231
## 3       0.8   0.9462    0.9474 0.9231 0.9351

Choosing a Classification Threshold

Based on my results, I would prefer the 0.2 threshold in a situation where identifying as many positive cases as possible is important. The 0.2 threshold had the highest recall at 94.87%. For example, in a medical screening test, I would prefer a lower threshold if missing someone who may have a disease is more serious than incorrectly identifying someone as at risk. This would allow more possible positive cases to be identified for further testing.

I would prefer the 0.8 threshold when false positive predictions are more costly. At this threshold, I found that precision increased to 94.74%, with only two false positives. For example, if a positive prediction leads to an expensive or time-consuming follow-up procedure, I would choose a higher threshold because the model would require stronger evidence before making a positive prediction.

From this comparison, I learned that the best threshold depends on the purpose of the classification model. I can use a lower threshold when reducing false negatives is more important, while I can use a higher threshold when reducing false positives is more important.

To provide a more complete comparison, I also created a second table that includes the true positives, false positives, true negatives, and false negatives along with the performance metrics. This helps me see how the confusion matrix values change as I change the classification threshold.

performance_table <- data.frame(
  Threshold = c(0.2, 0.5, 0.8),
  TP = c(TP_02, TP_05, TP_08),
  FP = c(FP_02, FP_05, FP_08),
  TN = c(TN_02, TN_05, TN_08),
  FN = c(FN_02, FN_05, FN_08),
  Accuracy = c(accuracy_02, accuracy_05, accuracy_08),
  Precision = c(precision_02, precision_05, precision_08),
  Recall = c(recall_02, recall_05, recall_08),
  F1 = c(f1_02, f1_05, f1_08)
)

performance_table[, 6:9] <- round(
  performance_table[, 6:9],
  4
)

performance_table
##   Threshold TP FP TN FN Accuracy Precision Recall     F1
## 1       0.2 37  6 48  2   0.9140    0.8605 0.9487 0.9024
## 2       0.5 36  3 51  3   0.9355    0.9231 0.9231 0.9231
## 3       0.8 36  2 52  3   0.9462    0.9474 0.9231 0.9351

Conclusion

In this analysis, I learned that changing the classification threshold affects the performance of a classification model. I calculated a null error rate of 41.94%, which gave me a baseline for evaluating the model.

At a threshold of 0.2, I found the highest recall at 94.87%, meaning that this threshold identified more of the actual female penguins. At a threshold of 0.8, I found the highest accuracy at 94.62%, precision at 94.74%, and F1 score at 93.51%. The 0.5 threshold produced results between the lower and higher thresholds.

Overall, I learned that there is not always one best classification threshold. I would choose a threshold based on whether reducing false negatives or reducing false positives is more important for the situation.