Introduction

We are evaluating a binary classification model that predicts penguin sex. The dataset (penguin_predictions.csv) has the model’s predicted probability of “female,” its default predicted class, and the actual sex label. The goal is to understand how changing the probability threshold affects model performance.

Body

penguins <- read_csv("https://raw.githubusercontent.com/acatlin/data/master/penguin_predictions.csv")
## Rows: 93 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): .pred_class, sex
## dbl (1): .pred_female
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
glimpse(penguins)
## Rows: 93
## Columns: 3
## $ .pred_female <dbl> 0.99217462, 0.95423945, 0.98473504, 0.18702056, 0.9947012…
## $ .pred_class  <chr> "female", "female", "female", "male", "female", "female",…
## $ sex          <chr> "female", "female", "female", "female", "female", "female…
class_counts <- count(penguins, sex)
class_counts
## # A tibble: 2 × 2
##   sex        n
##   <chr>  <int>
## 1 female    39
## 2 male      54
majority_count <- max(class_counts$n)
total <- nrow(penguins)
null_error_rate <- 1 - (majority_count / total)
null_error_rate
## [1] 0.4193548
ggplot(penguins, aes(x = sex)) +
  geom_bar(fill = "steelblue") +
  labs(title = "Distribution of Actual Sex Class", x = "Sex", y = "Count") +
  theme_minimal()

threshold_results <- function(threshold) {
  pred <- ifelse(penguins$.pred_female > threshold, "female", "male")

  TP <- sum(pred == "female" & penguins$sex == "female")
  FP <- sum(pred == "female" & penguins$sex == "male")
  TN <- sum(pred == "male"   & penguins$sex == "male")
  FN <- sum(pred == "male"   & penguins$sex == "female")

  tibble(threshold = threshold, TP = TP, FP = FP, TN = TN, FN = FN)
}

thresholds <- c(0.2, 0.5, 0.8)
confusion_table <- bind_rows(lapply(thresholds, threshold_results))
confusion_table
## # A tibble: 3 × 5
##   threshold    TP    FP    TN    FN
##       <dbl> <int> <int> <int> <int>
## 1       0.2    37     6    48     2
## 2       0.5    36     3    51     3
## 3       0.8    36     2    52     3
for (t in thresholds) {
  pred <- ifelse(penguins$.pred_female > t, "female", "male")

  cm <- matrix(
    c(
      sum(pred == "female" & penguins$sex == "female"),
      sum(pred == "male"   & penguins$sex == "female"),
      sum(pred == "female" & penguins$sex == "male"),
      sum(pred == "male"   & penguins$sex == "male")
    ),
    nrow = 2,
    dimnames = list(
      "Predicted" = c("Female", "Male"),
      "Actual"    = c("Female", "Male")
    )
  )

  print(paste("Threshold:", t))
  print(kable(cm))
}
## [1] "Threshold: 0.2"
## 
## 
## |       | Female| Male|
## |:------|------:|----:|
## |Female |     37|    6|
## |Male   |      2|   48|
## [1] "Threshold: 0.5"
## 
## 
## |       | Female| Male|
## |:------|------:|----:|
## |Female |     36|    3|
## |Male   |      3|   51|
## [1] "Threshold: 0.8"
## 
## 
## |       | Female| Male|
## |:------|------:|----:|
## |Female |     36|    2|
## |Male   |      3|   52|
metrics_table <- confusion_table %>%
  mutate(
    accuracy  = (TP + TN) / (TP + FP + TN + FN),
    precision = TP / (TP + FP),
    recall    = TP / (TP + FN),
    f1        = 2 * TP / (2 * TP + FP + FN)
  )

metrics_table
## # A tibble: 3 × 9
##   threshold    TP    FP    TN    FN accuracy precision recall    f1
##       <dbl> <int> <int> <int> <int>    <dbl>     <dbl>  <dbl> <dbl>
## 1       0.2    37     6    48     2    0.914     0.860  0.949 0.902
## 2       0.5    36     3    51     3    0.935     0.923  0.923 0.923
## 3       0.8    36     2    52     3    0.946     0.947  0.923 0.935

Conclusions

The null error rate for this dataset is 41.9% always guessing “male” (the majority class) would be wrong nearly 42% of the time. Every threshold tested dramatically outperforms this baseline, with accuracy ranging from 91.4% (threshold 0.2) to 94.6% (threshold 0.8), confirming the model carries genuine predictive signal rather than just reflecting class imbalance.

As the threshold increases from 0.2 to 0.8, precision rises steadily (0.860 to 0.947) while recall stays roughly flat (0.949 to 0.923), meaning higher thresholds mostly filtered out false positives here without costing many true positives, which is why accuracy and F1 both peaked at the highest threshold (0.8) for this particular dataset.

A 0.2 threshold is better when missing a true positive is costly and false alarms are cheap to double check, for example, if researchers are using this model to flag candidate female penguins for a follow-up field study, they’d rather over flag and manually verify a few extra birds than miss real females entirely.

A 0.8 threshold is preferable when a false positive carries real downstream cost, for example, if the sex prediction feeds directly into a genetics or population dataset without human review, a wrongly-labeled penguin could quietly corrupt that dataset, so it’s worth being more conservative before committing to a “female” label.