Introduction

This analysis evaluates the performance of a binary classification model that predicts the probability that a penguin belongs to the female class. The analysis examines how changing the classification threshold affects the model’s predictions and performance.

The null error rate will first be calculated to establish a baseline for comparison. The model will then be evaluated using probability thresholds of 0.2, 0.5, and 0.8. Confusion matrices and performance metrics including accuracy, precision, recall, and F1 score will be used to compare the results.

library(readr)
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
library(ggplot2)
penguins <- read_csv(
  "https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv"
)
## Rows: 93 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): .pred_class, sex
## dbl (1): .pred_female
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
head(penguins)
## # A tibble: 6 × 3
##   .pred_female .pred_class sex   
##          <dbl> <chr>       <chr> 
## 1        0.992 female      female
## 2        0.954 female      female
## 3        0.985 female      female
## 4        0.187 male        female
## 5        0.995 female      female
## 6        1.000 female      female
str(penguins)
## spc_tbl_ [93 × 3] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ .pred_female: num [1:93] 0.992 0.954 0.985 0.187 0.995 ...
##  $ .pred_class : chr [1:93] "female" "female" "female" "male" ...
##  $ sex         : chr [1:93] "female" "female" "female" "female" ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   .pred_female = col_double(),
##   ..   .pred_class = col_character(),
##   ..   sex = col_character()
##   .. )
##  - attr(*, "problems")=<externalptr>

Data Cleaning

colSums(is.na(penguins))
## .pred_female  .pred_class          sex 
##            0            0            0

There are no missing values in the provided dataset, so no rows need to be removed or modified before analysis.

Data Analysis

NULL error rate

The null error rate represents the error rate that would result from predicting the majority class for every observation. This establishes a baseline that can be used to determine whether the classification model performs better than a simple majority-class prediction.

class_balance <- penguins %>%
  count(sex)

class_balance
## # A tibble: 2 × 2
##   sex        n
##   <chr>  <int>
## 1 female    39
## 2 male      54
ggplot(penguins, aes(x = sex)) +
  geom_bar() +
  labs(title = "Distribution of Actual Penguin Sex",x = "Sex",y = "Count")

null_error_rate <- 1 - max(class_balance$n) / sum(class_balance$n)

null_error_rate
## [1] 0.4193548

Confusion Matrices

Because .pred_female represents the probability that the observation is female, female will be treated as the positive class (1) and male as the negative class (0).

penguins <- penguins %>%
  mutate(actual = ifelse(sex == "female", 1, 0))

Threshold = 0.2

A probability greater than 0.2 will be classified as female.

pred_02 <- ifelse(penguins$.pred_female > 0.2, 1, 0)

TP_02 <- sum(pred_02 == 1 & penguins$actual == 1)
FP_02 <- sum(pred_02 == 1 & penguins$actual == 0)
TN_02 <- sum(pred_02 == 0 & penguins$actual == 0)
FN_02 <- sum(pred_02 == 0 & penguins$actual == 1)

cm_02 <- matrix(
  c(TN_02, FP_02,
    FN_02, TP_02),
  nrow = 2,
  byrow = TRUE,
  dimnames = list(
    Actual = c("Male (0)", "Female (1)"),
    Predicted = c("Male (0)", "Female (1)")
  )
)

cm_02
##             Predicted
## Actual       Male (0) Female (1)
##   Male (0)         48          6
##   Female (1)        2         37

Threshold = 0.5

pred_05 <- ifelse(penguins$.pred_female > 0.5, 1, 0)

TP_05 <- sum(pred_05 == 1 & penguins$actual == 1)
FP_05 <- sum(pred_05 == 1 & penguins$actual == 0)
TN_05 <- sum(pred_05 == 0 & penguins$actual == 0)
FN_05 <- sum(pred_05 == 0 & penguins$actual == 1)

cm_05 <- matrix(
  c(TN_05, FP_05,
    FN_05, TP_05),
  nrow = 2,
  byrow = TRUE,
  dimnames = list(
    Actual = c("Male (0)", "Female (1)"),
    Predicted = c("Male (0)", "Female (1)")
  )
)

cm_05
##             Predicted
## Actual       Male (0) Female (1)
##   Male (0)         51          3
##   Female (1)        3         36

Threshold = 0.8

pred_08 <- ifelse(penguins$.pred_female > 0.8, 1, 0)

TP_08 <- sum(pred_08 == 1 & penguins$actual == 1)
FP_08 <- sum(pred_08 == 1 & penguins$actual == 0)
TN_08 <- sum(pred_08 == 0 & penguins$actual == 0)
FN_08 <- sum(pred_08 == 0 & penguins$actual == 1)

cm_08 <- matrix(
  c(TN_08, FP_08,
    FN_08, TP_08),
  nrow = 2,
  byrow = TRUE,
  dimnames = list(
    Actual = c("Male (0)", "Female (1)"),
    Predicted = c("Male (0)", "Female (1)")
  )
)

cm_08
##             Predicted
## Actual       Male (0) Female (1)
##   Male (0)         52          2
##   Female (1)        3         36

Performance Metrics

The confusion-matrix results can now be used to calculate accuracy, precision, recall, and F1 score for each threshold.

metrics <- data.frame(
  threshold = c(0.2, 0.5, 0.8),
  TP = c(TP_02, TP_05, TP_08),
  FP = c(FP_02, FP_05, FP_08),
  TN = c(TN_02, TN_05, TN_08),
  FN = c(FN_02, FN_05, FN_08)
)

metrics <- metrics %>%
  mutate(
    accuracy = (TP + TN) / (TP + TN + FP + FN),
    precision = TP / (TP + FP),
    recall = TP / (TP + FN),
    F1 = 2 * (precision * recall) / (precision + recall)
  )

metrics
##   threshold TP FP TN FN  accuracy precision    recall        F1
## 1       0.2 37  6 48  2 0.9139785 0.8604651 0.9487179 0.9024390
## 2       0.5 36  3 51  3 0.9354839 0.9230769 0.9230769 0.9230769
## 3       0.8 36  2 52  3 0.9462366 0.9473684 0.9230769 0.9350649

Threshold Use Cases

A 0.2 threshold may be useful when detecting as many positive cases as possible is more important than avoiding false positives. For example, in an initial medical screening, a lower threshold could identify more patients who may have a condition for additional testing. This favors recall because missing a true positive may have greater consequences than performing additional testing on a false positive.

A 0.8 threshold may be more appropriate when a positive classification should only be made with a high level of confidence. For example, an automated fraud system that suspends an account may use a higher threshold to reduce the number of legitimate users incorrectly classified as fraudulent. This places greater importance on precision and reducing false positives.

Conclusion

The results demonstrate that changing the probability threshold changes the balance between false positives and false negatives. At the 0.2 threshold, the model produced the highest recall, approximately 94.9%, but also produced more false positives and lower precision.

Increasing the threshold reduced the number of false positives. The 0.8 threshold produced the highest accuracy and precision in this dataset, while recall remained slightly lower than at the 0.2 threshold. All three thresholds performed substantially better than the 41.9% null error rate. The results demonstrate why the appropriate classification threshold depends on whether false positives or false negatives are more costly for a particular application.