Overview

For this assignment, I will use the provided penguin predictions dataset to look at how well a classification model predicts whether a penguin is female. I want to understand what happens to the results when I change the probability threshold used by the model.

Planned Approach

First, I will read the provided article to better understand classification measurements and then load the CSV file directly into R. I will look at the columns and check how the predicted probabilities, predicted classes, and actual classes are recorded.

I will treat female as the positive class and test different probability thresholds, such as 0.20, 0.50, and 0.80. For each threshold, I will count the true positives, false positives, true negatives, and false negatives. I will use those numbers to calculate accuracy, precision, recall, and F1 score. Finally, I will place the results in a table and compare how the measurements change depending on the threshold.

Anticipated Challenges

I think the most challenging part will be keeping track of true positives, false positives, true negatives, and false negatives without mixing them up. I will also need to check for missing values and make sure I do not divide by zero while calculating the measurements. I will check each result carefully before comparing the different thresholds.

Loading and Inspecting the Data

I loaded the prediction data directly from the course GitHub repository so the original data source remains accessible through the code. I first inspected the dataframe and the values in the actual class column before performing any calculations.

data_url <- "https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv"

penguin_predictions <- read.csv(data_url)

head(penguin_predictions)
##   .pred_female .pred_class    sex
## 1    0.9921746      female female
## 2    0.9542394      female female
## 3    0.9847350      female female
## 4    0.1870206        male female
## 5    0.9947012      female female
## 6    0.9999891      female female
str(penguin_predictions)
## 'data.frame':    93 obs. of  3 variables:
##  $ .pred_female: num  0.992 0.954 0.985 0.187 0.995 ...
##  $ .pred_class : chr  "female" "female" "female" "male" ...
##  $ sex         : chr  "female" "female" "female" "female" ...
colSums(is.na(penguin_predictions))
## .pred_female  .pred_class          sex 
##            0            0            0
table(penguin_predictions$sex, useNA = "ifany")
## 
## female   male 
##     39     54

Null Error Rate and Class Distribution

The dataset contains 39 female penguins and 54 male penguins. Because male is the majority class, a null model would classify every observation as male. It would incorrectly classify the 39 female observations, producing a null error rate of approximately 41.94%.

Knowing the null error rate provides a baseline for evaluating the classification model. A useful model should perform better than simply predicting the majority class for every observation.

class_counts <- table(penguin_predictions$sex)
total_observations <- sum(class_counts)
majority_count <- max(class_counts)

null_error_rate <- 1 - (majority_count / total_observations)

null_error_results <- data.frame(
  total_observations = total_observations,
  majority_class = names(which.max(class_counts)),
  majority_count = majority_count,
  null_error_rate = round(null_error_rate, 4),
  null_error_percentage = paste0(
    round(null_error_rate * 100, 2),
    "%"
  )
)

null_error_results
##   total_observations majority_class majority_count null_error_rate
## 1                 93           male             54          0.4194
##   null_error_percentage
## 1                41.94%
barplot(
  class_counts,
  main = "Distribution of Actual Penguin Sex",
  xlab = "Actual Class",
  ylab = "Number of Penguins",
  col = c("lightcoral", "steelblue")
)

Confusion Matrices at Different Thresholds

I treated female as the positive class because .pred_female represents the predicted probability that a penguin is female. I recalculated the predicted class using thresholds of 0.2, 0.5, and 0.8.

create_confusion_matrix <- function(threshold) {
  
  actual_positive <- penguin_predictions$sex == "female"
  predicted_positive <- penguin_predictions$.pred_female > threshold
  
  TP <- sum(predicted_positive & actual_positive)
  FP <- sum(predicted_positive & !actual_positive)
  TN <- sum(!predicted_positive & !actual_positive)
  FN <- sum(!predicted_positive & actual_positive)
  
  matrix(
    c(TN, FP, FN, TP),
    nrow = 2,
    byrow = TRUE,
    dimnames = list(
      "Actual Class" = c("Male (Negative)", "Female (Positive)"),
      "Predicted Class" = c("Male (Negative)", "Female (Positive)")
    )
  )
}

confusion_matrix_0.2 <- create_confusion_matrix(0.2)
confusion_matrix_0.5 <- create_confusion_matrix(0.5)
confusion_matrix_0.8 <- create_confusion_matrix(0.8)

cat("Confusion Matrix — Threshold 0.2\n")
## Confusion Matrix — Threshold 0.2
confusion_matrix_0.2
##                    Predicted Class
## Actual Class        Male (Negative) Female (Positive)
##   Male (Negative)                48                 6
##   Female (Positive)               2                37
cat("\nConfusion Matrix — Threshold 0.5\n")
## 
## Confusion Matrix — Threshold 0.5
confusion_matrix_0.5
##                    Predicted Class
## Actual Class        Male (Negative) Female (Positive)
##   Male (Negative)                51                 3
##   Female (Positive)               3                36
cat("\nConfusion Matrix — Threshold 0.8\n")
## 
## Confusion Matrix — Threshold 0.8
confusion_matrix_0.8
##                    Predicted Class
## Actual Class        Male (Negative) Female (Positive)
##   Male (Negative)                52                 2
##   Female (Positive)               3                36

Performance Metrics

I calculated accuracy, precision, recall, and F1 score from each confusion matrix. These metrics show how the model’s performance changes when the classification threshold changes.

calculate_metrics <- function(threshold) {
  
  confusion_matrix <- create_confusion_matrix(threshold)
  
  TN <- confusion_matrix[1, 1]
  FP <- confusion_matrix[1, 2]
  FN <- confusion_matrix[2, 1]
  TP <- confusion_matrix[2, 2]
  
  accuracy <- (TP + TN) / (TP + TN + FP + FN)
  precision <- TP / (TP + FP)
  recall <- TP / (TP + FN)
  f1_score <- 2 * (precision * recall) / (precision + recall)
  
  data.frame(
    threshold = threshold,
    TP = TP,
    FP = FP,
    TN = TN,
    FN = FN,
    accuracy = accuracy,
    precision = precision,
    recall = recall,
    f1_score = f1_score
  )
}

metrics_table <- do.call(
  rbind,
  lapply(c(0.2, 0.5, 0.8), calculate_metrics)
)

metrics_table[, c("accuracy", "precision", "recall", "f1_score")] <-
  round(metrics_table[, c("accuracy", "precision", "recall", "f1_score")], 3)

metrics_table
##   threshold TP FP TN FN accuracy precision recall f1_score
## 1       0.2 37  6 48  2    0.914     0.860  0.949    0.902
## 2       0.5 36  3 51  3    0.935     0.923  0.923    0.923
## 3       0.8 36  2 52  3    0.946     0.947  0.923    0.935

Interpretation of the Metrics

At the 0.2 threshold, the model had the highest recall at 0.949 because it identified 37 of the 39 female penguins. However, its precision was lower because it also incorrectly classified six male penguins as female.

At the 0.8 threshold, the model had the highest accuracy at 0.946, the highest precision at 0.947, and the highest F1 score at 0.935. The 0.5 threshold produced balanced precision and recall values of 0.923.

Real-World Threshold Use Cases

A threshold of 0.2 could be useful during an initial medical screening for a serious disease. The lower threshold would identify more people who might have the disease, reducing the chance of missing someone who needs further testing. More false-positive results may be acceptable because patients can receive additional tests afterward.

A threshold of 0.8 could be useful when a positive prediction leads to an expensive or serious action. For example, a bank might use a higher threshold before automatically blocking a customer’s transaction for suspected fraud. The higher threshold would reduce the number of legitimate transactions incorrectly blocked.

Conclusions

The model performed better than the majority-class baseline at all three thresholds. Lowering the threshold to 0.2 increased recall, but it also created more false positives. Raising it to 0.8 reduced the false positives and produced the highest accuracy, precision, and F1 score for this dataset.

This comparison shows that a classification threshold should be selected based on the consequences of false positives and false negatives, rather than accuracy alone. I could extend this analysis by testing additional thresholds and creating an ROC curve to examine the model’s performance across a wider range of values.

Generative AI Use

I used ChatGPT to help draft and explain parts of the R code, check the metric calculations, and troubleshoot errors.

OpenAI. (2026). ChatGPT (GPT-5) [Large language model]. https://chatgpt.com. Accessed September 12, 2026.