Introduction

This project evaluates a binary classification model using the assigned article Performance Metrics for Classification Problems in Machine Learning as background for interpreting confusion matrices and classification metrics. The analysis uses predicted probabilities from the penguin_predictions.csv dataset to examine how thresholds of 0.2, 0.5, and 0.8 affect model decisions and performance.

Planned Approach

I will begin by examining the distribution of the actual target variable, sex, and calculating the null error rate. I will then use .pred_female to create new predictions at thresholds of 0.2, 0.5, and 0.8, calculate a confusion matrix for each threshold, and compare accuracy, precision, recall, and F1 score.

The main data question is: How does changing the probability threshold affect the types of classification errors the model makes and the resulting performance metrics?

Load Packages and Data

The dataset is read directly from the course GitHub repository so that the analysis is reproducible without relying on a local file path.

# Load packages used for visualization and formatted tables
library(ggplot2)
library(knitr)

# Read the dataset directly from the course GitHub repository
data_url <- "https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv"

penguins <- read.csv(
  data_url,
  check.names = FALSE,
  stringsAsFactors = FALSE
)

# Inspect the first rows and structure of the dataset
head(penguins)
##   .pred_female .pred_class    sex
## 1    0.9921746      female female
## 2    0.9542394      female female
## 3    0.9847350      female female
## 4    0.1870206        male female
## 5    0.9947012      female female
## 6    0.9999891      female female
str(penguins)
## 'data.frame':    93 obs. of  3 variables:
##  $ .pred_female: num  0.992 0.954 0.985 0.187 0.995 ...
##  $ .pred_class : chr  "female" "female" "female" "male" ...
##  $ sex         : chr  "female" "female" "female" "female" ...

Explore the Actual Class Distribution

Before evaluating the model, I first examine the balance of the actual outcome variable. This is important because accuracy can be misleading when one class is much more common than the other.

# Count the number of observations in each actual class
class_counts <- as.data.frame(table(penguins$sex))
names(class_counts) <- c("sex", "count")

class_counts
##      sex count
## 1 female    39
## 2   male    54
# Plot the distribution of the actual class
ggplot(class_counts, aes(x = sex, y = count, fill = sex)) +
  geom_col(width = 0.65, show.legend = FALSE) +
  geom_text(aes(label = count), vjust = -0.4) +
  labs(
    title = "Distribution of Actual Penguin Sex",
    subtitle = "Actual class labels in the prediction dataset",
    x = "Actual Sex",
    y = "Number of Observations"
  ) +
  expand_limits(y = max(class_counts$count) + 5) +
  theme_minimal()

The dataset contains more male observations than female observations. Because the classes are not perfectly balanced, I use the null error rate as a baseline before interpreting the model’s accuracy.

Null Error Rate

The null model predicts the majority class for every observation. Since male is the majority class, the null model would classify every penguin as male and would be wrong for every female observation.

# Identify the majority class and calculate the null error rate
majority_class <- class_counts$sex[which.max(class_counts$count)]
majority_count <- max(class_counts$count)
total_observations <- nrow(penguins)

null_accuracy <- majority_count / total_observations
null_error_rate <- 1 - null_accuracy

null_results <- data.frame(
  majority_class = majority_class,
  total_observations = total_observations,
  majority_count = majority_count,
  null_accuracy = null_accuracy,
  null_error_rate = null_error_rate
)

kable(
  null_results,
  digits = 3,
  caption = "Null Model Baseline"
)
Null Model Baseline
majority_class total_observations majority_count null_accuracy null_error_rate
male 93 54 0.581 0.419

The majority class is male, with 54 of the 93 observations. A model that predicts male for every case would therefore have a null accuracy of approximately 58.1% and a null error rate of approximately 41.9%. The classification model should perform meaningfully better than this simple baseline.

Create Predictions at Multiple Thresholds

The provided .pred_class column uses a threshold of 0.5. For this assignment, I recompute the predicted class directly from .pred_female so that the effect of different thresholds can be compared consistently.

A prediction is classified as female when .pred_female is greater than the selected threshold. Female is therefore treated as the positive class and male as the negative class.

# Function to calculate confusion-matrix counts and performance metrics
evaluate_threshold <- function(data, threshold) {

  # Create a predicted class using the selected probability threshold
  predicted <- ifelse(data$.pred_female > threshold, "female", "male")
  actual <- data$sex

  # Calculate the four confusion-matrix components
  TP <- sum(predicted == "female" & actual == "female")
  FP <- sum(predicted == "female" & actual == "male")
  TN <- sum(predicted == "male" & actual == "male")
  FN <- sum(predicted == "male" & actual == "female")

  # Derive performance metrics from the confusion-matrix counts
  accuracy <- (TP + TN) / (TP + FP + TN + FN)

  precision <- if ((TP + FP) == 0) {
    NA
  } else {
    TP / (TP + FP)
  }

  recall <- if ((TP + FN) == 0) {
    NA
  } else {
    TP / (TP + FN)
  }

  f1 <- if (is.na(precision) || is.na(recall) || (precision + recall) == 0) {
    NA
  } else {
    2 * precision * recall / (precision + recall)
  }

  data.frame(
    threshold = threshold,
    TP = TP,
    FP = FP,
    TN = TN,
    FN = FN,
    accuracy = accuracy,
    precision = precision,
    recall = recall,
    F1 = f1
  )
}

Confusion Matrices

I evaluate the model at thresholds of 0.2, 0.5, and 0.8. A lower threshold should generally classify more observations as positive, while a higher threshold should classify fewer observations as positive.

Threshold = 0.2

# Create predictions at a threshold of 0.2
pred_02 <- ifelse(penguins$.pred_female > 0.2, "female", "male")

# Build the confusion matrix
cm_02 <- table(
  Predicted = pred_02,
  Actual = penguins$sex
)

cm_02
##          Actual
## Predicted female male
##    female     37    6
##    male        2   48

At the 0.2 threshold, the model identifies 37 true positives and 48 true negatives. It also produces 6 false positives and 2 false negatives. The lower threshold captures nearly all of the actual female observations, but it also incorrectly classifies more male observations as female.

Threshold = 0.5

# Create predictions at a threshold of 0.5
pred_05 <- ifelse(penguins$.pred_female > 0.5, "female", "male")

# Build the confusion matrix
cm_05 <- table(
  Predicted = pred_05,
  Actual = penguins$sex
)

cm_05
##          Actual
## Predicted female male
##    female     36    3
##    male        3   51

At the 0.5 threshold, the model produces 36 true positives and 51 true negatives, with 3 false positives and 3 false negatives. Compared with the 0.2 threshold, the model makes fewer false-positive predictions while missing one additional female observation.

Threshold = 0.8

# Create predictions at a threshold of 0.8
pred_08 <- ifelse(penguins$.pred_female > 0.8, "female", "male")

# Build the confusion matrix
cm_08 <- table(
  Predicted = pred_08,
  Actual = penguins$sex
)

cm_08
##          Actual
## Predicted female male
##    female     36    2
##    male        3   52

At the 0.8 threshold, the model produces 36 true positives and 52 true negatives, with 2 false positives and 3 false negatives. In this dataset, increasing the threshold from 0.5 to 0.8 removes one false positive without creating any additional false negatives.

Performance Metrics

The confusion-matrix counts are used to calculate accuracy, precision, recall, and F1 score for each threshold. Comparing the metrics together provides more information than looking at accuracy alone.

# Evaluate all three required thresholds
metrics_02 <- evaluate_threshold(penguins, 0.2)
metrics_05 <- evaluate_threshold(penguins, 0.5)
metrics_08 <- evaluate_threshold(penguins, 0.8)

# Combine the results into one table
metrics_table <- rbind(
  metrics_02,
  metrics_05,
  metrics_08
)

# Display the metrics as percentages for easier interpretation
metrics_display <- metrics_table

metrics_display$accuracy <- round(metrics_display$accuracy * 100, 1)
metrics_display$precision <- round(metrics_display$precision * 100, 1)
metrics_display$recall <- round(metrics_display$recall * 100, 1)
metrics_display$F1 <- round(metrics_display$F1 * 100, 1)

names(metrics_display)[6:9] <- c(
  "Accuracy (%)",
  "Precision (%)",
  "Recall (%)",
  "F1 (%)"
)

kable(
  metrics_display,
  caption = "Classification Performance at Three Probability Thresholds"
)
Classification Performance at Three Probability Thresholds
threshold TP FP TN FN Accuracy (%) Precision (%) Recall (%) F1 (%)
0.2 37 6 48 2 91.4 86.0 94.9 90.2
0.5 36 3 51 3 93.5 92.3 92.3 92.3
0.8 36 2 52 3 94.6 94.7 92.3 93.5

The 0.2 threshold has the highest recall at approximately 94.9%, which is consistent with the idea that a lower threshold makes it easier for an observation to be classified as positive. However, precision is lower because the model also produces more false positives.

The results at 0.5 and 0.8 are somewhat different from my original expectation. I expected recall to decrease further at 0.8, but both thresholds produce the same 36 true positives and 3 false negatives, so recall remains approximately 92.3%. The 0.8 threshold reduces false positives from 3 to 2, which increases precision to approximately 94.7%, accuracy to approximately 94.6%, and F1 to approximately 93.5%.

Comparing the Metrics Visually

A visual comparison makes it easier to see how the threshold affects the balance among the evaluation metrics.

# Reshape the metric values into long form using base R
metrics_plot <- data.frame(
  threshold = rep(metrics_table$threshold, 4),
  metric = rep(
    c("Accuracy", "Precision", "Recall", "F1"),
    each = nrow(metrics_table)
  ),
  value = c(
    metrics_table$accuracy,
    metrics_table$precision,
    metrics_table$recall,
    metrics_table$F1
  )
)

# Compare all four performance metrics across the three thresholds
ggplot(
  metrics_plot,
  aes(
    x = factor(threshold),
    y = value,
    group = metric,
    linetype = metric,
    shape = metric
  )
) +
  geom_line(linewidth = 0.8) +
  geom_point(size = 2.5) +
  scale_y_continuous(
    labels = function(x) paste0(round(x * 100), "%"),
    limits = c(0.80, 1.00)
  ) +
  labs(
    title = "Model Performance Changes Across Probability Thresholds",
    x = "Probability Threshold",
    y = "Metric Value",
    linetype = "Metric",
    shape = "Metric"
  ) +
  theme_minimal()

The plot shows that lowering the threshold to 0.2 improves recall but decreases precision. In this particular dataset, the 0.8 threshold provides the strongest overall combination of accuracy, precision, and F1 while maintaining the same recall as the 0.5 threshold.

Threshold Interpretation

The best threshold depends on the consequences of false positives and false negatives rather than on a single metric alone. The assigned reading emphasizes that the relative importance of these errors depends on the real-world context of the classification problem.

When a 0.2 Threshold May Be Preferable

A lower threshold such as 0.2 may be appropriate for a medical screening system designed to identify patients who may need additional testing. In that setting, missing a person who truly has a serious condition could be more harmful than sending some people without the condition for additional evaluation.

The lower threshold makes the model more willing to predict the positive class. In this penguin dataset, the 0.2 threshold has the highest recall and only 2 false negatives, but it produces more false positives than the higher thresholds. This illustrates why a lower threshold may be appropriate when reducing false negatives is the priority.

When a 0.8 Threshold May Be Preferable

A higher threshold such as 0.8 may be appropriate in a system where a positive classification triggers an expensive or consequential action. For example, a fraud-detection system might use a high threshold before automatically blocking a transaction, while lower-probability cases are sent for manual review.

A higher threshold requires stronger model confidence before assigning the positive class. In this dataset, the 0.8 threshold produces only 2 false positives and has the highest precision of the three thresholds.

Conclusions

The model performs substantially better than the null baseline at all three thresholds. The null model would have an accuracy of approximately 58.1%, while the tested classification thresholds produce accuracies above 91%.

The threshold changes the balance between false positives and false negatives. At 0.2, the model achieves the highest recall because it identifies 37 of the 39 actual female observations, but this comes with 6 false positives. At 0.5, false positives fall to 3 while false negatives increase to 3. At 0.8, the model reduces false positives again to 2 without increasing false negatives beyond the 3 observed at the 0.5 threshold.

An important result is that the data do not show a simple continuous decline in recall as the threshold rises. Although I originally expected the 0.8 threshold to miss more positive cases than the 0.5 threshold, both thresholds have the same recall in this dataset. This reinforces the importance of evaluating thresholds empirically rather than assuming that every threshold change will affect the observed sample in exactly the same way.

Recommendations and Future Work

If the goal is to minimize false negatives, the 0.2 threshold would be the better choice among the three tested thresholds because it produces the highest recall. If reducing false positives is more important, the 0.8 threshold is more appropriate for this dataset because it has the highest precision and the fewest false positives.

A useful extension would be to evaluate additional thresholds between 0 and 1 rather than examining only three values. This could be used to create a precision-recall curve or ROC curve and provide a more complete picture of model performance. The analysis could also be repeated on a separate test dataset to verify that the threshold behavior observed here generalizes to new data.