Introduction

This project evaluates the performance of a binary classification model on a dataset of penguin observations. The model predicts the probability that a penguin belongs to the female class. Rather than relying solely on predicted class labels, this analysis examines how different probability thresholds affect classification decisions. The analysis begins by exploring the dataset to understand the distribution of the target variable (sex) and determine whether the classes are balanced. A null error rate is then calculated to establish a baseline level of performance. Next, new class predictions are generated using probability thresholds of 0.2, 0.5, and 0.8. For each threshold, a confusion matrix is created to identify true positives, false positives, true negatives, and false negatives. These values are then used to calculate accuracy, precision, recall, and F1 score. Comparing the results across thresholds provides insight into the tradeoff between correctly identifying positive cases and minimizing classification errors.

load library and data

In this section, the required libraries are loaded, and the penguin prediction dataset is imported into R. Loading the dataset allows the observations and model predictions to be examined and prepared for analysis.

library(readr)
penguin_predictions <- read_csv("https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv")
## Rows: 93 Columns: 3
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): .pred_class, sex
## dbl (1): .pred_female
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
View(penguin_predictions)
head((penguin_predictions))
## # A tibble: 6 × 3
##   .pred_female .pred_class sex   
##          <dbl> <chr>       <chr> 
## 1        0.992 female      female
## 2        0.954 female      female
## 3        0.985 female      female
## 4        0.187 male        female
## 5        0.995 female      female
## 6        1.000 female      female

Exploratory Data Analysis

This exploratory analysis is to understand the distribution of the target variable (sex). By counting the number of male and female observations and visualizing the results with a bar chart, it becomes easier to identify whether the classes are balanced. The results show that the dataset contains more male penguins than female penguins, indicating a slight class imbalance. Understanding class distribution is important because it can influence how classification models and performance metrics should be interpreted.

count (penguin_predictions,sex)
## # A tibble: 2 × 2
##   sex        n
##   <chr>  <int>
## 1 female    39
## 2 male      54
ggplot(penguin_predictions, aes(x = sex, fill = sex)) +
 geom_bar() +
 labs(
  title = "Distribution of Actual Classes",
  x = "Sex",
  y = "Count"
 ) +
 theme_minimal()

Null Error Rate

The null error rate represents the error produced by a model that always predicts the majority class. Since male penguins represent the majority class in this dataset, a null model would classify every observation as male. This metric provides a baseline for evaluating model performance, as any useful classification model should outperform a simple majority-class prediction.

table (penguin_predictions$sex)
## 
## female   male 
##     39     54
1-(max(table (penguin_predictions$sex))/sum(table(penguin_predictions$sex)))
## [1] 0.4193548

Confusion Matrices

In this section, new predicted class labels are generated using probability thresholds of 0.2, 0.5, and 0.8. Adjusting the threshold changes how confident the model must be before classifying an observation as female. Lower thresholds tend to classify more observations as female, while higher thresholds require greater confidence and produce fewer positive classifications.

#threshhold (0.2,0.5,0.8)
penguin_predictions <- penguin_predictions %>% 
  mutate(pred_02 = ifelse(.pred_female>0.2, "female","male"),
         pred_05 = ifelse(.pred_female>0.5, "female","male"),
         pred_08 = ifelse(.pred_female>0.8, "female","male"))

head(penguin_predictions)
## # A tibble: 6 × 6
##   .pred_female .pred_class sex    pred_02 pred_05 pred_08
##          <dbl> <chr>       <chr>  <chr>   <chr>   <chr>  
## 1        0.992 female      female female  female  female 
## 2        0.954 female      female female  female  female 
## 3        0.985 female      female female  female  female 
## 4        0.187 male        female male    male    male   
## 5        0.995 female      female female  female  female 
## 6        1.000 female      female female  female  female
table (penguin_predictions$pred_02)
## 
## female   male 
##     43     50
table (penguin_predictions$pred_05)
## 
## female   male 
##     39     54
table (penguin_predictions$pred_08)
## 
## female   male 
##     38     55

Performance Metrics

Treshold_0.2

At the 0.2 threshold, the model correctly classified 37 female penguins and 48 male penguins. However, it also produced 6 false positives and 2 false negatives. Because this threshold is relatively low, the model is more likely to classify observations as female, resulting in higher recall but also more false-positive predictions.

table(
  Actual= penguin_predictions$sex,
  predicted = penguin_predictions$pred_02
)
##         predicted
## Actual   female male
##   female     37    2
##   male        6   48
treshhold =0.2
treshhold_0.2 <- data.frame(
  metric =c("TP","FP","TN","FN"),
  value = c(37,6,48,2)
)
#treshhold_0.2

TP <- treshhold_0.2$value[treshhold_0.2$metric == "TP"]
FP <- treshhold_0.2$value[treshhold_0.2$metric == "FP"]
TN <- treshhold_0.2$value[treshhold_0.2$metric == "TN"]
FN <- treshhold_0.2$value[treshhold_0.2$metric == "FN"]

metrics_02 <- data.frame(
 Metric = c("Accuracy", "Precision", "Recall", "F1 Score"),
 Value = round(c(
  (TP + TN) / (TP + TN + FP + FN),
  TP / (TP + FP),
  TP / (TP + FN),
  2 * ((TP / (TP + FP)) * (TP / (TP + FN))) /
   ((TP / (TP + FP)) + (TP / (TP + FN)))
 ), 4)
)

metrics_02
##      Metric  Value
## 1  Accuracy 0.9140
## 2 Precision 0.8605
## 3    Recall 0.9487
## 4  F1 Score 0.9024

Treshold_0.5

At the 0.5 threshold, the model correctly identified 36 female penguins and 51 male penguins. Compared to the 0.2 threshold, the number of false positives decreased from 6 to 3, while false negatives increased slightly from 2 to 3. This threshold provides a more balanced tradeoff between precision and recall.

table(
  Actual= penguin_predictions$sex,
  predicted = penguin_predictions$pred_05
)
##         predicted
## Actual   female male
##   female     36    3
##   male        3   51
treshhold =0.5
treshhold_0.5 <- data.frame(
  metric =c("TP","FP","TN","FN"),
  value = c(36,3,51,3)
)

TP <- treshhold_0.5$value[treshhold_0.5$metric == "TP"]
FP <- treshhold_0.5$value[treshhold_0.5$metric == "FP"]
TN <- treshhold_0.5$value[treshhold_0.5$metric == "TN"]
FN <- treshhold_0.5$value[treshhold_0.5$metric == "FN"]

metrics_05 <- data.frame(
 Metric = c("Accuracy", "Precision", "Recall", "F1 Score"),
 Value = round(c(
  (TP + TN) / (TP + TN + FP + FN),
  TP / (TP + FP),
  TP / (TP + FN),
  2 * ((TP / (TP + FP)) * (TP / (TP + FN))) /
   ((TP / (TP + FP)) + (TP / (TP + FN)))
 ), 4)
)

metrics_05
##      Metric  Value
## 1  Accuracy 0.9355
## 2 Precision 0.9231
## 3    Recall 0.9231
## 4  F1 Score 0.9231

Treshold_0.8

At the 0.8 threshold, the model produced the fewest false positives (2) while maintaining the same number of false negatives as the 0.5 threshold. This indicates that the model became more selective when predicting the female class, leading to improved precision and overall accuracy.

table(
  Actual= penguin_predictions$sex,
  predicted = penguin_predictions$pred_08
)
##         predicted
## Actual   female male
##   female     36    3
##   male        2   52
treshhold =0.8
treshhold_0.8 <- data.frame(
  metric =c("TP","FP","TN","FN"),
  value = c(36,2,52,3)
)

 
TP <- treshhold_0.8$value[treshhold_0.8$metric == "TP"]
FP <- treshhold_0.8$value[treshhold_0.8$metric == "FP"]
TN <- treshhold_0.8$value[treshhold_0.8$metric == "TN"]
FN <- treshhold_0.8$value[treshhold_0.8$metric == "FN"]

metrics_08 <- data.frame(
 Metric = c("Accuracy", "Precision", "Recall", "F1 Score"),
 Value = round(c(
  (TP + TN) / (TP + TN + FP + FN),
  TP / (TP + FP),
  TP / (TP + FN),
  2 * ((TP / (TP + FP)) * (TP / (TP + FN))) /
   ((TP / (TP + FP)) + (TP / (TP + FN)))
 ), 4)
)

metrics_08
##      Metric  Value
## 1  Accuracy 0.9462
## 2 Precision 0.9474
## 3    Recall 0.9231
## 4  F1 Score 0.9351

Threshold Analysis

The results demonstrate how changing the probability threshold affects classification performance. At the 0.2 threshold, the model achieved the highest recall (94.87%), meaning it identified most female penguins. However, this came at the cost of more false positives. As the threshold increased to 0.8, the number of false positives decreased, resulting in the highest precision (94.74%) and highest overall accuracy (94.62%). These results illustrate the tradeoff between identifying positive cases and minimizing incorrect positive predictions.

metrics <- data.frame(
 Threshold = c("0.2", "0.5", "0.8"),
 Accuracy = c(0.9140, 0.9355, 0.9462),
 Precision = c(0.8605, 0.9231, 0.9474),
 Recall = c(0.9487, 0.9231, 0.9231),
 F1_Score = c(0.9024, 0.9231, 0.9351)
)

metrics
##   Threshold Accuracy Precision Recall F1_Score
## 1       0.2   0.9140    0.8605 0.9487   0.9024
## 2       0.5   0.9355    0.9231 0.9231   0.9231
## 3       0.8   0.9462    0.9474 0.9231   0.9351

Real-World Use Cases

Threshold 0.2. A threshold of 0.2 may be preferred in medical screening applications where missing a positive case could have serious consequences. For example, when screening for a disease, it is often more important to identify as many potentially positive cases as possible, even if some false positives occur. In these situations, maximizing recall is typically more important than maximizing precision.

Threshold 0.8. A threshold of 0.8 may be preferred in applications where false positives are costly. Examples include spam filtering, loan approvals, or fraud investigations. In these situations, organizations may prefer to act only when the model is highly confident, prioritizing precision over recall.

Conclusion

This project evaluated the performance of a binary classification model using different probability thresholds. The null error rate established a baseline for comparison, while confusion matrices and performance metrics provided insight into how threshold selection influences model behavior. The results showed that lower thresholds generally improve recall by identifying more positive cases, whereas higher thresholds improve precision by reducing false positive predictions. Overall, the analysis demonstrates that there is no universally optimal threshold. Instead, the most appropriate threshold depends on the specific objectives of the problem and the relative costs of false positives and false negatives.