Week 2B Approach: Evaluating Classification Model Performance

Author

Dillon Leeper

Published

September 10, 2026

Approach

I plan to use the provided penguin predictions data in R to see how changing the probability threshold affects a binary classification model. I will use sex as the actual class and .pred_female as the predicted probability of female. I will treat female as the positive class throughout the analysis.

Check the data and baseline

First, I will load the CSV into R and check the column types, missing values, and class labels. I will also check that .pred_female is between 0 and 1. If actual labels or probabilities are missing, I will report how many rows are affected and use the same complete rows for all three threshold comparisons.

I will count the female and male observations and calculate their proportions to check class balance. Then I will calculate the null error rate: the proportion of observations that would be wrong if I always predicted the most common class. This is 1 - proportion in the majority class. I will compare the model’s error rate with that baseline, or compare accuracy with the corresponding majority-class accuracy.

Compare thresholds

I will create new predicted classes at thresholds of 0.2, 0.5, and 0.8. At each threshold, I will predict female when .pred_female is greater than or equal to the threshold and male otherwise. I will derive these predictions from the probabilities instead of reusing .pred_class for every threshold.

For each threshold, I will build a confusion matrix comparing the new predictions with sex. Using female as the positive class, I will identify true positives, false positives, true negatives, and false negatives. Then I will calculate:

Accuracy: (TP + TN) / (TP + TN + FP + FN)
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
F1: 2 * TP / (2 * TP + FP + FN)

If a metric has a zero denominator, I will report it as undefined (NA) and explain why. I will put the metrics for all three thresholds in one table so that the differences are easy to compare.

What I will discuss

I expect a lower threshold to classify more observations as female. This can help catch more actual positives, but it can also create more false positives. A higher threshold requires a larger predicted probability before assigning the positive class, so it can miss more actual positives. I will use the results to see how precision, recall, and F1 change rather than assuming that the highest threshold is best.

For a low-threshold use case, I will discuss flagging items for a second review when missing a positive would be costly. For a high-threshold use case, I will discuss an alert that triggers a costly action, where false positives would be a bigger concern. I will connect these examples to the confusion matrices and explain why accuracy alone may not be enough to choose a threshold.

Load and validate the data

library(readr)
library(dplyr)
library(ggplot2)
library(knitr)

penguins_pred <- read_csv(
  "https://raw.githubusercontent.com/dillonleeper/DATA-607/main/assignments/week02/2b/penguin_predictions.csv",
  show_col_types = FALSE
)
glimpse(penguins_pred)

Rows: 93
Columns: 3
$ .pred_female <dbl> 0.99217462, 0.95423945, 0.98473504, 0.18702056, 0.9947012…
$ .pred_class  <chr> "female", "female", "female", "male", "female", "female",…
$ sex          <chr> "female", "female", "female", "female", "female", "female…

Before doing anything else, I checked the data for missing values, confirmed .pred_female is a valid probability (between 0 and 1), and confirmed the actual and predicted class columns only contain "female" and "male".

# missing values
colSums(is.na(penguins_pred))

.pred_female  .pred_class          sex 
           0            0            0

# probability range check
range(penguins_pred$.pred_female)

[1] 5.599077e-12 1.000000e+00

# class labels present
unique(penguins_pred$sex)

[1] "female" "male"

unique(penguins_pred$.pred_class)

[1] "female" "male"

There are 93 complete rows with no missing values in sex or .pred_female, so all rows are used in every threshold comparison below. .pred_female falls entirely within [0, 1] as expected for a predicted probability.

derived_0.5 <- ifelse(penguins_pred$.pred_female >= 0.5, "female", "male")
sum(derived_0.5 != penguins_pred$.pred_class)

[1] 0

As a sanity check, I also compared a class derived from .pred_female >= 0.5 against the .pred_class column already in the file: zero rows disagree, confirming .pred_class was generated at the standard 0.5 cutoff and that this same threshold logic reproduces it correctly before applying it at 0.2 and 0.8.

Class balance and the null error rate

class_counts <- penguins_pred |>
  count(sex) |>
  mutate(proportion = n / sum(n))

kable(class_counts, digits = 3)

sex	n	proportion
female	39	0.419
male	54	0.581

ggplot(penguins_pred, aes(x = sex, fill = sex)) +
  geom_bar() +
  labs(
    title = "Distribution of the Actual Class (sex)",
    x = "Actual sex",
    y = "Count"
  ) +
  theme_minimal() +
  theme(legend.position = "none")

Bar chart showing the count of female and male penguins in the actual sex column

n <- nrow(penguins_pred)
majority_class <- class_counts$sex[which.max(class_counts$n)]
majority_count <- max(class_counts$n)
null_error_rate <- 1 - (majority_count / n)

null_error_rate

[1] 0.4193548

The majority class is male (54 of 93 observations). Always predicting male would be wrong 41.9% of the time, so the null error rate is 0.419 and the corresponding null accuracy is 0.581. Any threshold I evaluate below needs to beat that baseline to be worth using over just guessing the majority class.

Threshold comparison

I derived new predicted classes directly from .pred_female at thresholds of 0.2, 0.5, and 0.8, classifying an observation as female when .pred_female is greater than or equal to the threshold and male otherwise, rather than reusing the provided .pred_class column for every threshold.

confusion_and_metrics <- function(data, threshold, positive = "female") {
  pred <- ifelse(data$.pred_female >= threshold, "female", "male")
  actual <- data$sex

  TP <- sum(pred == "female" & actual == "female")
  FP <- sum(pred == "female" & actual == "male")
  TN <- sum(pred == "male"   & actual == "male")
  FN <- sum(pred == "male"   & actual == "female")

  accuracy  <- (TP + TN) / (TP + TN + FP + FN)
  precision <- if ((TP + FP) == 0) NA_real_ else TP / (TP + FP)
  recall    <- if ((TP + FN) == 0) NA_real_ else TP / (TP + FN)
  f1        <- if (is.na(precision) || is.na(recall) || (precision + recall) == 0) {
    NA_real_
  } else {
    2 * precision * recall / (precision + recall)
  }

  list(
    threshold = threshold,
    confusion = matrix(c(TP, FP, FN, TN), nrow = 2, byrow = TRUE,
                        dimnames = list("Predicted" = c("female", "male"),
                                        "Actual"    = c("female", "male"))),
    TP = TP, FP = FP, TN = TN, FN = FN,
    accuracy = accuracy, precision = precision, recall = recall, f1 = f1
  )
}

thresholds <- c(0.2, 0.5, 0.8)
results <- lapply(thresholds, confusion_and_metrics, data = penguins_pred)
names(results) <- paste0("t_", thresholds)

Threshold = 0.2

results$t_0.2$confusion

         Actual
Predicted female male
   female     37    6
   male        2   48

Threshold = 0.5

results$t_0.5$confusion

         Actual
Predicted female male
   female     36    3
   male        3   51

Threshold = 0.8

results$t_0.8$confusion

         Actual
Predicted female male
   female     36    2
   male        3   52

Results

metrics_table <- tibble(
  Threshold = thresholds,
  TP = sapply(results, function(r) r$TP),
  FP = sapply(results, function(r) r$FP),
  TN = sapply(results, function(r) r$TN),
  FN = sapply(results, function(r) r$FN),
  Accuracy  = sapply(results, function(r) r$accuracy),
  Precision = sapply(results, function(r) r$precision),
  Recall    = sapply(results, function(r) r$recall),
  F1        = sapply(results, function(r) r$f1)
)

kable(metrics_table, digits = 3)

Threshold	TP	FP	TN	FN	Accuracy	Precision	Recall	F1
0.2	37	6	48	2	0.914	0.860	0.949	0.902
0.5	36	3	51	3	0.935	0.923	0.923	0.923
0.8	36	2	52	3	0.946	0.947	0.923	0.935

For reference, the null accuracy from always predicting the majority class (male) was 0.581. All three thresholds clear that baseline comfortably.

Discussion

Raising the threshold from 0.2 to 0.8 steadily reduced false positives (6 -> 3 -> 2) at the cost of missing one additional true positive (37 -> 36 -> 36 true positives, 2 -> 3 -> 3 false negatives), which is exactly the precision/recall tradeoff I expected going in. Accuracy and F1 both nudge upward as the threshold increases in this particular dataset, but that is a property of this data, not a general rule – precision rose (0.860 -> 0.923 -> 0.947) while recall slightly fell (0.949 -> 0.923 -> 0.923), so the “best” threshold really depends on which error type is more costly, not on which threshold produces the highest accuracy.

Low threshold (0.2) – flagging for review: a low threshold is preferable when missing an actual positive is expensive and a false positive is cheap to catch later. For example, a content-moderation model flagging posts for possible policy violations: setting the threshold low means more posts get sent for human review (more false positives), but far fewer genuine violations slip through unflagged (fewer false negatives). The cost of a reviewer clearing an innocent post is small compared to the cost of missing a real violation.

High threshold (0.8) – triggering a costly, hard-to-reverse action: a high threshold is preferable when a false positive triggers something expensive, embarrassing, or hard to undo, and it is acceptable to miss some true positives to avoid that. For example, a fraud-detection model that automatically freezes a customer’s account: freezing a legitimate customer’s account (a false positive) causes real harm and support cost, so the model should only act when it is very confident, even if that means a few actual fraud cases initially slip through and get caught by other means.

These examples map directly onto the confusion matrices above: the 0.2 threshold is the “cast a wide net” setting (lower FN, higher FP), and the 0.8 threshold is the “only act when confident” setting (lower FP, slightly higher FN). Accuracy alone would not have surfaced this tradeoff, since all three thresholds scored within about three points of each other on accuracy despite meaningfully different confusion matrices.

Conclusions

All three thresholds substantially outperform the null error rate baseline, confirming the model has real predictive value beyond guessing the majority class. There is no single “correct” threshold in the abstract – 0.2 favors catching more true positives at the expense of more false alarms, while 0.8 favors precision and minimizes false alarms at the expense of a few more missed positives. Choosing between them is a business decision about which error type is more costly in the specific application, not a modeling decision.

To extend or verify this work, I’d want to: (1) evaluate the model on a held-out or resampled set rather than the same 93 rows used to pick a threshold, since threshold selection on the evaluation set risks overfitting the cutoff to this particular sample; (2) plot a full ROC or precision-recall curve across all thresholds, not just 0.2/0.5/0.8, to see whether these three points represent the full range of tradeoffs available; and (3) check the model’s calibration (e.g., a reliability plot of predicted probability vs. observed frequency) to confirm the model’s confident predictions are genuinely well-calibrated rather than overconfident.

AI disclosure and citation

I used OpenAI’s ChatGPT to help organize the analysis plan and draft the “Approach,” “Check the data and baseline,” “Compare thresholds,” and “What I will discuss” sections of this document from the assignment instructions. Those sections describe planned work rather than completed results.

I used Claude (Anthropic, Sonnet 5, accessed via Claude for Cowork) to draft the executed R analysis: loading and validating the data, the class-balance summary and bar chart, the confusion_and_metrics() function, the confusion matrices, the results table, and the concrete low- vs. high-threshold examples in the Discussion section.

I used Claude Code (Anthropic, Sonnet 5, this CLI harness) to merge that executed analysis into this single continuous document alongside the original ChatGPT-drafted planning sections (per the assignment’s requirement to continue in the same file as the Approach deliverable), independently re-verify the null error rate, confusion matrix counts, and accuracy/precision/recall/F1 at all three thresholds against the raw CSV before merging, and fix two issues found while reviewing the code: the confusion-matrix cells originally had false positives and false negatives swapped due to a matrix() fill-order bug, and the CSV was being loaded from a local file path rather than a URL, which the course syllabus requires for reproducibility in the instructor’s environment. Claude Code then rendered this merged document end-to-end against the live raw GitHub URL to confirm it knits without errors and that every number matches the independently verified values.

AI citations:

OpenAI. (2026). ChatGPT (GPT-5.6 Sol) [Large language model]. Accessed September 10, 2026. https://chatgpt.com/.

Anthropic. (2026). Claude (Sonnet 5) [Large language model]. Accessed September 14, 2026. https://claude.ai/.

Anthropic. (2026). Claude Code (Sonnet 5) [Computer software]. https://claude.com/product/claude-code. Accessed September 14, 2026.

--- title: "Week 2B Approach: Evaluating Classification Model Performance" author: "Dillon Leeper" date: "2026-09-10" format: html: theme: cosmo toc: true toc-location: left toc-depth: 3 code-tools: true df-print: paged smooth-scroll: true embed-resources: true execute: warning: false message: false editor: visual --- ## Approach I plan to use the provided penguin predictions data in R to see how changing the probability threshold affects a binary classification model. I will use `sex` as the actual class and `.pred_female` as the predicted probability of female. I will treat female as the positive class throughout the analysis. ## Check the data and baseline First, I will load the CSV into R and check the column types, missing values, and class labels. I will also check that `.pred_female` is between 0 and 1. If actual labels or probabilities are missing, I will report how many rows are affected and use the same complete rows for all three threshold comparisons. I will count the female and male observations and calculate their proportions to check class balance. Then I will calculate the null error rate: the proportion of observations that would be wrong if I always predicted the most common class. This is `1 - proportion in the majority class`. I will compare the model's error rate with that baseline, or compare accuracy with the corresponding majority-class accuracy. ## Compare thresholds I will create new predicted classes at thresholds of 0.2, 0.5, and 0.8. At each threshold, I will predict female when `.pred_female` is greater than or equal to the threshold and male otherwise. I will derive these predictions from the probabilities instead of reusing `.pred_class` for every threshold. For each threshold, I will build a confusion matrix comparing the new predictions with `sex`. Using female as the positive class, I will identify true positives, false positives, true negatives, and false negatives. Then I will calculate: - Accuracy: `(TP + TN) / (TP + TN + FP + FN)` - Precision: `TP / (TP + FP)` - Recall: `TP / (TP + FN)` - F1: `2 * TP / (2 * TP + FP + FN)` If a metric has a zero denominator, I will report it as undefined (`NA`) and explain why. I will put the metrics for all three thresholds in one table so that the differences are easy to compare. ## What I will discuss I expect a lower threshold to classify more observations as female. This can help catch more actual positives, but it can also create more false positives. A higher threshold requires a larger predicted probability before assigning the positive class, so it can miss more actual positives. I will use the results to see how precision, recall, and F1 change rather than assuming that the highest threshold is best. For a low-threshold use case, I will discuss flagging items for a second review when missing a positive would be costly. For a high-threshold use case, I will discuss an alert that triggers a costly action, where false positives would be a bigger concern. I will connect these examples to the confusion matrices and explain why accuracy alone may not be enough to choose a threshold. ## Load and validate the data ```{r setup, message=FALSE, warning=FALSE} library(readr) library(dplyr) library(ggplot2) library(knitr) ``` ```{r load-data} penguins_pred <- read_csv( "https://raw.githubusercontent.com/dillonleeper/DATA-607/main/assignments/week02/2b/penguin_predictions.csv", show_col_types = FALSE ) glimpse(penguins_pred) ``` Before doing anything else, I checked the data for missing values, confirmed `.pred_female` is a valid probability (between 0 and 1), and confirmed the actual and predicted class columns only contain `"female"` and `"male"`. ```{r validate} # missing values colSums(is.na(penguins_pred)) # probability range check range(penguins_pred$.pred_female) # class labels present unique(penguins_pred$sex) unique(penguins_pred$.pred_class) ``` There are `r nrow(penguins_pred)` complete rows with no missing values in `sex` or `.pred_female`, so all rows are used in every threshold comparison below. `.pred_female` falls entirely within `[0, 1]` as expected for a predicted probability. ```{r sanity-check-0.5} derived_0.5 <- ifelse(penguins_pred$.pred_female >= 0.5, "female", "male") sum(derived_0.5 != penguins_pred$.pred_class) ``` As a sanity check, I also compared a class derived from `.pred_female >= 0.5` against the `.pred_class` column already in the file: zero rows disagree, confirming `.pred_class` was generated at the standard 0.5 cutoff and that this same threshold logic reproduces it correctly before applying it at 0.2 and 0.8. ## Class balance and the null error rate ```{r class-balance} class_counts <- penguins_pred |> count(sex) |> mutate(proportion = n / sum(n)) kable(class_counts, digits = 3) ``` ```{r class-balance-plot} #| fig-alt: "Bar chart showing the count of female and male penguins in the actual sex column" ggplot(penguins_pred, aes(x = sex, fill = sex)) + geom_bar() + labs( title = "Distribution of the Actual Class (sex)", x = "Actual sex", y = "Count" ) + theme_minimal() + theme(legend.position = "none") ``` ```{r null-error} n <- nrow(penguins_pred) majority_class <- class_counts$sex[which.max(class_counts$n)] majority_count <- max(class_counts$n) null_error_rate <- 1 - (majority_count / n) null_error_rate ``` The majority class is **`r majority_class`** (`r majority_count` of `r n` observations). Always predicting `r majority_class` would be wrong `r scales::percent(null_error_rate, accuracy = 0.1)` of the time, so the null error rate is **`r round(null_error_rate, 3)`** and the corresponding null accuracy is **`r round(1 - null_error_rate, 3)`**. Any threshold I evaluate below needs to beat that baseline to be worth using over just guessing the majority class. ## Threshold comparison I derived new predicted classes directly from `.pred_female` at thresholds of 0.2, 0.5, and 0.8, classifying an observation as `female` when `.pred_female` is greater than or equal to the threshold and `male` otherwise, rather than reusing the provided `.pred_class` column for every threshold. ```{r metrics-function} confusion_and_metrics <- function(data, threshold, positive = "female") { pred <- ifelse(data$.pred_female >= threshold, "female", "male") actual <- data$sex TP <- sum(pred == "female" & actual == "female") FP <- sum(pred == "female" & actual == "male") TN <- sum(pred == "male" & actual == "male") FN <- sum(pred == "male" & actual == "female") accuracy <- (TP + TN) / (TP + TN + FP + FN) precision <- if ((TP + FP) == 0) NA_real_ else TP / (TP + FP) recall <- if ((TP + FN) == 0) NA_real_ else TP / (TP + FN) f1 <- if (is.na(precision) || is.na(recall) || (precision + recall) == 0) { NA_real_ } else { 2 * precision * recall / (precision + recall) } list( threshold = threshold, confusion = matrix(c(TP, FP, FN, TN), nrow = 2, byrow = TRUE, dimnames = list("Predicted" = c("female", "male"), "Actual" = c("female", "male"))), TP = TP, FP = FP, TN = TN, FN = FN, accuracy = accuracy, precision = precision, recall = recall, f1 = f1 ) } thresholds <- c(0.2, 0.5, 0.8) results <- lapply(thresholds, confusion_and_metrics, data = penguins_pred) names(results) <- paste0("t_", thresholds) ``` ### Threshold = 0.2 ```{r cm-02} results$t_0.2$confusion ``` ### Threshold = 0.5 ```{r cm-05} results$t_0.5$confusion ``` ### Threshold = 0.8 ```{r cm-08} results$t_0.8$confusion ``` ## Results ```{r results-table} metrics_table <- tibble( Threshold = thresholds, TP = sapply(results, function(r) r$TP), FP = sapply(results, function(r) r$FP), TN = sapply(results, function(r) r$TN), FN = sapply(results, function(r) r$FN), Accuracy = sapply(results, function(r) r$accuracy), Precision = sapply(results, function(r) r$precision), Recall = sapply(results, function(r) r$recall), F1 = sapply(results, function(r) r$f1) ) kable(metrics_table, digits = 3) ``` For reference, the null accuracy from always predicting the majority class (`r majority_class`) was `r round(1 - null_error_rate, 3)`. All three thresholds clear that baseline comfortably. ## Discussion Raising the threshold from 0.2 to 0.8 steadily reduced false positives (6 -\> 3 -\> 2) at the cost of missing one additional true positive (37 -\> 36 -\> 36 true positives, 2 -\> 3 -\> 3 false negatives), which is exactly the precision/recall tradeoff I expected going in. Accuracy and F1 both nudge upward as the threshold increases in this particular dataset, but that is a property of this data, not a general rule -- precision rose (0.860 -\> 0.923 -\> 0.947) while recall slightly fell (0.949 -\> 0.923 -\> 0.923), so the "best" threshold really depends on which error type is more costly, not on which threshold produces the highest accuracy. **Low threshold (0.2) -- flagging for review:** a low threshold is preferable when missing an actual positive is expensive and a false positive is cheap to catch later. For example, a content-moderation model flagging posts for possible policy violations: setting the threshold low means more posts get sent for human review (more false positives), but far fewer genuine violations slip through unflagged (fewer false negatives). The cost of a reviewer clearing an innocent post is small compared to the cost of missing a real violation. **High threshold (0.8) -- triggering a costly, hard-to-reverse action:** a high threshold is preferable when a false positive triggers something expensive, embarrassing, or hard to undo, and it is acceptable to miss some true positives to avoid that. For example, a fraud-detection model that automatically freezes a customer's account: freezing a legitimate customer's account (a false positive) causes real harm and support cost, so the model should only act when it is very confident, even if that means a few actual fraud cases initially slip through and get caught by other means. These examples map directly onto the confusion matrices above: the 0.2 threshold is the "cast a wide net" setting (lower FN, higher FP), and the 0.8 threshold is the "only act when confident" setting (lower FP, slightly higher FN). Accuracy alone would not have surfaced this tradeoff, since all three thresholds scored within about three points of each other on accuracy despite meaningfully different confusion matrices. ## Conclusions All three thresholds substantially outperform the null error rate baseline, confirming the model has real predictive value beyond guessing the majority class. There is no single "correct" threshold in the abstract -- 0.2 favors catching more true positives at the expense of more false alarms, while 0.8 favors precision and minimizes false alarms at the expense of a few more missed positives. Choosing between them is a business decision about which error type is more costly in the specific application, not a modeling decision. To extend or verify this work, I'd want to: (1) evaluate the model on a held-out or resampled set rather than the same 93 rows used to pick a threshold, since threshold selection on the evaluation set risks overfitting the cutoff to this particular sample; (2) plot a full ROC or precision-recall curve across all thresholds, not just 0.2/0.5/0.8, to see whether these three points represent the full range of tradeoffs available; and (3) check the model's calibration (e.g., a reliability plot of predicted probability vs. observed frequency) to confirm the model's confident predictions are genuinely well-calibrated rather than overconfident. ## AI disclosure and citation I used OpenAI's ChatGPT to help organize the analysis plan and draft the "Approach," "Check the data and baseline," "Compare thresholds," and "What I will discuss" sections of this document from the assignment instructions. Those sections describe planned work rather than completed results. I used Claude (Anthropic, Sonnet 5, accessed via Claude for Cowork) to draft the executed R analysis: loading and validating the data, the class-balance summary and bar chart, the `confusion_and_metrics()` function, the confusion matrices, the results table, and the concrete low- vs. high-threshold examples in the Discussion section. I used Claude Code (Anthropic, Sonnet 5, this CLI harness) to merge that executed analysis into this single continuous document alongside the original ChatGPT-drafted planning sections (per the assignment's requirement to continue in the same file as the Approach deliverable), independently re-verify the null error rate, confusion matrix counts, and accuracy/precision/recall/F1 at all three thresholds against the raw CSV before merging, and fix two issues found while reviewing the code: the confusion-matrix cells originally had false positives and false negatives swapped due to a `matrix()` fill-order bug, and the CSV was being loaded from a local file path rather than a URL, which the course syllabus requires for reproducibility in the instructor's environment. Claude Code then rendered this merged document end-to-end against the live raw GitHub URL to confirm it knits without errors and that every number matches the independently verified values. AI citations: OpenAI. (2026). *ChatGPT (GPT-5.6 Sol)* \[Large language model\]. Accessed September 10, 2026. <https://chatgpt.com/>. Anthropic. (2026). *Claude (Sonnet 5)* \[Large language model\]. Accessed September 14, 2026. <https://claude.ai/>. Anthropic. (2026). *Claude Code (Sonnet 5)* \[Computer software\]. <https://claude.com/product/claude-code>. Accessed September 14, 2026.