Introduction

For this assignment, I will analyze the performance of a binary classification model using the penguin_predictions.csv dataset provided in the course materials. The dataset contains the predicted probability that each penguin is female, the predicted class, and the actual sex. My goal is to understand how changing the probability threshold affects the model’s predictions and performance.

Planned Approach

I will first load the dataset from its public GitHub URL and review its structure and class distribution. I will calculate the null error rate to establish a baseline for evaluating the model. I will then use probability thresholds of 0.2, 0.5, and 0.8 to generate predicted classes and calculate true positives, false positives, true negatives, and false negatives.

For each threshold, I will calculate accuracy, precision, recall, and F1 score. I will present the results using confusion matrices, a comparison table, and simple visualizations. Finally, I will explain the tradeoff between identifying more positive cases and avoiding false positive predictions.

Anticipated Data Challenges

I anticipate that the sex variable may need to be converted into numeric class labels before it can be compared with the predicted probabilities. I will also check for missing or invalid values before calculating the metrics.

Another challenge is that some metric formulas can produce undefined results when their denominator is zero. I will include checks for this possibility. I will carefully define “female” as the positive class and use the same definition throughout the analysis.

Expected Outcome

I expect a lower threshold to classify more observations as female, increasing recall but potentially creating more false positives. A higher threshold should classify fewer observations as female, which may improve precision but create more false negatives. The analysis will show why the best threshold depends on the real-world purpose of the model.

Data Loading and Validation

The original CSV was downloaded from the course GitHub repository and stored as a fixed file in my public GitHub repository. The code reads this copy through its GitHub Raw URL, allowing the analysis to run without using a local file path.

During data validation, I found that the assignment description presents the predicted classes as 1 and 0, but the actual CSV stores .pred_class and sex as the text labels female and male. For the analysis, I will convert female to 1 and male to 0, consistently treating female as the positive class.

data_url <- paste0(
  "https://raw.githubusercontent.com/",
  "howtwo388-cyber/DATA607-Week2B-Classification-Metrics/",
  "main/data/penguin_predictions.csv"
)

penguins_raw <- readr::read_csv(
  data_url,
  show_col_types = FALSE
)

dim(penguins_raw)
## [1] 93  3
names(penguins_raw)
## [1] ".pred_female" ".pred_class"  "sex"
dplyr::glimpse(penguins_raw)
## Rows: 93
## Columns: 3
## $ .pred_female <dbl> 0.99217462, 0.95423945, 0.98473504, 0.18702056, 0.9947012…
## $ .pred_class  <chr> "female", "female", "female", "male", "female", "female",…
## $ sex          <chr> "female", "female", "female", "female", "female", "female…

The following code checks the dataset and converts the character labels into binary values.

data_validation <- tibble::tibble(
  number_of_rows = nrow(penguins_raw),
  number_of_columns = ncol(penguins_raw),
  missing_values = sum(is.na(penguins_raw)),
  minimum_probability = min(
    penguins_raw$.pred_female,
    na.rm = TRUE
  ),
  maximum_probability = max(
    penguins_raw$.pred_female,
    na.rm = TRUE
  )
)

class_distribution <- penguins_raw |>
  dplyr::count(sex, name = "observations")

penguins_clean <- penguins_raw |>
  dplyr::mutate(
    actual_class = dplyr::if_else(
      sex == "female",
      1L,
      0L
    ),
    original_predicted_class = dplyr::if_else(
      .pred_class == "female",
      1L,
      0L
    ),
    predicted_class_from_probability = dplyr::if_else(
      .pred_female > 0.5,
      1L,
      0L
    )
  )

data_validation
## # A tibble: 1 × 5
##   number_of_rows number_of_columns missing_values minimum_probability
##            <int>             <int>          <int>               <dbl>
## 1             93                 3              0            5.60e-12
## # ℹ 1 more variable: maximum_probability <dbl>
class_distribution
## # A tibble: 2 × 2
##   sex    observations
##   <chr>         <int>
## 1 female           39
## 2 male             54
dplyr::glimpse(penguins_clean)
## Rows: 93
## Columns: 6
## $ .pred_female                     <dbl> 0.99217462, 0.95423945, 0.98473504, 0…
## $ .pred_class                      <chr> "female", "female", "female", "male",…
## $ sex                              <chr> "female", "female", "female", "female…
## $ actual_class                     <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
## $ original_predicted_class         <int> 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1…
## $ predicted_class_from_probability <int> 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1…

AI Use

OpenAI. (2026). ChatGPT (GPT-5) [Large language model]. https://chat.openai.com/. Accessed September 10, 2026.

ChatGPT was used to help interpret the assignment requirements, organize the planned approach, improve the English writing, and provide coding guidance. I will run the code, review the results, and make sure that I understand the analysis.