This dataset is from Professor Catlin’s public repository on GitHub called “data” located here
Tackling the problem: My plan is to review some of the concepts such as null error rates and harmonic mean that tripped me up in the reading Performance Metrics for Classification Problems in Machine Learning. Then, I will ingest the dataset “penguin_predictions” and go through the calculation exercises. I will build the matrices manually.
Anticipated data challenges: The biggest hurdle calculating these tasks in R rather than pen and paper. Second, is making sure the performance metrics are correctly done.
Citations:
Anthropic. (2025). Claude Opus 4.5 [Large language model]. (https://claude.ai/) Accessed January 8, 2026. Link to chat.
Analysis
#Not shown: Location on computer where I am grabbing the data and creating a saved location
#Install packages needed pacman::p_load(# Package Install and Management pacman, # package install/load janitor, #clean up data names # Project and File Management readr, # import data httr, #github passkey # General Data Management dplyr, # data management tidyr, # data management lubridate, # work with dates zoo, # work with dates tidyverse, # work with dates gmodels # freq tables SAS )#Load data#desktoppenguins <-read.csv(paste0(csv_location, "penguin_predictions.csv"), header=TRUE, stringsAsFactors =FALSE)#Review and prepare data for #looking at dataframe ls(penguins)
#change notation to easier format options(scipen =100) # take out sci notation clean$pred_female <-round(clean$pred_female, 2) #round #sex = actual#pred_female = model predicted probability that sex is female #pred_class = #1 = if the model probability is greater than .5#0 = if not #change pred_class in 1/0 binary clean <- clean%>%mutate( pred_class_50edit =case_when( pred_female > .5~1, pred_female <= .5~2), true_class_edit =case_when( sex =="female"~1, TRUE~2 ))
Task #1: Null Error Rate
A. Calculate the null error rate (majority-class error rate).
#what is majority class (higher count of actual data) majority <-table(clean$true_class_edit)addmargins(majority)
1 2 Sum
39 54 93
prop.table(majority)
1 2
0.4193548 0.5806452
#calc null error rate = 1 - (majority class proportion) = lower count of actual data # 1- (54/93)
[1] 0.4193548
B. Create a plot showing the distribution of the actual class (sex).
#sum up true counts (sex) and percents clean_graph<- clean%>%count(sex)%>%mutate( percent = n /sum(n))ggplot(clean_graph, aes(x = sex, y = percent)) +geom_col(fill ="darkblue") +scale_y_continuous(labels = scales::percent) +labs(title ="True Class Percent",x ="Class",y ="Percent" ) +theme_minimal()
C. Explain why knowing the null error rate is important when evaluating models.
It’s important to understand the proportion of the actual distribution of positive and negative values (in this case (sex of penguin) before evaluating how the model does. This of course will be helpful once we examine the confusion matrix and other performance metrics.
Task #2: Confusion Matrices at Multiple Thresholds
#adding in thresholds clean <- clean%>%mutate( #first threshold : .2 pred_class_20edit =case_when( pred_female > .2~1, pred_female <= .2~2), #Second threshold: .5 (already done)#Third threshold: .8 pred_class_80edit =case_when( pred_female > .8~1, pred_female <= .8~2 ))#confusion matrix #looking at it in SAS style library(gmodels)#In order for it to work with out messing up the formatting 1 = TRUE 2 = FALSE #20% threshold CrossTable(clean$pred_class_20edit, clean$true_class_edit, expected =FALSE, prop.r =FALSE, prop.c =FALSE, prop.t =FALSE)
##Precision## #meaning: Estimated outcome or Positive Predictive Value (PPV) #“Among those the model called positive, how many were actually positive?”#equation: Precision = TP / (TP + FP) <- aka row totaladdmargins(table(clean$pred_class_20edit, clean$true_class_edit))
##Recall## #meaning: Sensitivity or True Positive Rate #“Among those who were actually positive, how many did the model identify?”#equation: Recall = TP / (TP + FN) <- aka column total#20% threshold CrossTable(clean$pred_class_20edit, clean$true_class_edit, expected =FALSE, prop.r =FALSE, prop.c =TRUE, prop.t =FALSE)
