data <- read.csv("https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv")Evaluating Classification Model Performance
DATA 607: Week 2 Assignment: Approach
Approach
The assignment is to analyze a binary classification model on gender (which is very cisheteronormative, but ok)! The model predicts that an observation is female, assigned to the value 1, if the probability is greater than 0.5. If less than or equal to 0.5, the observation is predicted as male, assigned to value 0. To test the performance of the model, these predicted classifications will be compared against the actual genders.
Roadmap
Since the assignment is basically a bunch of calculations with a few charts, instead of a narrative structure, I will explain my approach step-by-step in response to each assignment task. Broadly, I plan to do every calculation and chart in R.
First I will import the csv file:
Task 1
Opporture defines the null error rate as “the percentage of incorrect predictions you would make if you always selected the most common outcome.” Sunasra doesn’t define the null error rate, but he gives a great example: If a model is designed to detect a rare cancer, e.g. 5 out of 100 prevalence, then even a bad model that always fails to detect cancer will still be correct 95% of the time. As Opporture notes, the importance of the null error rate is that it can indicate a biased model that favors the majority class. The null error rate gives us a baseline, which we want to improve upon by building models that minimize the false negatives and reduces bias.
Since there are 39 females and 54 males (= 93-39) in the dataset, males are the majority class. I will calculate the null error rate by dividing the number of females by 93 (the total number of observations).
To create a plot showing the distribution of the actual class (sex), I will use ggplot2 to create a bar chart, which is ideal for binary data.
Task 2
To change the probability thresholds and calculate confusion matrices, I need to do some manipulations in R. Using the mutate function and the provided code in the assignment (ifelse(.pred_female > threshold, 1, 0) where threshold equals 0.2, 0.5, or 0.8, I will add three new columns to the dataset corresponding to the predicted class output for each of those three thresholds.
As per Sunasra, True Positives are the number of observations for which the model correctly predicts “female.” (Note that it is completely arbitrary whether male or female is designated as the “positive” case.) Since “female” was assigned “1” for this assignment, females are by definition the “positive” case. I will therefore calculate True Positives (TP) by counting in R, using booleans, the number of rows that satisfy the conditions .pred_class equals “female” & sex (actual class) equals “female.”
False Positives (FP) are observations for which the model predicted “female” but the actual class is “male.” I will calculate the number of rows that satisfy the conditions .pred_class equals “female” & sex (actual class) equals “male.”
True Negatives (TN) are observations for which the model predicted “male” and the actual class is “male.” I will calculate the number of rows that satisfy the conditions .pred_class equals “male” & sex (actual class) equals “male.”
False Negatives (FN) are observations for which the model predicted “male” and the actual class is “female.” I will calculate the number of rows that satisfy the conditions .pred_class equals “male” & sex (actual class) equals “female.”
To plot the confusion matrices in R, I will use the cvms package. (I found this package by asking Google AI for recommendations on how to plot a confusion matrix in R.)
Task 3
As per Sunasra, to calculate Accuracy, I divide (TP + TN) by (TP + FP + FN + TN).
To calculate Precision, I divide TP by (TP + FP).
To calculate Recall, I divide TP by (TP + FN).
To calculate F1 Score, I divide (2 * Precision * Recall) by (Precision + Recall).
I will do all of the above calculations for each of the three thresholds (02, 0.5, and 0.8). All calculations will be done in R and presented in tables.
Task 4
I will answer these questions during the Code Base part of the Assignment.
Code Base
Forthcoming.
References
Opporture. (no date). Null Error Rate. https://www.opporture.org/lexicon/null-error-rate/.
Sunasra M. (2017). Performance Metrics for Classification Problems in Machine Learning. https://github.com/acatlin/data/blob/master/Performance%20Metrics%20for%20Classification%20problems%20in%20Machine%20Learning.pdf.
AI Use
Google Gemini was used as an AI-assisted development tool for a part of this Assignment. Its suggested code was reviewed, tested, and revised by the author for help understanding how to implement a confusion matrix using the cvms package in R. The author remains responsible for all work submitted for this Assignment.
Tool/model: Gemini 3.5 Flash Developer: Google
Date accessed: September 10, 2026