library(tidyverse)
library(readr)Week 2B Assignment - Classification Metrics
Evaluating Classification Model Performance
Approach
Goal and Background Information:
The goal of this assignment is to analyze the performance of a binary classification model. Following that building and developing intuition on how probability affects model evaluation predictions.
However before diving into the dataset and evaluating the model, an understanding of what a Classification Problem is necessary to approach this problem and future datasets where classification is required.
Terminology:
To note what is mentioned in simple definitions are:
- Confusion Matrix: a 2x2 diagram of the model predicting against reality
| Predicted | Predicted | ||
|---|---|---|---|
| Positive | Negative | ||
| Actual | Positive | True Positive | False Negative |
| Actual | Negative | False Positive | True Negative |
Image with Description
Accuracy: Overall percentage of all correct predictions using the formula:
\[\begin{equation*} \text{Accuracy} = \frac{\text{True Positive} + \text{True Negative}}{\text{Total Predictions}} \end{equation*}\]
Precision: Of all the predicted positives, how many of those were positve?
\[\begin{equation*} \text{Precision} = \frac{\text{True Positive}}{\text{True Positive} + \text{False Positive}} \end{equation*}\]
Recall (or Sensitivity): Of all actual positives, how many did the model correctly identified?
\[\begin{equation*} \text{Recall} = \frac{\text{True Positive}}{\text{True Positive} + \text{False Negative}} \end{equation*}\]
Specificity: Of all actual negatives, how many did the model correctly identified?
\[\begin{equation*} \text{Specificity} = \frac{\text{True Negative}}{\text{True Negative} + \text{False Positive}} \end{equation*}\]
F1 Score: Calculates the harmonic mean uses both precision and recall in one variable and measures how balanced it is compared to an arithmetic mean calculation.
\[\begin{equation*} \text{F1 Score} = 2 \times \frac{\text{Precision $\times$ Recall}}{\text{Precision $+$ Recall}} \end{equation*}\]
Supplemental Videos:
After reading this, now we start importing the dataset, penguin_predictions.csv from Github.
The Plan:
The plan for this dataset is to practice fundamentals on machine learning, particularly the concept of a binary classification model. As I am doing this, I will dive into topics that will increase my understanding of modeling such as:
Null Error Rate
Confusion Matrices
Performance Metrics
Thresholds
But first I will need to explore the data, by visualizing a plot to compare the counts of females and males as a start, followed by steps to help me understand and approach this problem.
Codebase
Reading in the data and inspecting
raw_data <- read_csv("https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv")
glimpse(raw_data)Rows: 93
Columns: 3
$ .pred_female <dbl> 0.99217462, 0.95423945, 0.98473504, 0.18702056, 0.9947012…
$ .pred_class <chr> "female", "female", "female", "male", "female", "female",…
$ sex <chr> "female", "female", "female", "female", "female", "female…head(raw_data)# A tibble: 6 × 3
.pred_female .pred_class sex
<dbl> <chr> <chr>
1 0.992 female female
2 0.954 female female
3 0.985 female female
4 0.187 male female
5 0.995 female female
6 1.000 female femalesexis the class we want to use during model training
Null Error Rate
To calculate Null Error Rate, the formula is:
\[\begin{equation*} \text{Null Error Rate} = 1 - \frac{\text{Majority Count}}{\text{Total Number}} \end{equation*}\]
In this case, what we are going to calculate is the sex column, there are:
93 total counts, which is going to be our denominator
Our numerator, we need to do a count check and plot it to ensure it is correct
Bar Plot of Counts
ggplot(raw_data, aes(x=sex, fill=sex)) +
geom_bar(width = 0.4) +
geom_text(
stat = "count",
aes(label = after_stat(count)),
vjust = -0.1
) +
labs(
title = "Counts of Male vs Female",
x = "Sex",
y = "Counts"
) +
theme_gray()
After plotting, the counts are:
Males: 54
Females: 39
# Sanity Check
raw_data |> count(sex)# A tibble: 2 × 2
sex n
<chr> <int>
1 female 39
2 male 54And our table has confirmed that the plot is true so now we can calculate the Null Error Rate
Null Error Rate Calculation
\[\begin{equation*} \text{Null Error Rate} = 1 - \frac{\text{54}}{\text{93}} \\ = 1 - 0.58064516129 \\ \boxed{= 0.41935483871} \end{equation*}\]
What this result tells us are the following:
With an accuracy of 58.06%, the model guessing male for every penguin will be right 54 times
With a Null Error Rate of 41.93%, the model will guess males and get all 39 females incorrectly
This step is important because the Null Error Rate is the first step in model creation. What this tells us is that if the model were to keep guessing the most common answer, in this case “male”, it would be correct 58.06% of the time whilst being wrong 41.93% of the time because there are females that it would miss.
As this is the start of model creation, it is imperative we improve upon the score of 41.93% if further work was to be made to improve on.
Confusion Matrices
Before we make our confusion matrices, it is important to note there will be 3 thresholds the model will be trained on:
0.2
0.5
0.8
Making threshold columns
penguins_df <- raw_data %>%
mutate(
pred02 = ifelse(.pred_female > 0.2, 1, 0),
pred05 = ifelse(.pred_female > 0.5, 1, 0),
pred08 = ifelse(.pred_female > 0.8, 1, 0),
sex = ifelse(sex == "female", 1, 0)
)
head(penguins_df)# A tibble: 6 × 6
.pred_female .pred_class sex pred02 pred05 pred08
<dbl> <chr> <dbl> <dbl> <dbl> <dbl>
1 0.992 female 1 1 1 1
2 0.954 female 1 1 1 1
3 0.985 female 1 1 1 1
4 0.187 male 1 0 0 0
5 0.995 female 1 1 1 1
6 1.000 female 1 1 1 1Creating Confusion Matrices
To understand what makes something “True” or “False and”Positive” or “Negative” here is a pattern to follow below:
True/False: Did the model get the prediction right (True) or wrong (False)
Positive/Negative: Did the model say “yes”? (Positive) or “no” (Negative)
Example:
Model says gender is female (Positive), the actual gender is a female (True); True Positive.
It is almost like reading it backwards to get the label of True Positive
Implementing Confusion Matrices
cm_02 <- penguins_df %>%
summarise(
TP = sum(pred02 == 1 & sex == 1),
FP = sum(pred02 == 1 & sex == 0),
TN = sum(pred02 == 0 & sex == 0),
FN = sum(pred02 == 0 & sex == 1)
)
cm_05 <- penguins_df %>%
summarise(
TP = sum(pred05 == 1 & sex == 1),
FP = sum(pred05 == 1 & sex == 0),
TN = sum(pred05 == 0 & sex == 0),
FN = sum(pred05 == 0 & sex == 1)
)
cm_08 <- penguins_df %>%
summarise(
TP = sum(pred08 == 1 & sex == 1),
FP = sum(pred08 == 1 & sex == 0),
TN = sum(pred08 == 0 & sex == 0),
FN = sum(pred08 == 0 & sex == 1)
)cm_02 <- cm_02 %>% mutate(threshold = 0.2, .before = 1)
cm_05 <- cm_05 %>% mutate(threshold = 0.5, .before = 1)
cm_08 <- cm_08 %>% mutate(threshold = 0.8, .before = 1)
#.before tells us to place in position 1
cm_02# A tibble: 1 × 5
threshold TP FP TN FN
<dbl> <int> <int> <int> <int>
1 0.2 37 6 48 2cm_05# A tibble: 1 × 5
threshold TP FP TN FN
<dbl> <int> <int> <int> <int>
1 0.5 36 3 51 3cm_08# A tibble: 1 × 5
threshold TP FP TN FN
<dbl> <int> <int> <int> <int>
1 0.8 36 2 52 3Performance Metrics
Calculating Accuracy
cm_02 <- cm_02 %>%
mutate(accuracy = (TP + TN) / (TP + FP + TN + FN))
cm_05 <- cm_05 %>%
mutate(accuracy = (TP + TN) / (TP + FP + TN + FN))
cm_08 <- cm_08 %>%
mutate(accuracy = (TP + TN) / (TP + FP + TN + FN))Calculating Precision
cm_02 <- cm_02 %>%
mutate(precision = (TP) / (TP + FP))
cm_05 <- cm_05 %>%
mutate(precision = (TP) / (TP + FP))
cm_08 <- cm_08 %>%
mutate(precision = (TP) / (TP + FP))Calculating Recall
cm_02 <- cm_02 %>%
mutate(recall = (TP) / (TP + FN))
cm_05 <- cm_05 %>%
mutate(recall = (TP) / (TP + FN))
cm_08 <- cm_08 %>%
mutate(recall = (TP) / (TP + FN))Calculating F1 Score
cm_02 <- cm_02 %>%
mutate(f1_score = 2 * ((precision * recall) / (precision + recall)))
cm_05 <- cm_05 %>%
mutate(f1_score = 2 * ((precision * recall) / (precision + recall)))
cm_08 <- cm_08 %>%
mutate(f1_score = 2 * ((precision * recall) / (precision + recall)))Showing the final confusion matrices
cm_02# A tibble: 1 × 9
threshold TP FP TN FN accuracy precision recall f1_score
<dbl> <int> <int> <int> <int> <dbl> <dbl> <dbl> <dbl>
1 0.2 37 6 48 2 0.914 0.860 0.949 0.902cm_05# A tibble: 1 × 9
threshold TP FP TN FN accuracy precision recall f1_score
<dbl> <int> <int> <int> <int> <dbl> <dbl> <dbl> <dbl>
1 0.5 36 3 51 3 0.935 0.923 0.923 0.923cm_08# A tibble: 1 × 9
threshold TP FP TN FN accuracy precision recall f1_score
<dbl> <int> <int> <int> <int> <dbl> <dbl> <dbl> <dbl>
1 0.8 36 2 52 3 0.946 0.947 0.923 0.935Comparing all these confusion matrices, as the threshold increases, all the performance metric increases either slightly (in their respective direction).
Threshold Use Cases
What is a Threshold?
To understand how 0.2 or a 0.8 threshold case can be used with a real life example, we need to understand what exactly is a “0.2 threshold” or a “0.8 threshold”. To note: the default threshold is 0.5.
0.2 Threshold: A 20% chance that something is present. So in this assignment’s case, 20% that someone is a female, say they are a female.
0.8: Threshold: A 80% chance that the model is certain of something, so the expectation is much higher to pass off if someone is a female as an example.
Real World Example
One real world scenario where a 0.2 threshold would be preferable would be in a medical field. If someone were to be scanned for a disease, it is better to catch it before it grows into something bigger.
A real world example where a 0.8 threshold would be preferable would be to check if an email is spam. Since the threshold is high, it means that it does not want a false positive result back.
Conclusion
This assignment was my first time tackling machine learning concepts hands on rather than theory. I had no idea how to properly tackle this assignment without AI assistance for terminology as well as coding approach. But nonetheless I had a lot of fun working with machine learning and wish to do it more.
In the future I would love to work with a dataset where I get to compare model performance metrics and see why certain models wouldn’t work in certain situations.