library(tidyverse)
library(janitor)
library(scales)
url <- "https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv"
df <- read_csv(
file = url,
show_col_types = FALSE,
progress = FALSE
)Assignment 2B - KCH - Codebase
Overview and Introduction
Data is provided in the form of a .csv file. A machine-learning classification model was previously used to generate predictions of the apparent sex of a penguin, ostensibly based on physical features such as height, fin length, color, etc. The feature set required to train the model is not included in the assignment’s files. The main task will be to evaluate the performance of this classification model using several key viewpoints: null error rate, confusion matrices, and performance metrics.
Anticipated challenges include the need to calculate the same metrics for various probability thresholds without a function. This may motivate learning to cast functions or a brute-force approach will be employed.
Data Import
First, the data is loaded from a github-located .csv using read_csv.
Next, the column names are converted to snake_case and the first 10 rows of the data frame are viewed.
clean_df <- janitor::clean_names(df)
head(clean_df,10)# A tibble: 10 × 3
pred_female pred_class sex
<dbl> <chr> <chr>
1 0.992 female female
2 0.954 female female
3 0.985 female female
4 0.187 male female
5 0.995 female female
6 1.000 female female
7 0.959 female female
8 1.000 female female
9 1.000 female female
10 0.339 male femaleAnalysis
Task 1: Null Error Rate
First, the total number of penguins, number of female penguins, and number of male penguins are determined.
## print total count
clean_df |>
summarize(count_total = n())# A tibble: 1 × 1
count_total
<int>
1 93## print counts by sex
clean_df |>
count(sex)# A tibble: 2 × 2
sex n
<chr> <int>
1 female 39
2 male 54## new df with counts, percents, and labels
penguin_class_dist <- clean_df |>
filter(!is.na(sex)) |>
count(sex) |>
mutate(
pct = n/sum(n),
#concatenate percentage, newline, and (n= count)
label = paste0(percent(pct, accuracy=0.1), "\n(n = ", n, ")")
)
head(penguin_class_dist)# A tibble: 2 × 4
sex n pct label
<chr> <int> <dbl> <chr>
1 female 39 0.419 "41.9%\n(n = 39)"
2 male 54 0.581 "58.1%\n(n = 54)"The distribution is then visualized with a simple bar chart.
ggplot(penguin_class_dist, aes(x = sex, y = n, fill = sex)) +
geom_col(width = 0.5, show.legend = FALSE) +
# add labels of classification bins
geom_text(aes(label = label), vjust = -0.3, size = 3.8, lineheight = 0.9) +
#scale y to create room for labels
scale_y_continuous(expand = expansion(mult = c(0.05, 0.2))) +
# adding color distinction to bars
scale_fill_manual(values = c("female" = "#2c4cc6", "male" = "#d8581c")) +
# adding descriptive information to plot
labs(
title = "Distribution of Target Variable: Sex of Penguins (female vs. male)",
subtitle = "Relative frequencies (%) and absolute sample sizes (n)",
x = "Sex",
y = "Count of Observations"
) +
#adjusting theme settings
theme_minimal(base_size = 11) +
#removing unnecessary formatting and reformatting title/subtitle
theme(
panel.grid.major.x = element_blank(),
panel.grid.minor = element_blank(),
plot.title = element_text(face = "bold"),
plot.subtitle = element_text(color = "gray30", margin = margin(b = 15))
)
From this distribution, and the new data frame, the null error rate and null accuracy can be computed.
null_metrics <-penguin_class_dist |>
summarize( majority = sex[which.max(pct)],
accuracy = max(pct),
error_rate = 1 - accuracy)
print(null_metrics)# A tibble: 1 × 3
majority accuracy error_rate
<chr> <dbl> <dbl>
1 male 0.581 0.419Task 2: Confusion Matrices
In order to help visualize the performance of the classification model, confusion matrix calculations at three different probability thresholds (0.2, 0.5, and 0.8) will be performed. First, the true-positives, true-negatives, false-positives, and false-negatives at each threshold level are counted.
penguin_preds <- clean_df |>
mutate(pred_20 = if_else(pred_female>=0.2,"female","male"),
pred_50 = if_else(pred_female>=0.5,"female","male"),
pred_80 = if_else(pred_female>=0.8,"female","male"))
#head(penguin_preds,10)
penguin_stats <- penguin_preds |>
summarize(
TP_20 = sum(sex =="female" & pred_20 == "female"),
TN_20 = sum(sex =="male" & pred_20 == "male"),
FP_20 = sum(sex =="male" & pred_20 == "female"),
FN_20 = sum(sex == "female" & pred_20 == "male"),
TP_50 = sum(sex =="female" & pred_50 == "female"),
TN_50 = sum(sex =="male" & pred_50 == "male"),
FP_50 = sum(sex =="male" & pred_50 == "female"),
FN_50 = sum(sex == "female" & pred_50 == "male"),
TP_80 = sum(sex =="female" & pred_80 == "female"),
TN_80 = sum(sex =="male" & pred_80 == "male"),
FP_80 = sum(sex =="male" & pred_80 == "female"),
FN_80 = sum(sex == "female" & pred_80 == "male"))
head(penguin_stats,1)# A tibble: 1 × 12
TP_20 TN_20 FP_20 FN_20 TP_50 TN_50 FP_50 FN_50 TP_80 TN_80 FP_80 FN_80
<int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 37 48 6 2 36 51 3 3 36 52 2 3Confusion Matrix: Probability Threshold 0.20
Next, confusion matrices for each threshold level are computed. First, at a threshold of 0.2:
cm_20 <-matrix(c(penguin_stats$TP_20,
penguin_stats$FN_20,
penguin_stats$FP_20,
penguin_stats$TN_20
),
nrow=2, byrow=TRUE,
dimnames=list(actual = c("female", "male"),
predicted = c("female", "male")
)
)
cm_20 predicted
actual female male
female 37 2
male 6 48When the probability threshold is set at 0.20, there are 37 true-positives and 48 true-negatives, with only 6 false-positives and 2 false-negatives. Next, the confusion matrix at a threshold of 0.50 is computed.
Confusion Matrix: Probability Threshold 0.50
cm_50 <-matrix(c(penguin_stats$TP_50,
penguin_stats$FN_50,
penguin_stats$FP_50,
penguin_stats$TN_50
),
nrow=2, byrow=TRUE,
dimnames=list(actual = c("female", "male"),
predicted = c("female", "male")
)
)
cm_50 predicted
actual female male
female 36 3
male 3 51When the probability threshold is set at 0.50, there are fewer true-positives (36) and more true-negatives (51), with fewer false-positives (3) and one more false-negative (3). Next, the confusion matrix at a threshold of 0.80 is computed.
Confusion Matrix: Probability Threshold 0.80
cm_80 <- matrix(c(penguin_stats$TP_80,
penguin_stats$FN_80,
penguin_stats$FP_80,
penguin_stats$TN_80
),
nrow=2, byrow=TRUE,
dimnames=list(actual = c("female", "male"),
predicted = c("female", "male")
)
)
cm_80 predicted
actual female male
female 36 3
male 2 52When the probability threshold is set at 0.80, there are the same number of true-positives (36) as the 0.50 threshold, more true-negatives (52), and fewer false-positives (2) and the same false-negatives (3).
As the probability threshold increases, the number of true-positives generally decreases, while true-negatives generally increases. Meanwhile, the number of false-positives generally decreases, while false-negatives generally increases.
Task 3: Performance Metrics
To compute performance metrics for each probability threshold, the true-positive (TP), false-positive (FP), true-negative (TN), and false-negative (FN) values for each threshold are fed into specific formulas. The accuracy measures the rate of true predictions to total predictions, precision measures the ratio of true-positive predictions to total positive predictions, recall measures the ratio of true-positive predictions to true-positive and false-negative predictions, and the F1 score calculates the harmonic mean of precision and recall. In summary, these statistics provide insight into the performance of the classification model at each probability threshold.
Performance Metrics: Probability Threshold 0.20
perf_20 <- penguin_stats |>
summarize(
acc_20 = (TP_20+TN_20)/(TP_20+TN_20+FP_20+FN_20),
prec_20 = (TP_20)/(TP_20+FP_20),
rec_20 = (TP_20)/(TP_20+FN_20),
f1_20 = (2*rec_20*prec_20)/(rec_20+prec_20)
)
perf_20# A tibble: 1 × 4
acc_20 prec_20 rec_20 f1_20
<dbl> <dbl> <dbl> <dbl>
1 0.914 0.860 0.949 0.902At a probability threshold of 0.20, the accuracy and recall are 91.4% and above, while the precision is 86.0% and the F1 score is 90.2%.
Performance Metrics: Probability Threshold 0.50
perf_50 <- penguin_stats |>
summarize(
acc_50 = (TP_50+TN_50)/(TP_50+TN_50+FP_50+FN_50),
prec_50 = (TP_50)/(TP_50+FP_50),
rec_50 = (TP_50)/(TP_50+FN_50),
f1_50 = (2*rec_50*prec_50)/(rec_50+prec_50)
)
perf_50# A tibble: 1 × 4
acc_50 prec_50 rec_50 f1_50
<dbl> <dbl> <dbl> <dbl>
1 0.935 0.923 0.923 0.923Increasing the probability threshold to 0.50 results in an increase in the accuracy (93.5%), yet a decrease in the recall (92.3%, down from 94.9%). The precision increased from 86.0% to 92.3% and the F1 score also increased from 90.2% to 92.3%. All statistics captured at the 0.50 probability threshold exceed 92%.
Performance Metrics: Probability Threshold 0.80
perf_80 <- penguin_stats |>
summarize(
acc_80 = (TP_80+TN_80)/(TP_80+TN_80+FP_80+FN_80),
prec_80 = (TP_80)/(TP_80+FP_80),
rec_80 = (TP_80)/(TP_80+FN_80),
f1_80 = (2*rec_80*prec_80)/(rec_80+prec_80)
)
perf_80# A tibble: 1 × 4
acc_80 prec_80 rec_80 f1_80
<dbl> <dbl> <dbl> <dbl>
1 0.946 0.947 0.923 0.935Finally, increasing the probability threshold to 0.80 results in an increase in the accuracy to 94.6%. Recall remains unchanged at 92.3%, while precision increased from 92.3% to 94.7% and the F1 score correspondingly increased to 93.5%.
Task 4: Threshold Use Cases
In real-world applications of machine-learning classification models, various probability threshold values may be preferred due to contextual constraints. For instance, there are generally two goals that may be considered when making this decision: 1. Minimize the false-positive rate (maximize precision); and 2. Minimize the false-negative rate (maximize recall).
If the risk associated with a false-positive is high, then precision should be maximized. This can be necessary in several real-world contexts, such as spam filtering. In this context, the risk of falsely selecting an inbound message as spam can carry costs such as missing important emails related to work. Since accidentally allowing a spam message into the inbox does not carry such high costs, a high probability threshold may be preferred in this context.
If the risk associated with a false-negative is high, then recall should be maximized. In the context of medical diagnosis, a false-negative can have devastating consequences on a patient with an illness who goes undetected. For instance, if a machine-learning classification model predicts tumor probabilities based on imaging, then there is a high patient-risk associated with false-negatives, and the recall should be maximized (i.e. low probability threshold).
Findings and Recommendations
Outputs from a penguin classification model were analyzed to determine key statistical insights into the performance of the model. Namely, confusion matrices, recall, precision, accuracy, and F1-scores were calculated at various probability thresholds (0.20, 0.50, and 0.80). When the probability threshold was low (0.20), there were a greater number of true-positive counts (high recall), however the overall performance metrics demonstrated low accuracy and precision compared to higher probability threshold values. At a middle value (0.50), almost all performance metrics improved, while recall decreased. Further increasing the threshold to a high value (0.80) continues to increase the precision and accuracy, while the recall remained the same.
One improvement to this work could be expanding the data set that the classification model uses for testing. With more data in the test set for this classification model, one may observe a more pronounced impact of adjusting the probability-threshold. Another improvement could be to write R functions for calculating these statistics and confusion matrices. This would allow for easy changes to the code to test a different probability threshold and compute the resultant performance metrics.