Approach Deliverable:

Introduction

To begin this assignment, I am deciding to spend more time doing research as I am new to the machine learning concepts discussed in the lab. Once I have a basis understanding I will then begin the lab itself. Being that I am new to these concepts I am following of the Additional Instructions and conceptual roadmap. Areas where I see potential problems would be understanding and developing insights based off the various parts of the lab. While things like Null Error Rate, Confusion Matrices and Performance for the most part are formula based calculations. Each of these just require finding the right numbers to plug into the formula to get the necessary results. However since I am new to a lot of these concepts it may be more difficult to really understand or draw insights from these calculations. This is why I am prioritizing research and understanding, using Chat GPT to help explain some of these terms and why they are useful in simpler terms. As I begin the lab I will update my Approach or conclusion if I encounter any more issues than stated above. As for approaching the data frame, I am currently using glimpse to take a look at the different columns and what these all mean as a feature. Then as stated in the Step By Step Guidance section, I decided to plot distribution of the target variable to better understand the data set as a whole.

library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
library(tidyverse)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ forcats   1.0.1     ✔ readr     2.2.0
## ✔ ggplot2   4.0.3     ✔ stringr   1.6.0
## ✔ lubridate 1.9.5     ✔ tibble    3.3.1
## ✔ purrr     1.2.1     ✔ tidyr     1.3.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
penguins_df <- read.csv("https://raw.githubusercontent.com/acatlin/data/refs/heads/master/penguin_predictions.csv")
count_of_sex <- table(penguins_df$sex)
count_of_sex
## 
## female   male 
##     39     54

This count shows that the most common class would be male.

null_error_rate <- 1 - (max(count_of_sex) / sum(count_of_sex))
null_error_rate
## [1] 0.4193548

The null error rate is about 42%.

ggplot(penguins_df, aes(x=sex)) + geom_bar();

penguins_df$pred_02 <- ifelse(penguins_df$.pred_female > 0.2, 1, 0)

penguins_df$pred_05 <- ifelse(penguins_df$.pred_female > 0.5, 1, 0)

penguins_df$pred_08 <- ifelse(penguins_df$.pred_female > 0.8, 1, 0)

table02 <- table(Predicted = penguins_df$pred_02, Actual = penguins_df$sex)
table02
##          Actual
## Predicted female male
##         0      2   48
##         1     37    6
table05 <- table(Predicted = penguins_df$pred_05, Actual = penguins_df$sex)
table05
##          Actual
## Predicted female male
##         0      3   51
##         1     36    3
table08 <- table(Predicted = penguins_df$pred_08, Actual = penguins_df$sex)
table08
##          Actual
## Predicted female male
##         0      3   52
##         1     36    2

Threshold 0.2

FN: 2, TN: 48, TP: 37, FP 6

Threshold 0.5
FN: 3, TN: 51, TP: 36, FP 3

Threshold 0.8
FN: 3, TN 52, TP 36, FP 2

#Accuracy = TP + TN / Total Predictions
#sum(diag(conf_matrix)) / sum(conf_matrix)

#accuracy_02 <- mean(penguins_df$pred_02 == penguins_df$sex)
accuracy_02 <- sum(diag(table02)) / sum(table02)
accuracy_02
## [1] 0.08602151
accuracy_05 <- sum(diag(table05)) / sum(table05)
accuracy_05
## [1] 0.06451613
accuracy_08 <- sum(diag(table08)) / sum(table08)
accuracy_08
## [1] 0.05376344
precision02 <- 37 / (37 + 6)
precision02
## [1] 0.8604651
precision05 <- 36 / (36 + 3)
precision05
## [1] 0.9230769
precision08 <- 36 / (36 + 3)
precision08
## [1] 0.9230769
#recall = TP / (TP + FN)

recall02 <- 37 / (37 + 2)
recall02
## [1] 0.9487179
recall05 <- 36 / (36 + 3)
recall05
## [1] 0.9230769
recall08 <- 36 / (36 + 3)
recall08
## [1] 0.9230769
#f1 = 2 x (precision x recall) / (precision + recall)
f1_02 <- 2 * (precision02 * recall02)/(precision02 + recall02)
f1_02
## [1] 0.902439
f1_05 <- 2 * (precision05 * recall05)/(precision05 + recall05)
f1_05
## [1] 0.9230769
f1_08 <- 2 * (precision08 * recall08)/(precision08 + recall08)
f1_08
## [1] 0.9230769

Real World Scenarios for Threshold Use Cases:

If you were wanted to accurately predict cancer in patients than you would want a lower threshold because you would capture more cancer patients as a false negative would be more dangerous than a false positive who receives further testing anyways.

A higher threshold would be preferable for fingerprint scanners and security systems. The higher threshold would prevent random people accidentally gaining access, even though a false negative may inconvenience a user.

Conclusion

To conclude this lab has helped me to better understand Classification metrics. I know understand the uses of Null Error Rate, Confusion Matrices, Accuracy, Precision, Recall and F1 Score. And after researching these concepts working through the lab helped solidify my understanding by putting them into practice. While most of these concepts ended up being relatively straightforward overall, I found understanding Confusion Matrices the most challenging. The concept made sense, but in application I found myself having to refer to the context of the problem multiple times to understand the what the difference in threshold would accomplish. I found that lower thresholds would increase the amount of True Positive results, while higher thresholds would increase the amount of true negative results. high low thresholds increase false the number of false positives and high thresholds increase the number of false negatives respectively. So which is preferable is entire dependent on the context of the problem one is trying to solve. I also found that the Null Error Rate is important to determine the how effective the model predictive probability is. Ultimately the lab helped me to better understand the applications of these concepts moving forward.

Citations:

OpenAI. (2026, September 14). ChatGPT conversation about evaluating classification model performance [Large language model]. ChatGPT.

Link to chat: https://chatgpt.com/s/t_6aa7615bdff88191a2ee1bec7faa6deb