Approach Week 2B Classification Metrics

Author

Mubin Ejaz

Assignment 2B, Classification Metrics

Loading the penguin_predictions.csv into PenguinPrediction.

library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(dplyr)
PenguinPrediction = read.csv('penguin_predictions.csv')
view(PenguinPrediction)

  1. NULL Error Rate = (total number of observation - Samples in majority class) / Total Number of samples. In other words, minor class/total number of observation.

     TotalObservations <- count(PenguinPrediction)
 df<- PenguinPrediction |> filter(sex=='female') |> count()
 dm <-PenguinPrediction |> filter(sex=='male') |> count()
 minimum = min(df, dm)

 NullErrorRate = (minimum/TotalObservations) * 100

The Null Error Rate is :

NullErrorRate$n
[1] 41.93548

Class distribution:
 
I’ll be plotting actual number of female and actually number of male vs total number of students in the class. 

femalesInClass<- PenguinPrediction |> filter(sex=='female')
malesInClass <- PenguinPrediction |> filter(sex=='male')

ggplot(PenguinPrediction, aes(x=sex, fill=sex))+geom_bar()

Why knowing Null Error Rate is imp. When predicting the model:
Basically, This is how often you would be wrong if you always predicted the majority class.  So, in dire cases, let’s say there’s 100 cancer patients and 5 has cancer.  If my model always predicted no cancer, I would miss the actual patients with cancer but my model still be 95% accurate.

2.      Confusion Matrices at Multiple Thresholds

 I will use R to do the following on the penguin data that I’d load in a dataframe:

Probability Threshold 0.2 (if pred_female is 0.2 or above, it’s female meaning 1 else it’s male which is 0)

3.**Performance Metrics:** In my homework submission of codebase deliverable, I’ll use R to load, calculate, and plot the following three metrics **** Probability threshold 0.2:

….
Threshold = 0.2

PenguinPrediction_0point2 <- PenguinPrediction |> 
  mutate(.pred_female = round(.pred_female, 4))

PenguinPrediction_0point2 <- PenguinPrediction_0point2 |> 
  mutate(TP = if_else(.pred_female >= 0.2 & sex == "female", 1, 0))


PenguinPrediction_0point2 <- PenguinPrediction_0point2 |> 
  mutate(FN = if_else(.pred_female < 0.2 & sex == "female", 1, 0))

PenguinPrediction_0point2 <- PenguinPrediction_0point2 |> 
  mutate(TN = if_else(.pred_female < 0.2 & sex == "male", 1, 0))


PenguinPrediction_0point2 <- PenguinPrediction_0point2 |> 
  mutate(FP = if_else(.pred_female >= 0.2 & sex == "male", 1, 0))

PenguinPrediction_0point2
   .pred_female .pred_class    sex TP FN TN FP
1        0.9922      female female  1  0  0  0
2        0.9542      female female  1  0  0  0
3        0.9847      female female  1  0  0  0
4        0.1870        male female  0  1  0  0
5        0.9947      female female  1  0  0  0
6        1.0000      female female  1  0  0  0
7        0.9594      female female  1  0  0  0
8        0.9999      female female  1  0  0  0
9        1.0000      female female  1  0  0  0
10       0.3394        male female  1  0  0  0
11       0.9130      female female  1  0  0  0
12       0.9987      female female  1  0  0  0
13       0.9998      female female  1  0  0  0
14       0.9875      female female  1  0  0  0
15       0.9914      female female  1  0  0  0
16       0.9824      female female  1  0  0  0
17       0.9945      female female  1  0  0  0
18       1.0000      female female  1  0  0  0
19       1.0000      female female  1  0  0  0
20       0.9930      female female  1  0  0  0
21       0.9694      female female  1  0  0  0
22       0.9746      female female  1  0  0  0
23       0.0820        male female  0  1  0  0
24       1.0000      female female  1  0  0  0
25       0.9971      female female  1  0  0  0
26       1.0000      female female  1  0  0  0
27       0.9817      female female  1  0  0  0
28       0.9997      female female  1  0  0  0
29       0.9961      female female  1  0  0  0
30       0.9995      female female  1  0  0  0
31       0.9987      female female  1  0  0  0
32       0.9994      female female  1  0  0  0
33       0.9999      female female  1  0  0  0
34       1.0000      female female  1  0  0  0
35       0.9970      female female  1  0  0  0
36       0.9865      female female  1  0  0  0
37       1.0000      female female  1  0  0  0
38       0.9856      female female  1  0  0  0
39       0.9999      female female  1  0  0  0
40       0.1963        male   male  0  0  1  0
41       0.0000        male   male  0  0  1  0
42       0.0000        male   male  0  0  1  0
43       0.0000        male   male  0  0  1  0
44       0.0000        male   male  0  0  1  0
45       0.7763      female   male  0  0  0  1
46       0.1981        male   male  0  0  1  0
47       0.9007      female   male  0  0  0  1
48       0.3412        male   male  0  0  0  1
49       0.0000        male   male  0  0  1  0
50       0.0096        male   male  0  0  1  0
51       0.0322        male   male  0  0  1  0
52       0.0126        male   male  0  0  1  0
53       0.0942        male   male  0  0  1  0
54       0.0000        male   male  0  0  1  0
55       0.0038        male   male  0  0  1  0
56       0.0108        male   male  0  0  1  0
57       0.0004        male   male  0  0  1  0
58       0.0348        male   male  0  0  1  0
59       0.0058        male   male  0  0  1  0
60       0.0016        male   male  0  0  1  0
61       0.0001        male   male  0  0  1  0
62       0.0002        male   male  0  0  1  0
63       0.0005        male   male  0  0  1  0
64       0.0032        male   male  0  0  1  0
65       0.0000        male   male  0  0  1  0
66       0.1099        male   male  0  0  1  0
67       0.0001        male   male  0  0  1  0
68       0.0008        male   male  0  0  1  0
69       0.2207        male   male  0  0  0  1
70       0.0021        male   male  0  0  1  0
71       0.0353        male   male  0  0  1  0
72       0.0000        male   male  0  0  1  0
73       0.0003        male   male  0  0  1  0
74       0.0000        male   male  0  0  1  0
75       0.0000        male   male  0  0  1  0
76       0.2960        male   male  0  0  0  1
77       0.0359        male   male  0  0  1  0
78       0.0000        male   male  0  0  1  0
79       0.0000        male   male  0  0  1  0
80       0.0001        male   male  0  0  1  0
81       0.0012        male   male  0  0  1  0
82       0.0000        male   male  0  0  1  0
83       0.0000        male   male  0  0  1  0
84       0.0003        male   male  0  0  1  0
85       0.0086        male   male  0  0  1  0
86       0.0347        male   male  0  0  1  0
87       0.0029        male   male  0  0  1  0
88       0.0163        male   male  0  0  1  0
89       0.0000        male   male  0  0  1  0
90       0.0000        male   male  0  0  1  0
91       0.0004        male   male  0  0  1  0
92       0.0000        male   male  0  0  1  0
93       0.8409      female   male  0  0  0  1

Threshold = 0.5

PenguinPrediction_0point5 <- PenguinPrediction |> 
  mutate(.pred_female = round(.pred_female, 4))

PenguinPrediction_0point5 <- PenguinPrediction_0point5 |> 
  mutate(TP = if_else(.pred_female >= 0.5 & sex == "female", 1, 0))


PenguinPrediction_0point5 <- PenguinPrediction_0point5 |> 
  mutate(FN = if_else(.pred_female < 0.5 & sex == "female", 1, 0))

PenguinPrediction_0point5 <- PenguinPrediction_0point5 |> 
  mutate(TN = if_else(.pred_female < 0.5 & sex == "male", 1, 0))


PenguinPrediction_0point5 <- PenguinPrediction_0point5 |> 
  mutate(FP = if_else(.pred_female >= 0.5 & sex == "male", 1, 0))

PenguinPrediction_0point5
   .pred_female .pred_class    sex TP FN TN FP
1        0.9922      female female  1  0  0  0
2        0.9542      female female  1  0  0  0
3        0.9847      female female  1  0  0  0
4        0.1870        male female  0  1  0  0
5        0.9947      female female  1  0  0  0
6        1.0000      female female  1  0  0  0
7        0.9594      female female  1  0  0  0
8        0.9999      female female  1  0  0  0
9        1.0000      female female  1  0  0  0
10       0.3394        male female  0  1  0  0
11       0.9130      female female  1  0  0  0
12       0.9987      female female  1  0  0  0
13       0.9998      female female  1  0  0  0
14       0.9875      female female  1  0  0  0
15       0.9914      female female  1  0  0  0
16       0.9824      female female  1  0  0  0
17       0.9945      female female  1  0  0  0
18       1.0000      female female  1  0  0  0
19       1.0000      female female  1  0  0  0
20       0.9930      female female  1  0  0  0
21       0.9694      female female  1  0  0  0
22       0.9746      female female  1  0  0  0
23       0.0820        male female  0  1  0  0
24       1.0000      female female  1  0  0  0
25       0.9971      female female  1  0  0  0
26       1.0000      female female  1  0  0  0
27       0.9817      female female  1  0  0  0
28       0.9997      female female  1  0  0  0
29       0.9961      female female  1  0  0  0
30       0.9995      female female  1  0  0  0
31       0.9987      female female  1  0  0  0
32       0.9994      female female  1  0  0  0
33       0.9999      female female  1  0  0  0
34       1.0000      female female  1  0  0  0
35       0.9970      female female  1  0  0  0
36       0.9865      female female  1  0  0  0
37       1.0000      female female  1  0  0  0
38       0.9856      female female  1  0  0  0
39       0.9999      female female  1  0  0  0
40       0.1963        male   male  0  0  1  0
41       0.0000        male   male  0  0  1  0
42       0.0000        male   male  0  0  1  0
43       0.0000        male   male  0  0  1  0
44       0.0000        male   male  0  0  1  0
45       0.7763      female   male  0  0  0  1
46       0.1981        male   male  0  0  1  0
47       0.9007      female   male  0  0  0  1
48       0.3412        male   male  0  0  1  0
49       0.0000        male   male  0  0  1  0
50       0.0096        male   male  0  0  1  0
51       0.0322        male   male  0  0  1  0
52       0.0126        male   male  0  0  1  0
53       0.0942        male   male  0  0  1  0
54       0.0000        male   male  0  0  1  0
55       0.0038        male   male  0  0  1  0
56       0.0108        male   male  0  0  1  0
57       0.0004        male   male  0  0  1  0
58       0.0348        male   male  0  0  1  0
59       0.0058        male   male  0  0  1  0
60       0.0016        male   male  0  0  1  0
61       0.0001        male   male  0  0  1  0
62       0.0002        male   male  0  0  1  0
63       0.0005        male   male  0  0  1  0
64       0.0032        male   male  0  0  1  0
65       0.0000        male   male  0  0  1  0
66       0.1099        male   male  0  0  1  0
67       0.0001        male   male  0  0  1  0
68       0.0008        male   male  0  0  1  0
69       0.2207        male   male  0  0  1  0
70       0.0021        male   male  0  0  1  0
71       0.0353        male   male  0  0  1  0
72       0.0000        male   male  0  0  1  0
73       0.0003        male   male  0  0  1  0
74       0.0000        male   male  0  0  1  0
75       0.0000        male   male  0  0  1  0
76       0.2960        male   male  0  0  1  0
77       0.0359        male   male  0  0  1  0
78       0.0000        male   male  0  0  1  0
79       0.0000        male   male  0  0  1  0
80       0.0001        male   male  0  0  1  0
81       0.0012        male   male  0  0  1  0
82       0.0000        male   male  0  0  1  0
83       0.0000        male   male  0  0  1  0
84       0.0003        male   male  0  0  1  0
85       0.0086        male   male  0  0  1  0
86       0.0347        male   male  0  0  1  0
87       0.0029        male   male  0  0  1  0
88       0.0163        male   male  0  0  1  0
89       0.0000        male   male  0  0  1  0
90       0.0000        male   male  0  0  1  0
91       0.0004        male   male  0  0  1  0
92       0.0000        male   male  0  0  1  0
93       0.8409      female   male  0  0  0  1

Threshold 0.8:

PenguinPrediction_0point8 <- PenguinPrediction |> 
  mutate(.pred_female = round(.pred_female, 4))

PenguinPrediction_0point8 <- PenguinPrediction_0point8 |> 
  mutate(TP = if_else(.pred_female >= 0.8 & sex == "female", 1, 0))


PenguinPrediction_0point8 <- PenguinPrediction_0point8 |> 
  mutate(FN = if_else(.pred_female < 0.8 & sex == "female", 1, 0))

PenguinPrediction_0point8 <- PenguinPrediction_0point8 |> 
  mutate(TN = if_else(.pred_female < 0.8 & sex == "male", 1, 0))


PenguinPrediction_0point8 <- PenguinPrediction_0point8 |> 
  mutate(FP = if_else(.pred_female >= 0.8 & sex == "male", 1, 0))

PenguinPrediction_0point8
   .pred_female .pred_class    sex TP FN TN FP
1        0.9922      female female  1  0  0  0
2        0.9542      female female  1  0  0  0
3        0.9847      female female  1  0  0  0
4        0.1870        male female  0  1  0  0
5        0.9947      female female  1  0  0  0
6        1.0000      female female  1  0  0  0
7        0.9594      female female  1  0  0  0
8        0.9999      female female  1  0  0  0
9        1.0000      female female  1  0  0  0
10       0.3394        male female  0  1  0  0
11       0.9130      female female  1  0  0  0
12       0.9987      female female  1  0  0  0
13       0.9998      female female  1  0  0  0
14       0.9875      female female  1  0  0  0
15       0.9914      female female  1  0  0  0
16       0.9824      female female  1  0  0  0
17       0.9945      female female  1  0  0  0
18       1.0000      female female  1  0  0  0
19       1.0000      female female  1  0  0  0
20       0.9930      female female  1  0  0  0
21       0.9694      female female  1  0  0  0
22       0.9746      female female  1  0  0  0
23       0.0820        male female  0  1  0  0
24       1.0000      female female  1  0  0  0
25       0.9971      female female  1  0  0  0
26       1.0000      female female  1  0  0  0
27       0.9817      female female  1  0  0  0
28       0.9997      female female  1  0  0  0
29       0.9961      female female  1  0  0  0
30       0.9995      female female  1  0  0  0
31       0.9987      female female  1  0  0  0
32       0.9994      female female  1  0  0  0
33       0.9999      female female  1  0  0  0
34       1.0000      female female  1  0  0  0
35       0.9970      female female  1  0  0  0
36       0.9865      female female  1  0  0  0
37       1.0000      female female  1  0  0  0
38       0.9856      female female  1  0  0  0
39       0.9999      female female  1  0  0  0
40       0.1963        male   male  0  0  1  0
41       0.0000        male   male  0  0  1  0
42       0.0000        male   male  0  0  1  0
43       0.0000        male   male  0  0  1  0
44       0.0000        male   male  0  0  1  0
45       0.7763      female   male  0  0  1  0
46       0.1981        male   male  0  0  1  0
47       0.9007      female   male  0  0  0  1
48       0.3412        male   male  0  0  1  0
49       0.0000        male   male  0  0  1  0
50       0.0096        male   male  0  0  1  0
51       0.0322        male   male  0  0  1  0
52       0.0126        male   male  0  0  1  0
53       0.0942        male   male  0  0  1  0
54       0.0000        male   male  0  0  1  0
55       0.0038        male   male  0  0  1  0
56       0.0108        male   male  0  0  1  0
57       0.0004        male   male  0  0  1  0
58       0.0348        male   male  0  0  1  0
59       0.0058        male   male  0  0  1  0
60       0.0016        male   male  0  0  1  0
61       0.0001        male   male  0  0  1  0
62       0.0002        male   male  0  0  1  0
63       0.0005        male   male  0  0  1  0
64       0.0032        male   male  0  0  1  0
65       0.0000        male   male  0  0  1  0
66       0.1099        male   male  0  0  1  0
67       0.0001        male   male  0  0  1  0
68       0.0008        male   male  0  0  1  0
69       0.2207        male   male  0  0  1  0
70       0.0021        male   male  0  0  1  0
71       0.0353        male   male  0  0  1  0
72       0.0000        male   male  0  0  1  0
73       0.0003        male   male  0  0  1  0
74       0.0000        male   male  0  0  1  0
75       0.0000        male   male  0  0  1  0
76       0.2960        male   male  0  0  1  0
77       0.0359        male   male  0  0  1  0
78       0.0000        male   male  0  0  1  0
79       0.0000        male   male  0  0  1  0
80       0.0001        male   male  0  0  1  0
81       0.0012        male   male  0  0  1  0
82       0.0000        male   male  0  0  1  0
83       0.0000        male   male  0  0  1  0
84       0.0003        male   male  0  0  1  0
85       0.0086        male   male  0  0  1  0
86       0.0347        male   male  0  0  1  0
87       0.0029        male   male  0  0  1  0
88       0.0163        male   male  0  0  1  0
89       0.0000        male   male  0  0  1  0
90       0.0000        male   male  0  0  1  0
91       0.0004        male   male  0  0  1  0
92       0.0000        male   male  0  0  1  0
93       0.8409      female   male  0  0  0  1

a. Accuracy = (# of correct predictions) / (All kinds of predictions made)

Accuracy  = (TP+TN )/TP+TN+FP+FN

TPTN <- PenguinPrediction_0point2 |> filter(TP==1 | TN==1) |>count()
Accuracy0p2 = (TPTN/TotalObservations)*100
Accuracy0p2
         n
1 91.39785
TPTN <- PenguinPrediction_0point5 |> filter(TP==1 | TN==1) |>count()
Accuracy0p5= (TPTN/TotalObservations)*100
Accuracy0p5
         n
1 93.54839
TPTN <- PenguinPrediction_0point8 |> filter(TP==1 | TN==1) |>count()
Accuracy0p8= (TPTN/TotalObservations)*100
Accuracy0p8
         n
1 94.62366

b.      Precision = TP / (TP+FP) (How often is our model correct)

TP0p2 <-PenguinPrediction_0point2 |> filter (TP==1) |> count()
FP0p2 <-PenguinPrediction_0point2 |> filter (FP==1) |>count()
Precision0p2 = (TP0p2/(TP0p2+FP0p2))
Precision0p2
          n
1 0.8604651
TP0p5 <-PenguinPrediction_0point5 |> filter (TP==1) |> count()
FP0p5 <-PenguinPrediction_0point5 |> filter (FP==1) |>count()
Precision0p5 = (TP0p5/(TP0p5+FP0p5))
Precision0p5
          n
1 0.9230769
TP0p8 <-PenguinPrediction_0point8 |> filter (TP==1) |> count()
FP0p8 <-PenguinPrediction_0point8 |> filter (FP==1) |>count()
Precision0p8 = (TP0p8/(TP0p8+FP0p8)) 
Precision0p8
          n
1 0.9473684

c.      Recall = a performance metric for classification models that measures the fraction of actual positive instances that a model correctly identifies

Formula used is :

Recall = TP/(TP+FN)

FN0p2 <-PenguinPrediction_0point2 |> filter (FN==1) |>count()
Recall0p2 = (TP0p2/(TP0p2+FN0p2))
Recall0p2
          n
1 0.9487179
FN0p5 <-PenguinPrediction_0point5 |> filter(FN==1) |> count()
Recall0p5 = (TP0p5/(TP0p5+FN0p5))
Recall0p5
          n
1 0.9230769
FN0p8 <-PenguinPrediction_0point8 |> filter(FN==1) |> count()
Recall0p8 = (TP0p8/(TP0p8+FN0p8))
Recall0p8
          n
1 0.9230769

When the stakes are high, we use the Recall

d.      F1 score, this metric is a single metric that is obtained by combining precision and recall. It is often preferred over accuracy

F1 score = 2 x ((Precision x Recall)/(Precission + Recall)

F1Score0p2 = 2*(Precision0p2 * Recall0p2)/(Precision0p2+Recall0p2)
F1Score0p2
         n
1 0.902439
F1Score0p5 = 2*(Precision0p5 * Recall0p5)/(Precision0p5+Recall0p5)
F1Score0p5
          n
1 0.9230769
F1Score0p8 = 2*(Precision0p8 * Recall0p8)/(Precision0p8+Recall0p8)
F1Score0p8
          n
1 0.9350649


Precision0p2

Threshold Use Cases

Whether we choose 0.2 or 0.8 depends on what is more important. 
0.8 in cases where we prefer having false positive over false negative.  0.2 in a case where threshold is loosened to avoid risk of missing something imp. 

0 in case of detecting something like cancer
0.8 in predicting whether Professor Catlin will give a homework or not 😊