Approach Week 2B Classification Metrics

Author

Mubin Ejaz

I will start by loading thepenguin_predictions.csv file in a dataframe. 

  1. NULL Error Rate = (total number of observation - Samples in majority class) / Total Number of samples

Class distribution:
I’ll use ggplot to plot the actual sex  using facet_grid or facet_wrap() function. 
I’ll be plotting actual number of female and actually number of male vs total number of students in the class. 

Why knowing Null Error Rate is imp. When predicting the model:
Basically, This is how often you would be wrong if you always predicted the majority class.  So, in dire cases, let’s say there’s 100 cancer patients and 5 has cancer.  If my model always predicted no cancer, I would miss the actual patients with cancer but my model still be 95% accurate.

2.      Confusion Matrices at Multiple Thresholds

 I will use R to do the following on the penguin data that I’d load in a dataframe:

Probability Threshold 0.2 (if pred_female is 0.2 or above, it’s female meaning 1 else it’s male which is 0)

3.Performance Metrics:
In my homework submission of codebase deliverable, I’ll use R to load, calculate, and plot the following three metrics
Probability threshold 0.2:

a. Accuracy = (# of correct predictions) / (All kinds of predictions made)
 Accuracy  =  TP+TN
TP+TN+FP+FN

b.      Precision = TP / (TP+FP) (How often is our model correct)

c.      Recall = a performance metric for classification models that measures the fraction of actual positive instances that a model correctly identifies

Formula used is : Recall = TP/(TP+FN)

When the stakes are high, we use the Recall

d.      F1 score, this metric is a single metric that is obtained by combining precision and recall. It is often preferred over accuracy

F1 score = 2 x ((Precision x Recall)/(Precission + Recall)

Threshold Use Cases

Whether we choose 0.2 or 0.8 depends on what is more important. 
0.2 in cases where we prefer having false positive over false negative.  0.8 in a case where threshold is loosened to avoid risk of missing something imp. 

0.2 in case of detecting something like cancer
0.8 in predicting whether Professor Catlin will give a homework or not 😊