Logistic Regression Metrics

Overview

To evaluate the performance of a logistic regression, we need some metrics which will help us to know better about how well the model performs. Among those metrics are the accuracy score which predicts the overall accuracy of the model, the precision, the receiver operating characteristic curve (ROC) which illustrates the diagnostic ability of a binary classifier system….

In this blog, we are going to run a logistic model and check how well it performs on the data set by looking at different metrics.

The data are from an auto insurance company and can be found here to get more ideas on that as well.

Note that the data are already prepared and transformed in advance so no further cleansing are needed for this regression model that we are going to build.

The model

Let take a look on the data at first:

df1 <- read.csv("https://raw.githubusercontent.com/jnataky/Computational_Mathematics/main/records.csv")

glimpse(df1)

## Rows: 8,161
## Columns: 25
## $ TARGET_FLAG <int> 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1~
## $ TARGET_AMT  <dbl> 0.000, 0.000, 0.000, 0.000, 0.000, 2946.000, 0.000, 4021.0~
## $ KIDSDRIV    <int> 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
## $ AGE         <int> 60, 43, 35, 51, 50, 34, 54, 37, 34, 50, 53, 43, 55, 53, 45~
## $ HOMEKIDS    <int> 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 2, 1~
## $ YOJ         <int> 11, 11, 10, 14, 11, 12, 11, 11, 10, 7, 14, 5, 11, 11, 0, 1~
## $ PARENT1     <chr> "No", "No", "No", "No", "No", "Yes", "No", "No", "No", "No~
## $ TRAVTIME    <int> 14, 22, 5, 32, 36, 46, 33, 44, 34, 48, 15, 36, 25, 64, 48,~
## $ CAR_USE     <chr> "Private", "Commercial", "Private", "Private", "Private", ~
## $ TIF         <int> 11, 1, 4, 7, 1, 1, 1, 1, 1, 7, 1, 7, 7, 6, 1, 6, 6, 7, 4, ~
## $ RED_CAR     <chr> "yes", "yes", "no", "yes", "no", "no", "no", "yes", "no", ~
## $ CLM_FREQ    <int> 2, 0, 2, 0, 2, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 2~
## $ REVOKED     <chr> "No", "No", "No", "No", "Yes", "No", "No", "Yes", "No", "N~
## $ MVR_PTS     <int> 3, 0, 3, 0, 3, 0, 0, 10, 0, 1, 0, 0, 3, 3, 3, 0, 0, 0, 0, ~
## $ CAR_AGE     <int> 18, 1, 10, 6, 17, 7, 1, 7, 1, 17, 11, 1, 9, 10, 5, 13, 16,~
## $ INCOME      <int> 67349, 91449, 16039, 54028, 114986, 125301, 18755, 107961,~
## $ HOME_VAL    <int> 0, 257252, 124191, 306251, 243925, 0, 161160, 333680, 0, 0~
## $ BLUEBOOK    <int> 14230, 14940, 4010, 15440, 18000, 17430, 8780, 16970, 1120~
## $ OLDCLAIM    <int> 4461, 0, 38690, 0, 19217, 0, 0, 2374, 0, 0, 0, 0, 5028, 0,~
## $ MSTATUS     <chr> "No", "No", "Yes", "Yes", "Yes", "No", "Yes", "Yes", "No",~
## $ SEX         <chr> "M", "M", "F", "M", "F", "F", "F", "M", "F", "M", "F", "F"~
## $ EDUCATION   <chr> "PhD", "High School", "High School", "<High School", "PhD"~
## $ JOB         <chr> "Professional", "Blue Collar", "Clerical", "Blue Collar", ~
## $ CAR_TYPE    <chr> "Minivan", "Minivan", "SUV", "Minivan", "SUV", "Sports Car~
## $ URBANICITY  <chr> "Highly Urban/ Urban", "Highly Urban/ Urban", "Highly Urba~

Let build the first full model for logistic regression…

## 
## Call:
## glm(formula = TARGET_FLAG ~ ., family = "binomial", data = df_rm)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.5654  -0.7168  -0.3972   0.6148   3.1535  
## 
## Coefficients:
##                                 Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                   -2.994e+00  3.520e-01  -8.506  < 2e-16 ***
## KIDSDRIV                       3.455e-01  6.381e-02   5.414 6.16e-08 ***
## AGE                            3.193e-04  4.145e-03   0.077 0.938603    
## HOMEKIDS                       6.304e-02  3.822e-02   1.649 0.099118 .  
## YOJ                           -1.136e-02  8.872e-03  -1.280 0.200450    
## PARENT1Yes                     3.853e-01  1.131e-01   3.406 0.000659 ***
## TRAVTIME                       1.492e-02  1.953e-03   7.641 2.16e-14 ***
## CAR_USEPrivate                -7.534e-01  9.459e-02  -7.965 1.65e-15 ***
## TIF                           -5.512e-02  7.590e-03  -7.261 3.83e-13 ***
## RED_CARyes                     1.800e-03  8.927e-02   0.020 0.983911    
## CLM_FREQ                       2.025e-01  2.938e-02   6.892 5.51e-12 ***
## REVOKEDYes                     8.804e-01  9.439e-02   9.327  < 2e-16 ***
## MVR_PTS                        1.142e-01  1.405e-02   8.132 4.22e-16 ***
## CAR_AGE                       -4.559e-03  7.797e-03  -0.585 0.558745    
## INCOME                        -3.107e-06  1.116e-06  -2.783 0.005386 ** 
## HOME_VAL                      -1.290e-06  3.525e-07  -3.658 0.000254 ***
## BLUEBOOK                      -2.147e-05  5.439e-06  -3.949 7.86e-05 ***
## OLDCLAIM                      -1.288e-05  4.011e-06  -3.211 0.001324 ** 
## MSTATUSYes                    -4.902e-01  8.656e-02  -5.663 1.49e-08 ***
## SEXM                           8.549e-02  1.154e-01   0.741 0.458957    
## EDUCATIONBachelors            -3.626e-01  1.198e-01  -3.027 0.002470 ** 
## EDUCATIONHigh School           3.528e-02  9.786e-02   0.361 0.718470    
## EDUCATIONMasters              -2.129e-01  1.865e-01  -1.141 0.253692    
## EDUCATIONPhD                  -1.214e-01  2.231e-01  -0.544 0.586367    
## JOBBlue Collar                 3.198e-01  1.922e-01   1.664 0.096189 .  
## JOBClerical                    4.362e-01  2.034e-01   2.144 0.031997 *  
## JOBDoctor                     -3.528e-01  2.716e-01  -1.299 0.193892    
## JOBHome Maker                  2.764e-01  2.176e-01   1.270 0.204072    
## JOBLawyer                      1.035e-01  1.749e-01   0.592 0.554008    
## JOBManager                    -6.151e-01  1.784e-01  -3.449 0.000563 ***
## JOBProfessional                1.439e-01  1.842e-01   0.781 0.434798    
## JOBStudent                     1.991e-01  2.226e-01   0.894 0.371160    
## CAR_TYPEPanel Truck            6.109e-01  1.669e-01   3.661 0.000251 ***
## CAR_TYPEPickup                 5.267e-01  1.041e-01   5.058 4.24e-07 ***
## CAR_TYPESports Car             9.979e-01  1.341e-01   7.440 1.01e-13 ***
## CAR_TYPESUV                    7.535e-01  1.147e-01   6.570 5.03e-11 ***
## CAR_TYPEVan                    5.872e-01  1.304e-01   4.503 6.71e-06 ***
## URBANICITYHighly Urban/ Urban  2.373e+00  1.164e-01  20.385  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 8814.1  on 7650  degrees of freedom
## Residual deviance: 6831.7  on 7613  degrees of freedom
##   (510 observations deleted due to missingness)
## AIC: 6907.7
## 
## Number of Fisher Scoring iterations: 5

Metrics

It is time to go to our metrics and see how well the model performs.

To do so, we are going to construct a confusion which will details different metrics. confusionMatrix function we are going to use here is from the package “caret”. You will need to load that library and call the function.

probabilities <- predict(rm_model1, df_rm, type = "response")
predicted1.classes <- ifelse(probabilities > 0.5, 1, 0)
df_rm$pred1.class <- predicted1.classes
table("Predictions" = df_rm$pred1.class, "Actual" = df_rm$TARGET_FLAG)

##            Actual
## Predictions    0    1
##           0 5216 1172
##           1  424  839

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    0    1
##          0 5216 1172
##          1  424  839
##                                           
##                Accuracy : 0.7914          
##                  95% CI : (0.7821, 0.8005)
##     No Information Rate : 0.7372          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.3885          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##             Sensitivity : 0.9248          
##             Specificity : 0.4172          
##          Pos Pred Value : 0.8165          
##          Neg Pred Value : 0.6643          
##              Prevalence : 0.7372          
##          Detection Rate : 0.6817          
##    Detection Prevalence : 0.8349          
##       Balanced Accuracy : 0.6710          
##                                           
##        'Positive' Class : 0               
## 

The confusion matrix gave us most the metrics and now we are going to build ROC using roc from pROC package

Here you go! The confusion matrix and the ROC curve show us how well our logistic model performs as all the metrics are determined.