AS3-2 Group-4 Predicting Parole Violators

rm(list=ls(all=T))
options(digits=4, scipen=12)
library(magrittr)

---

Introduction

議題：用申請假釋者的屬性，預測他會不會違反假釋規定

學習重點：

• 設定隨機種子set.seed()，依比例分割資料
• 從邏輯式回歸的係數推算自變數的邊際效果
• 勝率和機率、勝率比和機率差 之間的換算
• 臨界機率對混淆矩陣(期望報酬)的影響payoff = matrix(c(0,-100,-10,-50),2,2); payoff
• AUC的實質意義payoff = matrix(c(100, -80, -20, 100),2,2); payoff
• 如何(從報酬矩陣)決定臨界機率
• 什麼是抽樣偏差,如何避免、如何修正

---

1 資料處理 Loading the Dataset

【1.1 讀進資料】How many parolees are contained in the dataset?

nrow(parole)

[1] 675

【1.2 底線機率】How many of the parolees in the dataset violated the terms of their parole?

sum(parole$violator==1)

[1] 78

---

2 整理資料框 Creating Our Prediction Model

【2.1 類別變數】Which variables in this dataset are unordered factors with at least three levels?

str(parole)

'data.frame':   675 obs. of  9 variables:
 $ male             : int  1 0 1 1 1 1 1 0 0 1 ...
 $ race             : int  1 1 2 1 2 2 1 1 1 2 ...
 $ age              : num  33.2 39.7 29.5 22.4 21.6 46.7 31 24.6 32.6 29.1 ...
 $ state            : int  1 1 1 1 1 1 1 1 1 1 ...
 $ time.served      : num  5.5 5.4 5.6 5.7 5.4 6 6 4.8 4.5 4.7 ...
 $ max.sentence     : int  18 12 12 18 12 18 18 12 13 12 ...
 $ multiple.offenses: int  0 0 0 0 0 0 0 0 0 0 ...
 $ crime            : int  4 3 3 1 1 4 3 1 3 2 ...
 $ violator         : int  0 0 0 0 0 0 0 0 0 0 ...

• state , crime

【2.2 資料框摘要】How does the output of summary() change for a factor variable as compared to a numerical variable?

summary(parole)

      male            race           age       state  
 Min.   :0.000   Min.   :1.00   Min.   :18.4   1:143  
 1st Qu.:1.000   1st Qu.:1.00   1st Qu.:25.4   2:120  
 Median :1.000   Median :1.00   Median :33.7   3: 82  
 Mean   :0.807   Mean   :1.42   Mean   :34.5   4:330  
 3rd Qu.:1.000   3rd Qu.:2.00   3rd Qu.:42.5          
 Max.   :1.000   Max.   :2.00   Max.   :67.0          
  time.served    max.sentence  multiple.offenses crime  
 Min.   :0.00   Min.   : 1.0   Min.   :0.000     1:315  
 1st Qu.:3.25   1st Qu.:12.0   1st Qu.:0.000     2:106  
 Median :4.40   Median :12.0   Median :1.000     3:153  
 Mean   :4.20   Mean   :13.1   Mean   :0.536     4:101  
 3rd Qu.:5.20   3rd Qu.:15.0   3rd Qu.:1.000            
 Max.   :6.00   Max.   :18.0   Max.   :1.000            
    violator    
 Min.   :0.000  
 1st Qu.:0.000  
 Median :0.000  
 Mean   :0.116  
 3rd Qu.:0.000  
 Max.   :1.000  

---

3 資料分割 Splitting into a Training and Testing Set

【3.1 指定隨機種子、依比例分割資料】Roughly what proportion of parolees have been allocated to the training and testing sets?

set.seed(144)
library(caTools)
split = sample.split(parole$violator, SplitRatio = 0.7)
train = subset(parole, split == TRUE)
test = subset(parole, split == FALSE)

【3.2 隨機種子的功用】Now, suppose you re-ran lines [1]-[5] of Problem 3.1. What would you expect? If you instead ONLY re-ran lines [3]-[5], what would you expect? If you instead called set.seed() with a different number and then re-ran lines [3]-[5] of Problem 3.1, what would you expect?

# The exact same training/testing set split as the first execution of [1]-[5] 
# A different training/testing set split from the first execution of [1]-[5] 
# A different training/testing set split from the first execution of [1]-[5]

---

4 建立模型 Building a Logistic Regression Model

【4.1 顯著性】What variables are significant in this model?

model1 = glm(violator ~ .,data= train , family = "binomial")
summary(model1)

Call:
glm(formula = violator ~ ., family = "binomial", data = train)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.704  -0.424  -0.272  -0.169   2.837  

Coefficients:
                   Estimate Std. Error z value    Pr(>|z|)    
(Intercept)       -4.241157   1.293885   -3.28       0.001 ** 
male               0.386990   0.437961    0.88       0.377    
race               0.886719   0.395066    2.24       0.025 *  
age               -0.000176   0.016085   -0.01       0.991    
state2             0.443301   0.481662    0.92       0.357    
state3             0.834980   0.556270    1.50       0.133    
state4            -3.396788   0.611586   -5.55 0.000000028 ***
time.served       -0.123887   0.120423   -1.03       0.304    
max.sentence       0.080295   0.055375    1.45       0.147    
multiple.offenses  1.611992   0.385305    4.18 0.000028683 ***
crime2             0.683714   0.500355    1.37       0.172    
crime3            -0.278105   0.432836   -0.64       0.521    
crime4            -0.011763   0.571304   -0.02       0.984    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 340.04  on 472  degrees of freedom
Residual deviance: 251.48  on 460  degrees of freedom
AIC: 277.5

Number of Fisher Scoring iterations: 6

• race , state4 , multiple.offenses

【4.2 從回歸係數估計邊際效用】What can we say based on the coefficient of the multiple.offenses variable?

exp(1.611992)

[1] 5.013

【4.3 從預測值估計勝率和機率】According to the model, what are the odds this individual is a violator? What is the probability this individual is a violator?

p

[1] 0.1544

---

5 驗證模型 Evaluating the Model on the Testing Set

【5.1 從測試資料預測機率】What is the maximum predicted probability of a violation?

Pred=predict(model1,type="response",test)
max(Pred)

[1] 0.9073

【5.2 從混淆矩陣計算敏感性、明確性、正確率】What is the model’s sensitivity, specificity, accuracy?

table(test$violator,Pred>0.5)

   
    FALSE TRUE
  0   167   12
  1    11   12

12/(11+12)

[1] 0.5217

167/(167+12)

[1] 0.933

(167+12)/(167+12+11+12)

[1] 0.8861

【5.3 底線機率】What is the accuracy of a simple model that predicts that every parolee is a non-violator?

table(test$violator)

  0   1 
179  23 

179/(179+23)

[1] 0.8861

【5.4 根據報償矩陣調整臨界機率】Which of the following most likely describes their preferences and best course of action?

• The board assigns more cost to a false negative than a false positive, and should therefore use a logistic regression cutoff less than 0.5.

【5.5 正確率 vs 辨識率】Which of the following is the most accurate assessment of the value of the logistic regression model with a cutoff 0.5 to a parole board, based on the model’s accuracy as compared to the simple baseline model?

• The model is likely of value to the board, and using a different logistic regression cutoff is likely to improve the model’s value.

【5.6 計算辨識率】Using the ROCR package, what is the AUC value for the model?

auc

[1] 0.7421

【5.7 辨識率的定義】Describe the meaning of AUC in this context.

• The probability the model can correctly differentiate between a randomly selected parole violator and a randomly selected parole non-violator.

---

6 抽樣偏差 Identifying Bias in Observational Data

【6.1 如何避免、診斷、修正抽樣偏差】How could we improve our dataset to best address selection bias?

• We should use a dataset tracking a group of parolees from the start of their parole until either they violated parole or they completed their term.

---