AS3-2 Predicting Parole Violators

---

Introduction

議題：用申請假釋者的屬性，預測他會不會違反假釋規定

學習重點：

• 設定隨機種子set.seed()，依比例分割資料
• 從邏輯式回歸的係數推算自變數的邊際效果
• 勝率和機率、勝率比和機率差 之間的換算
• 臨界機率對混淆矩陣(期望報酬)的影響payoff = matrix(c(0,-100,-10,-50),2,2); payoff
• AUC的實質意義payoff = matrix(c(100, -80, -20, 100),2,2); payoff
• 如何(從報酬矩陣)決定臨界機率
• 什麼是抽樣偏差,如何避免、如何修正

---

1 資料處理 Loading the Dataset

【1.1 讀進資料】How many parolees are contained in the dataset?

#675

【1.2 底線機率】How many of the parolees in the dataset violated the terms of their parole?

table(parole$violator)

  0   1 
597  78 

---

2 整理資料框 Creating Our Prediction Model

【2.1 類別變數】Which variables in this dataset are unordered factors with at least three levels?

table(parole$state)

  1   2   3   4 
143 120  82 330 

#state crime

【2.2 資料框摘要】How does the output of summary() change for a factor variable as compared to a numerical variable?

summary(parole$age)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   18.4    25.4    33.7    34.5    42.5    67.0 

as.factor(parole$crime)

  [1] 4 3 3 1 1 4 3 1 3 2 1 1 3 3 3 1 3 1 3 1 1 3 3 3 1 1 3 1 2 1 1 1 1 1 3 1 1
 [38] 1 3 1 2 1 3 2 1 4 1 4 4 4 1 1 4 1 1 1 1 1 1 1 1 1 2 1 1 3 1 1 1 3 2 1 2 1
 [75] 3 1 1 3 3 3 3 3 1 1 3 2 3 3 4 1 1 3 3 3 3 3 3 3 3 1 3 1 3 1 3 1 3 1 3 3 1
[112] 1 1 3 3 3 2 1 3 3 3 3 3 3 3 1 1 3 3 1 3 3 3 2 1 3 2 2 2 2 2 3 3 1 3 3 1 1
[149] 1 1 3 3 3 1 3 1 1 1 1 3 3 3 3 1 1 3 3 3 3 3 3 3 1 1 1 1 4 4 4 1 1 1 3 3 3
[186] 3 3 1 1 2 4 3 1 1 1 4 2 3 3 3 1 1 3 3 1 1 1 1 1 1 1 3 1 3 3 1 2 1 1 1 1 1
[223] 4 1 1 2 1 2 2 1 3 2 1 1 2 2 3 2 2 3 1 3 3 3 2 2 3 1 1 2 1 1 3 3 1 1 1 1 1
[260] 2 4 1 4 1 1 3 1 3 1 1 1 3 3 1 3 1 1 3 3 1 4 1 1 2 1 1 1 3 3 1 1 1 3 4 1 4
[297] 1 1 1 1 3 3 1 1 1 2 3 1 1 4 3 4 2 3 4 4 4 4 4 4 4 4 4 1 1 4 4 3 2 4 4 4 3
[334] 4 4 1 4 4 4 4 4 1 1 1 3 1 1 1 1 1 1 2 2 1 1 2 1 1 1 3 3 2 2 4 4 1 1 3 2 1
[371] 2 2 4 4 1 2 2 3 4 1 2 1 3 1 1 4 4 2 3 4 2 2 4 1 1 1 2 1 4 1 2 1 1 4 1 1 1
[408] 1 2 1 1 1 4 2 3 2 3 2 1 1 4 4 1 4 1 1 1 3 2 1 2 4 1 4 2 2 2 1 3 4 2 1 1 1
[445] 1 1 1 2 3 3 1 4 2 1 2 4 4 1 1 1 2 1 3 1 2 1 4 2 1 1 4 4 2 4 3 1 1 1 1 1 4
[482] 1 1 3 2 4 1 1 3 2 2 1 2 1 1 2 2 3 4 1 4 2 1 1 2 1 4 4 2 3 1 1 1 1 1 2 1 1
[519] 3 1 1 3 2 1 1 1 4 4 3 4 1 2 1 4 1 1 1 3 4 3 4 1 4 1 3 1 2 4 3 4 4 1 2 2 2
[556] 1 4 3 1 1 1 4 1 2 1 1 3 4 2 2 2 3 1 1 1 3 2 1 1 4 2 2 3 2 1 1 1 1 2 1 1 4
[593] 1 4 1 2 1 4 1 1 1 3 1 1 1 1 3 1 1 2 1 4 2 1 2 4 1 1 1 1 2 2 1 1 1 1 1 4 1
[630] 1 1 3 1 2 1 1 4 1 1 1 2 1 1 4 1 2 2 1 1 4 1 1 1 1 4 2 3 1 4 1 1 1 4 2 1 2
[667] 1 4 2 1 1 3 3 1 4
Levels: 1 2 3 4

table(parole$state)

  1   2   3   4 
143 120  82 330 

summary(parole$state)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00    2.00    3.00    2.89    4.00    4.00 

#can't transform into factor by as.factor?

---

3 資料分割 Splitting into a Training and Testing Set

【3.1 指定隨機種子、依比例分割資料】Roughly what proportion of parolees have been allocated to the training and testing sets?

set.seed(144)
library(caTools)
split = sample.split(parole$violator, SplitRatio = 0.7)
train = subset(parole, split == TRUE)
test = subset(parole, split == FALSE)
#70% of data allocated to training data, 30% to testing

【3.2 隨機種子的功用】Now, suppose you re-ran lines [1]-[5] of Problem 3.1. What would you expect? If you instead ONLY re-ran lines [3]-[5], what would you expect? If you instead called set.seed() with a different number and then re-ran lines [3]-[5] of Problem 3.1, what would you expect?

split = sample.split(parole$violator, SplitRatio = 0.7)
train = subset(parole, split == TRUE)
test = subset(parole, split == FALSE)
#[1]-[5] exact same result
#not the same as we can see in the "split value" for question 2&3

---

4 建立模型 Building a Logistic Regression Model

【4.1 顯著性】What variables are significant in this model?

model1 = glm(violator~., data=train, family="binomial")
summary(model1)

Call:
glm(formula = violator ~ ., family = "binomial", data = train)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.393  -0.495  -0.354  -0.250   2.677  

Coefficients:
                  Estimate Std. Error z value  Pr(>|z|)    
(Intercept)       -0.58813    1.16445   -0.51    0.6135    
male               0.59110    0.47233    1.25    0.2108    
race               0.85326    0.35384    2.41    0.0159 *  
age                0.00598    0.01492    0.40    0.6888    
state             -0.72208    0.15540   -4.65 0.0000034 ***
time.served       -0.24854    0.11148   -2.23    0.0258 *  
max.sentence      -0.09382    0.04519   -2.08    0.0379 *  
multiple.offenses  1.38955    0.37390    3.72    0.0002 ***
crime             -0.06962    0.13947   -0.50    0.6177    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 340.04  on 472  degrees of freedom
Residual deviance: 290.43  on 464  degrees of freedom
AIC: 308.4

Number of Fisher Scoring iterations: 5

【4.2 從回歸係數估計邊際效用】What can we say based on the coefficient of the multiple.offenses variable?

#a e^1.661 times higher of odds

【4.3 從預測值估計勝率和機率】According to the model, what are the odds this individual is a violator? What is the probability this individual is a violator?

#log(odds) = -4.2411574 + 0.3869904*male + 0.8867192*race - 0.0001756*age + 0.4433007*state2 + 0.8349797*state3 - 3.3967878*state4 - 0.1238867*time.served + 0.0802954*max.sentence + 1.6119919*multiple.offenses + 0.6837143*crime2 - 0.2781054*crime3 - 0.0117627*crime4
 1/(1+exp(1.700629))

[1] 0.1544

---

5 驗證模型 Evaluating the Model on the Testing Set

【5.1 從測試資料預測機率】What is the maximum predicted probability of a violation?

predictions = predict(model1, newdata=test, type="response")
summary(predictions)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0071  0.0456  0.0692  0.0976  0.1249  0.5269 

【5.2 從混淆矩陣計算敏感性、明確性、正確率】What is the model’s sensitivity, specificity, accuracy?

table(test$violator, as.numeric(predictions >= 0.5))

   
      0   1
  0 179   0
  1  22   1

#sensitivity= 12/(11+12) = 0.522
#specificity= 167/(167+12) = 0.933
#accuracy=(167+12)/202 = 0.886

【5.3 底線機率】What is the accuracy of a simple model that predicts that every parolee is a non-violator?

【5.4 根據報償矩陣調整臨界機率】Which of the following most likely describes their preferences and best course of action?

#The board assigns more cost to a false negative than a false positive, and should therefore use a logistic regression cutoff less than 0.5

【5.5 正確率 vs 辨識率】Which of the following is the most accurate assessment of the value of the logistic regression model with a cutoff 0.5 to a parole board, based on the model’s accuracy as compared to the simple baseline model?

#The model is likely of value to the board, and using a different logistic regression cutoff is likely to improve the model's value

【5.6 計算辨識率】Using the ROCR package, what is the AUC value for the model?

pred = prediction(predictions, test$violator)
as.numeric(performance(pred, "auc")@y.values)

[1] 0.7352

【5.7 辨識率的定義】Describe the meaning of AUC in this context.

---

6 抽樣偏差 Identifying Bias in Observational Data

【6.1 如何避免、診斷、修正抽樣偏差】How could we improve our dataset to best address selection bias?

#We should use a dataset tracking a group of parolees from the start of their parole until either they violated parole or they completed their term.

---