目錄

1. Logistic Regression
2. Decision Tree
3. Random Forest
4. Ensambling


1. 邏輯式迴歸分析(Logistic Regression)


Figure 1 - The Flowcart

Figure 1 - The Flowcart



1.1 Prepare the Dataset


【檢查缺項】

summary(X)
is.na(X$a)


【分析缺項:移除 or 填補】

missing_data = subset(is.na(x$a|x$c|x$e)) 
#X裡具至少一個以上NA的子項目(a,c,e)挑出來
#不僅是缺項單位格,該缺項所在一整列資料(row)
summary(missing_data) #包含原先X的所有自變數(a,b,c,d,e)欄位 #即沒有NA值的欄位(b,d)也在
nrow(missing_data)
table(missing$Y) #找出應變數Y,檢查要移除還是要填補NA項
table中是NA且Y=1的值/全部有幾個NA


【補缺項工具】

library(mice)
set.seed(144) #Note that the imputation results are not the same even if you set the random seed.
vars.for.imputation = setdiff(names(X), "Y") #設定vars.for.imputation來移除Y
imputed = complete(mice(X[vars.for.imputation])) #只用所有自變數來估算NA值


Imputation predicts missing variable values for a given observation using the variable values that are reported. We called the imputation on a data frame with the dependent Y removed, so we predicted the missing values using only other independent variables.


1.2 Random Split

library(caTools)
set.seed()
sample.split(X$Y, SplitRatio=)
TR = subset(X, spl==TRUE)
TS = subset(X, spl==FALSE)


Now that we have prepared the dataset, we need to split it into a training and testing set. To ensure everybody obtains the same split, set the random seed to 144 (even though you already did so earlier in the problem).


【隨機種子的功用】

  • Different seed -> get different splits.
  • Different splitRatio -> get different splits.
  • Re-ran all steps -> get the same result: set a random seed, split, set the seed again to the same value, and then split again, you will get the same split.
  • Re-ran step3~5 -> get different result: set the seed and then split twice, you will get different splits.


【從迴歸係數估計邊際效用】

  • ln(oddA) = ln(oddB) + 1.61
  • oddA = exp(1.61) * oddB
  • Rule: e^(a+b) = e^a * e^b.

Q: Application A has FICO credit score 700 while the borrower in Application B has FICO credit score 710. What is the value of Logit(A) - Logit(B)? What is the value of O(A)/O(B)?

# the difference of logits
logitA = -0.00931679*700 #係數相同,x值不同
logitB = -0.00931679*710
logitA - logitB #0.09317
# the ratio of odds
oddA = exp(logitA) #法1
oddB = exp(logitB)
oddA/oddB #1.098
exp(logitA - logitB) or exp(0.09317) #法2

1.3 Build the Model

modelLog = glm(Y~ x1+x2+..., data=TR, family=binomial)
pred = predict(modelLog, data=, type="response") #response:回傳機率


【訓練模型的AUC】

library(ROCR)
modelTR = glm(Y~ x1+x2+..., data=TR, family=binomial)
PredTR = predict(modelTR, type = "response")
PredROCR = prediction(PredTR, TR$Y)
PerfROCR = performance(PredROCR, "tpr","fpr") #為了畫AUC圖
par(cex=0.8)
plot(PerfROCR, colorize=TRUE, print.cutoffs.at=seq(0,1,0.1), text.adj=c(-0.2,1,7)) #AUC圖 #text.adj加了美觀,讓ROC線上的值右移一點,不重疊在線上
as.numeric(performance(PredROCR, "auc")@y.values) #AUC實際值


【預測模型的AUC】

library(ROCR)
modelTR = glm(Y~ x1+x2+..., data=TR, family=binomial)
predTS = predict(modelTR, newdata=TS, type="response")
predROCR = prediction(predTS, TS$Y)
as.numeric(performance(predROCR, "auc")@y.values) #AUC實際值


【Confusion Matrix】
※ 只有拿訓練模型去適應Testing Data(即預測模型)後才會做

table(TS$Y, predTS >= 0.5) #threshold=0.5 

1.4 Verify the Model


【準確性:訓練模型 V.S. Baseline】
※ ACC從Confusion Matrix得來

predTS = predict(modelTR, newdata=TS, type="response")
table(TS$Y, predTS >= 0.45) #Confusion Matrix(由pred和TS data得來)
ACC = (TN+TP)/(TN+FP+FN+TP)
#baseline ACC
table(TS$Y)
ACC = (TN+TP)/(TN+FP+FN+TP)


【敏感性 & 明確性】

#Using the values from Confusion Matrix: table(TS$Y, predTS >= 0.45)
sensitivity = TP/(TP+FN)
specificity = TN/(TN+FP)


在不同的threshold條件下,會產生出不同Confusion Matrix結果;結果中包含許多資訊,如:準確率(Accuracy)、辨識率(AUC)、敏滿性(Sensitivity)與明確性(Specificity)。
依照所欲達到的目的,或是一件事情發生結果的嚴重性,都可以有不同的決策結果,故即使模型不能夠大幅度增加ACC,但能夠以不同的角度來解釋問題並且對於決策有所助益,該模型就是一個好的模型。


1.5 Make the Decision

risk=-100
action=-60
cost=-10
ExpPayoff = TN*0 + TP*action + FN*risk + Total*cost
Figure 2 - The Excepted Payoff

Figure 2 - The Excepted Payoff






2. 決策樹分析(Decision Tree)


2.1 Build the Model

library(rpart)
cart -> rpart(Y~ x1+x2+..., data=TR, method="class", minbucket=25)  #class:讓結果輸出為類別 #prob:讓結果輸出為機率,都不寫也是機率(做regression時用) #minbucket:數小樹大;數大樹小
prp(cart) #看樹長什麼樣,簡化版
library(rpart.plot)
rpart.plot(caret, cex=0.6) #看樹長什麼樣,進階版


【預測模型的ACC】

predCart = predict(cart, newdata=TS, type = "class")
table(predCart, TS$Y)
ACC = (TN+TP)/(TN+FP+FN+TP)


【預測模型的AUC】

library(ROCR)
predCart = predict(cart, newdata=TS) #把type拿掉改顯示成機率
predCart #看一下此時呈現的機率會分成Y=0及Y=1兩行
predCart = predCart[,2] #我們只需要Y=1的機率, 將Y=0得捨去
predROCR = prediction(predCart, TS$Y)
predROCR = prediction(predCart[,2], TS$Y) #法2(簡化版)
plot(perf)
as.numeric(performance(predROCR, "auc")@y.values)

2.2 Cross Validation & Parameter Tuning


Let us select the cp parameter for our CART model using k-fold cross validation, with k = 10 folds.

library(caret)
library(e1071)
numFolds = trainControl(method="cv", number=10) #10 folds cv
cpGrid = expand.grid(.cp=seq(0.01,0.5,0.01)) #parameter
cp = train(Y~., data=TR, method="rpart",
            trControl=numFolds,tuneGrid=cpGrid)
cp #selecting cp by cross-validation

2.3 The Final Model

modelCV = rpart(Y~., data=TR, method="class", cp=0.18) #cp從上式得來
predCV = predict(modelCV, newdata=TS, type="class")
table(TS$Y, predCV)
ACC = (TN+TP)/(TN+FP+FN+TP)
prp(modelCV)


Compared to the original accuracy using the default value of cp, this new CART model is an improvement, and so we should clearly favor this new model over the old one – or should we? Plot the CART tree for this model. How many splits are there?

  • In some applications, such an improvement in accuracy would be worth the loss in interpretability.
  • In others, we may prefer a less accurate model that is simpler to understand and describe over a more accurate (but more complicated) model.






3. 隨機森林分析(Random Forest)


3.1 Build the Model

※ 通常順序:建立邏輯式迴歸模型→ CART模型→ 隨機森林模型→ 交叉驗證(CART2)→ 比較上述模型並選擇其中之一。

library(randomForest)
Forest = randomForest(Y~ x1+x2+..., data = TR, nodesize=25, ntree= 200)
predForest = predict(Forest, newdata = TS)
table(TS$Y,predForest)
ACC = (TN+TP)/(TN+FP+FN+TP)


【AUC和ACC精簡公式】

colAUC(pred, TS$a)
table(TS$a, pred > 0.5) %>% {sum(diag(.))/sum(.)} 






4. 組合模型(Ensambling)


4.1 Compare ACC

px = cbind(glm=predglm, cart1=pred1, cart2=pred2, cart3=pred3, cart4 = pred4, cart5 = pred5, cart6 = pred6, cart7 = pred7, rf = PredictForest)
apply(px, 2, function(x) {
  table(TS$buy, x > 0.5) %>% {sum(diag(.))/sum(.)} 
  }) %>% sort


4.2 Compare AUC

par(cex=0.7,mar=c(6,6,5,3))
colAUC(px, TS$buy, T)


4.3 Correlation & Ensambling

  • 兩個相關性高的模型不見得是最好的組合
  • 兩個極端相反的模型組合起來反而能幫忙解釋更多不同的角度
cor(px)
glm_cart = (px[,"glm"] + px[,"cart6"])/2
glm_rf = (px[,"glm"] + px[,"rf"])/2


4.4 ACC & AUC Table

px2 = cbind(px, glm_cart, glm_rf)
rbind(apply(px2, 2, function(x) {
        table(TS$buy, x > 0.5) %>% {sum(diag(.))/sum(.)} }),
      colAUC(px2, TS$buy)) %>% t %>% 
      data.frame %>% setNames(c("Accuracy", "AUC"))        
px3 = cbind(lm=RSquarelm, cart9=RSquarerp)
rbind(px3) %>% data.frame %>% setNames(c("lm","cart"))
LS0tDQp0aXRsZTogIuWVhualreaVuOaTmuWIhuaekCDmqKHlnovoo73kvZznrYboqJgiDQphdXRob3I6ICLllJDmgJ3nkKogQjA0MTAxMDAwNCINCm91dHB1dDogaHRtbF9ub3RlYm9vaw0KLS0tDQo8YnI+DQoNCiMjIyA8Yj7nm67pjIQ8L2I+DQoNClsxLiBMb2dpc3RpYyBSZWdyZXNzaW9uXSgjbjEpIDxicj4NClsyLiBEZWNpc2lvbiBUcmVlXSgjbjIpIDxicj4NClszLiBSYW5kb20gRm9yZXN0XSgjbjMpIDxicj4NCls0LiBFbnNhbWJsaW5nXSgjbjQpIDxicj4NCg0KLSAtIC0NCg0KIyMjIDxhIGlkPSJuMSI+PC9hPiA8Yj4xLiDpgo/ovK/lvI/ov7TmrbjliIbmnpAoTG9naXN0aWMgUmVncmVzc2lvbik8L2I+DQo8YnI+DQoNCiFbRmlndXJlIDEgLSBUaGUgRmxvd2NhcnRdKHNvbHV0aW9uLmpwZykNCg0KPGJyPjxicj4NCg0KIyMjIyA8Yj4xLjEgUHJlcGFyZSB0aGUgRGF0YXNldDwvYj4NCjxicj4NCuOAkOaqouafpee8uumgheOAkQ0KYGBge3J9DQpzdW1tYXJ5KFgpDQppcy5uYShYJGEpDQpgYGANCjxicj4NCuOAkOWIhuaekOe8uumghe+8muenu+mZpCBvciDloavoo5zjgJENCmBgYHtyfQ0KbWlzc2luZ19kYXRhID0gc3Vic2V0KGlzLm5hKHgkYXx4JGN8eCRlKSkgDQojWOijoeWFt+iHs+WwkeS4gOWAi+S7peS4ik5B55qE5a2Q6aCF55uuKGEsYyxlKeaMkeWHuuS+hg0KI+S4jeWDheaYr+e8uumgheWWruS9jeagvCzoqbLnvLrpoIXmiYDlnKjkuIDmlbTliJfos4fmlpkocm93KQ0Kc3VtbWFyeShtaXNzaW5nX2RhdGEpICPljIXlkKvljp/lhYhY55qE5omA5pyJ6Ieq6K6K5pW4KGEsYixjLGQsZSnmrITkvY0gI+WNs+aykuaciU5B5YC855qE5qyE5L2NKGIsZCnkuZ/lnKgNCm5yb3cobWlzc2luZ19kYXRhKQ0KdGFibGUobWlzc2luZyRZKSAj5om+5Ye65oeJ6K6K5pW4WSzmqqLmn6XopoHnp7vpmaTpgoTmmK/opoHloavoo5xOQemghQ0KdGFibGXkuK3mmK9OQeS4lFk9MeeahOWAvC/lhajpg6jmnInlub7lgItOQQ0KYGBgDQo8YnI+DQrjgJDoo5znvLrpoIXlt6XlhbfjgJENCmBgYHtyfQ0KbGlicmFyeShtaWNlKQ0Kc2V0LnNlZWQoMTQ0KSAjTm90ZSB0aGF0IHRoZSBpbXB1dGF0aW9uIHJlc3VsdHMgYXJlIG5vdCB0aGUgc2FtZSBldmVuIGlmIHlvdSBzZXQgdGhlIHJhbmRvbSBzZWVkLg0KdmFycy5mb3IuaW1wdXRhdGlvbiA9IHNldGRpZmYobmFtZXMoWCksICJZIikgI+ioreWumnZhcnMuZm9yLmltcHV0YXRpb27kvobnp7vpmaRZDQppbXB1dGVkID0gY29tcGxldGUobWljZShYW3ZhcnMuZm9yLmltcHV0YXRpb25dKSkgI+WPqueUqOaJgOacieiHquiuiuaVuOS+huS8sOeul05B5YC8DQpgYGANCjxicj4NCkltcHV0YXRpb24gcHJlZGljdHMgbWlzc2luZyB2YXJpYWJsZSB2YWx1ZXMgZm9yIGEgZ2l2ZW4gb2JzZXJ2YXRpb24gdXNpbmcgdGhlIHZhcmlhYmxlIHZhbHVlcyB0aGF0IGFyZSByZXBvcnRlZC4gV2UgY2FsbGVkIHRoZSBpbXB1dGF0aW9uIG9uIGEgZGF0YSBmcmFtZSB3aXRoIHRoZSBkZXBlbmRlbnQgWSByZW1vdmVkLCBzbyB3ZSBwcmVkaWN0ZWQgdGhlIG1pc3NpbmcgdmFsdWVzIHVzaW5nIG9ubHkgb3RoZXIgaW5kZXBlbmRlbnQgdmFyaWFibGVzLg0KDQotIC0gLQ0KDQojIyMjIDxiPjEuMiBSYW5kb20gU3BsaXQ8L2I+DQpgYGB7cn0NCmxpYnJhcnkoY2FUb29scykNCnNldC5zZWVkKCkNCnNhbXBsZS5zcGxpdChYJFksIFNwbGl0UmF0aW89KQ0KVFIgPSBzdWJzZXQoWCwgc3BsPT1UUlVFKQ0KVFMgPSBzdWJzZXQoWCwgc3BsPT1GQUxTRSkNCmBgYA0KPGJyPg0KTm93IHRoYXQgd2UgaGF2ZSBwcmVwYXJlZCB0aGUgZGF0YXNldCwgd2UgbmVlZCB0byBzcGxpdCBpdCBpbnRvIGEgdHJhaW5pbmcgYW5kIHRlc3Rpbmcgc2V0LiBUbyBlbnN1cmUgZXZlcnlib2R5IG9idGFpbnMgdGhlIHNhbWUgc3BsaXQsIHNldCB0aGUgcmFuZG9tIHNlZWQgdG8gMTQ0IChldmVuIHRob3VnaCB5b3UgYWxyZWFkeSBkaWQgc28gZWFybGllciBpbiB0aGUgcHJvYmxlbSkuDQoNCjxicj4NCuOAkOmaqOapn+eoruWtkOeahOWKn+eUqOOAkQ0KDQorIERpZmZlcmVudCBzZWVkIC0+IGdldCBkaWZmZXJlbnQgc3BsaXRzLg0KKyBEaWZmZXJlbnQgc3BsaXRSYXRpbyAtPiBnZXQgZGlmZmVyZW50IHNwbGl0cy4NCisgUmUtcmFuIGFsbCBzdGVwcyAtPiBnZXQgdGhlIHNhbWUgcmVzdWx0OiBzZXQgYSByYW5kb20gc2VlZCwgc3BsaXQsIHNldCB0aGUgc2VlZCBhZ2FpbiB0byB0aGUgc2FtZSB2YWx1ZSwgYW5kIHRoZW4gc3BsaXQgYWdhaW4sIHlvdSB3aWxsIGdldCB0aGUgc2FtZSBzcGxpdC4NCisgUmUtcmFuIHN0ZXAzfjUgLT4gZ2V0IGRpZmZlcmVudCByZXN1bHQ6IHNldCB0aGUgc2VlZCBhbmQgdGhlbiBzcGxpdCB0d2ljZSwgeW91IHdpbGwgZ2V0IGRpZmZlcmVudCBzcGxpdHMuDQoNCjxicj4NCuOAkOW+nui/tOatuOS/guaVuOS8sOioiOmCiumam+aViOeUqOOAkQ0KDQorIGxuKG9kZEEpID0gbG4ob2RkQikgKyAxLjYxDQorIG9kZEEgPSBleHAoMS42MSkgKiBvZGRCDQorIFJ1bGU6IGVeKGErYikgPSBlXmEgKiBlXmIuDQoNClE6IEFwcGxpY2F0aW9uIEEgaGFzIEZJQ08gY3JlZGl0IHNjb3JlIDcwMCB3aGlsZSB0aGUgYm9ycm93ZXIgaW4gQXBwbGljYXRpb24gQiBoYXMgRklDTyBjcmVkaXQgc2NvcmUgNzEwLiBXaGF0IGlzIHRoZSB2YWx1ZSBvZiBMb2dpdChBKSAtIExvZ2l0KEIpPyBXaGF0IGlzIHRoZSB2YWx1ZSBvZiBPKEEpL08oQik/DQpgYGB7cn0NCiMgdGhlIGRpZmZlcmVuY2Ugb2YgbG9naXRzDQpsb2dpdEEgPSAtMC4wMDkzMTY3OSo3MDAgI+S/guaVuOebuOWQjCx45YC85LiN5ZCMDQpsb2dpdEIgPSAtMC4wMDkzMTY3OSo3MTANCmxvZ2l0QSAtIGxvZ2l0QiAjMC4wOTMxNw0KIyB0aGUgcmF0aW8gb2Ygb2Rkcw0Kb2RkQSA9IGV4cChsb2dpdEEpICPms5UxDQpvZGRCID0gZXhwKGxvZ2l0QikNCm9kZEEvb2RkQiAjMS4wOTgNCmV4cChsb2dpdEEgLSBsb2dpdEIpIG9yIGV4cCgwLjA5MzE3KSAj5rOVMg0KYGBgDQoNCi0gLSAtDQoNCiMjIyMgPGI+MS4zIEJ1aWxkIHRoZSBNb2RlbDwvYj4NCg0KYGBge3J9DQptb2RlbExvZyA9IGdsbShZfiB4MSt4MisuLi4sIGRhdGE9VFIsIGZhbWlseT1iaW5vbWlhbCkNCnByZWQgPSBwcmVkaWN0KG1vZGVsTG9nLCBkYXRhPSwgdHlwZT0icmVzcG9uc2UiKSAjcmVzcG9uc2U65Zue5YKz5qmf546HDQpgYGANCjxicj4NCuOAkOiok+e3tOaooeWei+eahEFVQ+OAkQ0KYGBge3J9DQpsaWJyYXJ5KFJPQ1IpDQptb2RlbFRSID0gZ2xtKFl+IHgxK3gyKy4uLiwgZGF0YT1UUiwgZmFtaWx5PWJpbm9taWFsKQ0KUHJlZFRSID0gcHJlZGljdChtb2RlbFRSLCB0eXBlID0gInJlc3BvbnNlIikNClByZWRST0NSID0gcHJlZGljdGlvbihQcmVkVFIsIFRSJFkpDQpQZXJmUk9DUiA9IHBlcmZvcm1hbmNlKFByZWRST0NSLCAidHByIiwiZnByIikgI+eCuuS6hueVq0FVQ+Wclg0KcGFyKGNleD0wLjgpDQpwbG90KFBlcmZST0NSLCBjb2xvcml6ZT1UUlVFLCBwcmludC5jdXRvZmZzLmF0PXNlcSgwLDEsMC4xKSwgdGV4dC5hZGo9YygtMC4yLDEsNykpICNBVUPlnJYgI3RleHQuYWRq5Yqg5LqG576O6KeALOiuk1JPQ+e3muS4iueahOWAvOWPs+enu+S4gOm7nizkuI3ph43nlorlnKjnt5rkuIoNCmFzLm51bWVyaWMocGVyZm9ybWFuY2UoUHJlZFJPQ1IsICJhdWMiKUB5LnZhbHVlcykgI0FVQ+Wvpumam+WAvA0KYGBgDQo8YnI+DQrjgJDpoJDmuKzmqKHlnovnmoRBVUPjgJENCmBgYHtyfQ0KbGlicmFyeShST0NSKQ0KbW9kZWxUUiA9IGdsbShZfiB4MSt4MisuLi4sIGRhdGE9VFIsIGZhbWlseT1iaW5vbWlhbCkNCnByZWRUUyA9IHByZWRpY3QobW9kZWxUUiwgbmV3ZGF0YT1UUywgdHlwZT0icmVzcG9uc2UiKQ0KcHJlZFJPQ1IgPSBwcmVkaWN0aW9uKHByZWRUUywgVFMkWSkNCmFzLm51bWVyaWMocGVyZm9ybWFuY2UocHJlZFJPQ1IsICJhdWMiKUB5LnZhbHVlcykgI0FVQ+Wvpumam+WAvA0KYGBgDQo8YnI+DQrjgJBDb25mdXNpb24gTWF0cml444CRPGJyPg0K4oC7IOWPquacieaLv+iok+e3tOaooeWei+WOu+mBqeaHiVRlc3RpbmcgRGF0Ye+8iOWNs+mgkOa4rOaooeWei++8ieW+jOaJjeacg+WBmg0KYGBge3J9DQp0YWJsZShUUyRZLCBwcmVkVFMgPj0gMC41KSAjdGhyZXNob2xkPTAuNSANCmBgYA0KDQotIC0gLQ0KDQojIyMjIDxiPjEuNCBWZXJpZnkgdGhlIE1vZGVsPC9iPg0KPGJyPg0K44CQ5rqW56K65oCnOuiok+e3tOaooeWeiyBWLlMuIEJhc2VsaW5l44CRPGJyPg0K4oC7IEFDQ+W+nkNvbmZ1c2lvbiBNYXRyaXjlvpfkvoY8YnI+DQpgYGB7cn0NCnByZWRUUyA9IHByZWRpY3QobW9kZWxUUiwgbmV3ZGF0YT1UUywgdHlwZT0icmVzcG9uc2UiKQ0KdGFibGUoVFMkWSwgcHJlZFRTID49IDAuNDUpICNDb25mdXNpb24gTWF0cml4KOeUsXByZWTlkoxUUyBkYXRh5b6X5L6GKQ0KQUNDID0gKFROK1RQKS8oVE4rRlArRk4rVFApDQojYmFzZWxpbmUgQUNDDQp0YWJsZShUUyRZKQ0KQUNDID0gKFROK1RQKS8oVE4rRlArRk4rVFApDQpgYGANCjxicj4NCuOAkOaVj+aEn+aApyAmIOaYjueiuuaAp+OAkQ0KYGBge3J9DQojVXNpbmcgdGhlIHZhbHVlcyBmcm9tIENvbmZ1c2lvbiBNYXRyaXg6IHRhYmxlKFRTJFksIHByZWRUUyA+PSAwLjQ1KQ0Kc2Vuc2l0aXZpdHkgPSBUUC8oVFArRk4pDQpzcGVjaWZpY2l0eSA9IFROLyhUTitGUCkNCmBgYA0KPGJyPg0KPGZvbnQgZmFjZT0i5b6u6Luf5q2j6buR6auUIj4NCuWcqOS4jeWQjOeahHRocmVzaG9sZOaineS7tuS4i++8jOacg+eUoueUn+WHuuS4jeWQjENvbmZ1c2lvbiBNYXRyaXjntZDmnpzvvJvntZDmnpzkuK3ljIXlkKvoqLHlpJros4foqIrvvIzlpoLvvJrmupbnorrnjofvvIhBY2N1cmFjee+8ieOAgei+qOitmOeOh++8iEFVQ++8ieOAgeaVj+a7v+aAp++8iFNlbnNpdGl2aXR577yJ6IiH5piO56K65oCn77yIU3BlY2lmaWNpdHnvvInjgII8YnI+DQrkvp3nhafmiYDmrLLpgZTliLDnmoTnm67nmoTvvIzmiJbmmK/kuIDku7bkuovmg4XnmbznlJ/ntZDmnpznmoTlmrTph43mgKfvvIzpg73lj6/ku6XmnInkuI3lkIznmoTmsbrnrZbntZDmnpzvvIzmlYXljbPkvb/mqKHlnovkuI3og73lpKDlpKfluYXluqblop7liqBBQ0Ms5L2G6IO95aSg5Lul5LiN5ZCM55qE6KeS5bqm5L6G6Kej6YeL5ZWP6aGM5Lim5LiU5bCN5pa85rG6562W5pyJ5omA5Yqp55uK77yM6Kmy5qih5Z6L5bCx5piv5LiA5YCL5aW955qE5qih5Z6L44CCDQo8L2ZvbnQ+DQoNCi0gLSAtDQoNCiMjIyMgPGI+MS41IE1ha2UgdGhlIERlY2lzaW9uPC9iPg0KDQpgYGB7cn0NCnJpc2s9LTEwMA0KYWN0aW9uPS02MA0KY29zdD0tMTANCkV4cFBheW9mZiA9IFROKjAgKyBUUCphY3Rpb24gKyBGTipyaXNrICsgVG90YWwqY29zdA0KYGBgDQohW0ZpZ3VyZSAyIC0gVGhlIEV4Y2VwdGVkIFBheW9mZl0ocGF5b2ZmLmpwZykNCg0KPGJyPjxicj4NCg0KLSAtIC0NCg0KPGJyPjxicj4NCg0KDQojIyMgPGEgaWQ9Im4yIj48L2E+IDxiPjIuIOaxuuetluaoueWIhuaekChEZWNpc2lvbiBUcmVlKTwvYj4NCg0KPGJyPg0KDQojIyMjIDxiPjIuMSBCdWlsZCB0aGUgTW9kZWw8L2I+DQpgYGB7cn0NCmxpYnJhcnkocnBhcnQpDQpjYXJ0IC0+IHJwYXJ0KFl+IHgxK3gyKy4uLiwgZGF0YT1UUiwgbWV0aG9kPSJjbGFzcyIsIG1pbmJ1Y2tldD0yNSkgICNjbGFzczrorpPntZDmnpzovLjlh7rngrrpoZ7liKUgI3Byb2I66K6T57WQ5p6c6Ly45Ye654K65qmf546HLOmDveS4jeWvq+S5n+aYr+apn+eOhyjlgZpyZWdyZXNzaW9u5pmC55SoKSAjbWluYnVja2V0OuaVuOWwj+aoueWkpzvmlbjlpKfmqLnlsI8NCnBycChjYXJ0KSAj55yL5qi56ZW35LuA6bq85qijLOewoeWMlueJiA0KbGlicmFyeShycGFydC5wbG90KQ0KcnBhcnQucGxvdChjYXJldCwgY2V4PTAuNikgI+eci+aouemVt+S7gOm6vOaooyzpgLLpmo7niYgNCmBgYA0KPGJyPg0K44CQ6aCQ5ris5qih5Z6L55qEQUND44CRDQpgYGB7cn0NCnByZWRDYXJ0ID0gcHJlZGljdChjYXJ0LCBuZXdkYXRhPVRTLCB0eXBlID0gImNsYXNzIikNCnRhYmxlKHByZWRDYXJ0LCBUUyRZKQ0KQUNDID0gKFROK1RQKS8oVE4rRlArRk4rVFApDQpgYGANCjxicj4NCuOAkOmgkOa4rOaooeWei+eahEFVQ+OAkQ0KYGBge3J9DQpsaWJyYXJ5KFJPQ1IpDQpwcmVkQ2FydCA9IHByZWRpY3QoY2FydCwgbmV3ZGF0YT1UUykgI+aKinR5cGXmi7/mjonmlLnpoa/npLrmiJDmqZ/njocNCnByZWRDYXJ0ICPnnIvkuIDkuIvmraTmmYLlkYjnj77nmoTmqZ/njofmnIPliIbmiJBZPTDlj4pZPTHlhanooYwNCnByZWRDYXJ0ID0gcHJlZENhcnRbLDJdICPmiJHlgJHlj6rpnIDopoFZPTHnmoTmqZ/njocsIOWwh1k9MOW+l+aNqOWOuw0KcHJlZFJPQ1IgPSBwcmVkaWN0aW9uKHByZWRDYXJ0LCBUUyRZKQ0KcHJlZFJPQ1IgPSBwcmVkaWN0aW9uKHByZWRDYXJ0WywyXSwgVFMkWSkgI+azlTIo57Ch5YyW54mIKQ0KcGxvdChwZXJmKQ0KYXMubnVtZXJpYyhwZXJmb3JtYW5jZShwcmVkUk9DUiwgImF1YyIpQHkudmFsdWVzKQ0KYGBgDQoNCi0gLSAtDQoNCiMjIyMgPGI+Mi4yIENyb3NzIFZhbGlkYXRpb24gJiBQYXJhbWV0ZXIgVHVuaW5nPC9iPg0KPGJyPg0KTGV0IHVzIHNlbGVjdCB0aGUgY3AgcGFyYW1ldGVyIGZvciBvdXIgQ0FSVCBtb2RlbCB1c2luZyBrLWZvbGQgY3Jvc3MgdmFsaWRhdGlvbiwgd2l0aCBrID0gMTAgZm9sZHMuDQpgYGB7cn0NCmxpYnJhcnkoY2FyZXQpDQpsaWJyYXJ5KGUxMDcxKQ0KbnVtRm9sZHMgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlcj0xMCkgIzEwIGZvbGRzIGN2DQpjcEdyaWQgPSBleHBhbmQuZ3JpZCguY3A9c2VxKDAuMDEsMC41LDAuMDEpKSAjcGFyYW1ldGVyDQpjcCA9IHRyYWluKFl+LiwgZGF0YT1UUiwgbWV0aG9kPSJycGFydCIsDQogICAgICAgICAgICB0ckNvbnRyb2w9bnVtRm9sZHMsdHVuZUdyaWQ9Y3BHcmlkKQ0KY3AgI3NlbGVjdGluZyBjcCBieSBjcm9zcy12YWxpZGF0aW9uDQpgYGANCg0KLSAtIC0NCg0KIyMjIyA8Yj4yLjMgVGhlIEZpbmFsIE1vZGVsPC9iPg0KDQpgYGB7cn0NCm1vZGVsQ1YgPSBycGFydChZfi4sIGRhdGE9VFIsIG1ldGhvZD0iY2xhc3MiLCBjcD0wLjE4KSAjY3Dlvp7kuIrlvI/lvpfkvoYNCnByZWRDViA9IHByZWRpY3QobW9kZWxDViwgbmV3ZGF0YT1UUywgdHlwZT0iY2xhc3MiKQ0KdGFibGUoVFMkWSwgcHJlZENWKQ0KQUNDID0gKFROK1RQKS8oVE4rRlArRk4rVFApDQpwcnAobW9kZWxDVikNCmBgYA0KPGJyPg0KPGZvbnQgZmFjZT0i5b6u6Luf5q2j6buR6auUIj4NCkNvbXBhcmVkIHRvIHRoZSBvcmlnaW5hbCBhY2N1cmFjeSB1c2luZyB0aGUgZGVmYXVsdCB2YWx1ZSBvZiBjcCwgdGhpcyBuZXcgQ0FSVCBtb2RlbCBpcyBhbiBpbXByb3ZlbWVudCwgYW5kIHNvIHdlIHNob3VsZCBjbGVhcmx5IGZhdm9yIHRoaXMgbmV3IG1vZGVsIG92ZXIgdGhlIG9sZCBvbmUgLS0gb3Igc2hvdWxkIHdlPyBQbG90IHRoZSBDQVJUIHRyZWUgZm9yIHRoaXMgbW9kZWwuIEhvdyBtYW55IHNwbGl0cyBhcmUgdGhlcmU/DQoNCisgSW4gc29tZSBhcHBsaWNhdGlvbnMsIHN1Y2ggYW4gaW1wcm92ZW1lbnQgaW4gYWNjdXJhY3kgd291bGQgYmUgd29ydGggdGhlIGxvc3MgaW4gaW50ZXJwcmV0YWJpbGl0eS4NCisgSW4gb3RoZXJzLCB3ZSBtYXkgcHJlZmVyIGEgbGVzcyBhY2N1cmF0ZSBtb2RlbCB0aGF0IGlzIHNpbXBsZXIgdG8gdW5kZXJzdGFuZCBhbmQgZGVzY3JpYmUgb3ZlciBhIG1vcmUgYWNjdXJhdGUgKGJ1dCBtb3JlIGNvbXBsaWNhdGVkKSBtb2RlbC4NCjwvZm9udD4NCg0KPGJyPjxicj4NCg0KLSAtIC0NCg0KPGJyPjxicj4NCg0KIyMjIDxhIGlkPSJuMyI+PC9hPiA8Yj4zLiDpmqjmqZ/mo67mnpfliIbmnpAoUmFuZG9tIEZvcmVzdCk8L2I+DQoNCjxicj4NCg0KIyMjIyA8Yj4zLjEgQnVpbGQgdGhlIE1vZGVsPC9iPg0K4oC7IOmAmuW4uOmghuW6j++8muW7uueri+mCj+i8r+W8j+i/tOatuOaooeWei+KGkiBDQVJU5qih5Z6L4oaSIOmaqOapn+ajruael+aooeWei+KGkiDkuqTlj4npqZforYnvvIhDQVJUMu+8ieKGkiDmr5TovIPkuIrov7DmqKHlnovkuKbpgbjmk4flhbbkuK3kuYvkuIDjgIINCmBgYHtyfQ0KbGlicmFyeShyYW5kb21Gb3Jlc3QpDQpGb3Jlc3QgPSByYW5kb21Gb3Jlc3QoWX4geDEreDIrLi4uLCBkYXRhID0gVFIsIG5vZGVzaXplPTI1LCBudHJlZT0gMjAwKQ0KcHJlZEZvcmVzdCA9IHByZWRpY3QoRm9yZXN0LCBuZXdkYXRhID0gVFMpDQp0YWJsZShUUyRZLHByZWRGb3Jlc3QpDQpBQ0MgPSAoVE4rVFApLyhUTitGUCtGTitUUCkNCmBgYA0KPGJyPg0K44CQQVVD5ZKMQUND57K+57Ch5YWs5byP44CRDQpgYGB7cn0NCmNvbEFVQyhwcmVkLCBUUyRhKQ0KdGFibGUoVFMkYSwgcHJlZCA+IDAuNSkgJT4lIHtzdW0oZGlhZyguKSkvc3VtKC4pfSANCmBgYA0KDQo8YnI+PGJyPg0KDQotIC0gLQ0KDQo8YnI+PGJyPg0KDQojIyMgPGEgaWQ9Im40Ij48L2E+IDxiPjQuIOe1hOWQiOaooeWeiyhFbnNhbWJsaW5nKTwvYj4NCg0KPGJyPg0KDQojIyMjIDxiPjQuMSBDb21wYXJlIEFDQzwvYj4NCg0KYGBge3J9DQpweCA9IGNiaW5kKGdsbT1wcmVkZ2xtLCBjYXJ0MT1wcmVkMSwgY2FydDI9cHJlZDIsIGNhcnQzPXByZWQzLCBjYXJ0NCA9IHByZWQ0LCBjYXJ0NSA9IHByZWQ1LCBjYXJ0NiA9IHByZWQ2LCBjYXJ0NyA9IHByZWQ3LCByZiA9IFByZWRpY3RGb3Jlc3QpDQphcHBseShweCwgMiwgZnVuY3Rpb24oeCkgew0KICB0YWJsZShUUyRidXksIHggPiAwLjUpICU+JSB7c3VtKGRpYWcoLikpL3N1bSguKX0gDQogIH0pICU+JSBzb3J0DQpgYGANCg0KDQo8YnI+DQoNCiMjIyMgPGI+NC4yIENvbXBhcmUgQVVDPC9iPg0KDQpgYGB7cn0NCnBhcihjZXg9MC43LG1hcj1jKDYsNiw1LDMpKQ0KY29sQVVDKHB4LCBUUyRidXksIFQpDQpgYGANCjxicj4NCg0KIyMjIyA8Yj40LjMgQ29ycmVsYXRpb24gJiBFbnNhbWJsaW5nPC9iPg0KDQorIOWFqeWAi+ebuOmXnOaAp+mrmOeahOaooeWei+S4jeimi+W+l+aYr+acgOWlveeahOe1hOWQiA0KKyDlhanlgIvmpbXnq6/nm7jlj43nmoTmqKHlnovntYTlkIjotbfkvoblj43ogIzog73luavlv5nop6Pph4vmm7TlpJrkuI3lkIznmoTop5LluqYNCg0KYGBge3J9DQpjb3IocHgpDQpnbG1fY2FydCA9IChweFssImdsbSJdICsgcHhbLCJjYXJ0NiJdKS8yDQpnbG1fcmYgPSAocHhbLCJnbG0iXSArIHB4WywicmYiXSkvMg0KYGBgDQo8YnI+DQoNCiMjIyMgPGI+NC40IEFDQyAmIEFVQyBUYWJsZTwvYj4NCg0KYGBge3J9DQpweDIgPSBjYmluZChweCwgZ2xtX2NhcnQsIGdsbV9yZikNCnJiaW5kKGFwcGx5KHB4MiwgMiwgZnVuY3Rpb24oeCkgew0KICAgICAgICB0YWJsZShUUyRidXksIHggPiAwLjUpICU+JSB7c3VtKGRpYWcoLikpL3N1bSguKX0gfSksDQogICAgICBjb2xBVUMocHgyLCBUUyRidXkpKSAlPiUgdCAlPiUgDQogICAgICBkYXRhLmZyYW1lICU+JSBzZXROYW1lcyhjKCJBY2N1cmFjeSIsICJBVUMiKSkgICAgICAgIA0KcHgzID0gY2JpbmQobG09UlNxdWFyZWxtLCBjYXJ0OT1SU3F1YXJlcnApDQpyYmluZChweDMpICU+JSBkYXRhLmZyYW1lICU+JSBzZXROYW1lcyhjKCJsbSIsImNhcnQiKSkNCmBgYA0K