MLM Case study X8
Dealing with imbalanced data in h2o
Oleg Baydakov
December 02, 2017
1 Introduction
1.1 Objective
The main objective of this case study X8 is to evaluate the effect of different fold_assignment and balance classes settings on the performance of Random Forest algorithm applied to a binary classfication problem. Our study will imply the Thoracic surgery dataset (Wroclaw Medical University, Poland). This dataset was collected retrospectively at Wroclaw Thoracic Surgery Centre for patients who underwent major lung resections for primary lung cancer in the period of 2007 to 2011. Our classfication task aims to make 1 year Survival prognosis of those patients, based on their preoperative physical and functional characteristics.
1.2 Materials and method
First, we will prepare the ggplot theme for our experiment
library(tidyverse)
my_theme <- function(base_size = 10, base_family = "sans"){
theme_minimal(base_size = base_size, base_family = base_family) +
theme(
axis.text = element_text(size = 10),
axis.text.x = element_text(angle = 0, vjust = 0.5, hjust = 0.5),
axis.title = element_text(size = 12),
panel.grid.major = element_line(color = "grey"),
panel.grid.minor = element_blank(),
panel.background = element_rect(fill = "#ffffef"),
strip.background = element_rect(fill = "#ffbb00", color = "black", size =0.5),
strip.text = element_text(face = "bold", size = 10, color = "black"),
legend.position = "bottom",
legend.justification = "center",
legend.background = element_blank(),
panel.border = element_rect(color = "grey30", fill = NA, size = 0.5)
)
}
theme_set(my_theme())
mycolors=c("#f32440","#ffd700","#ff8c00","#c9e101","#c100e6","#39d3d6","#e84412")1.3 Data load
Now we load the dataset from UCI website and perform a descriptive analysis
require(foreign)
df=read.arff("https://archive.ics.uci.edu/ml/machine-learning-databases/00277/ThoraricSurgery.arff")%>%as_tibble()
names(df)=c("Diagnosis","FVC","FEV1","Zubrod","Pain","Haemoptysis","Dyspnoea","Cough","Weakness","T_grade","DBtype2","MI","PAD","Smoking","Asthma","Age","Survival")
df$Survival=df$Survival%>%recode_factor(.,`F` = "Survived", `T` = "Dead")
df$Tiffneau=df$FEV1/df$FVC
Hmisc::describe(df)## df
##
## 18 Variables 470 Observations
## ---------------------------------------------------------------------------
## Diagnosis
## n missing distinct
## 470 0 7
##
## Value DGN1 DGN2 DGN3 DGN4 DGN5 DGN6 DGN8
## Frequency 1 52 349 47 15 4 2
## Proportion 0.002 0.111 0.743 0.100 0.032 0.009 0.004
## ---------------------------------------------------------------------------
## FVC
## n missing distinct Info Mean Gmd .05 .10
## 470 0 134 1 3.282 0.9818 2.018 2.316
## .25 .50 .75 .90 .95
## 2.600 3.160 3.808 4.560 4.900
##
## lowest : 1.44 1.46 1.70 1.81 1.82, highest: 5.52 5.56 5.60 6.08 6.30
## ---------------------------------------------------------------------------
## FEV1
## n missing distinct Info Mean Gmd .05 .10
## 470 0 136 1 4.569 4.805 1.440 1.640
## .25 .50 .75 .90 .95
## 1.960 2.400 3.080 3.762 4.311
##
## Value 1 2 3 4 5 9 52 61 64 66
## Frequency 27 224 151 47 6 1 1 1 1 1
## Proportion 0.057 0.477 0.321 0.100 0.013 0.002 0.002 0.002 0.002 0.002
##
## Value 67 69 71 73 76 77 78 79 86
## Frequency 1 1 1 2 1 1 1 1 1
## Proportion 0.002 0.002 0.002 0.004 0.002 0.002 0.002 0.002 0.002
## ---------------------------------------------------------------------------
## Zubrod
## n missing distinct
## 470 0 3
##
## Value PRZ0 PRZ1 PRZ2
## Frequency 130 313 27
## Proportion 0.277 0.666 0.057
## ---------------------------------------------------------------------------
## Pain
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 439 31
## Proportion 0.934 0.066
## ---------------------------------------------------------------------------
## Haemoptysis
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 402 68
## Proportion 0.855 0.145
## ---------------------------------------------------------------------------
## Dyspnoea
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 439 31
## Proportion 0.934 0.066
## ---------------------------------------------------------------------------
## Cough
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 147 323
## Proportion 0.313 0.687
## ---------------------------------------------------------------------------
## Weakness
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 392 78
## Proportion 0.834 0.166
## ---------------------------------------------------------------------------
## T_grade
## n missing distinct
## 470 0 4
##
## Value OC11 OC12 OC13 OC14
## Frequency 177 257 19 17
## Proportion 0.377 0.547 0.040 0.036
## ---------------------------------------------------------------------------
## DBtype2
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 435 35
## Proportion 0.926 0.074
## ---------------------------------------------------------------------------
## MI
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 468 2
## Proportion 0.996 0.004
## ---------------------------------------------------------------------------
## PAD
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 462 8
## Proportion 0.983 0.017
## ---------------------------------------------------------------------------
## Smoking
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 84 386
## Proportion 0.179 0.821
## ---------------------------------------------------------------------------
## Asthma
## n missing distinct
## 470 0 2
##
## Value F T
## Frequency 468 2
## Proportion 0.996 0.004
## ---------------------------------------------------------------------------
## Age
## n missing distinct Info Mean Gmd .05 .10
## 470 0 45 0.998 62.53 9.737 49.45 52.00
## .25 .50 .75 .90 .95
## 57.00 62.00 69.00 74.00 77.00
##
## lowest : 21 37 38 39 40, highest: 78 79 80 81 87
## ---------------------------------------------------------------------------
## Survival
## n missing distinct
## 470 0 2
##
## Value Survived Dead
## Frequency 400 70
## Proportion 0.851 0.149
## ---------------------------------------------------------------------------
## Tiffneau
## n missing distinct Info Mean Gmd .05 .10
## 470 0 406 1 1.47 1.484 0.5806 0.6375
## .25 .50 .75 .90 .95
## 0.7134 0.7738 0.8307 0.8909 0.9859
##
## Value 0.5 1.0 3.0 16.0 17.5 21.0 23.5 24.0 25.0 25.5
## Frequency 181 274 1 3 1 2 1 1 1 1
## Proportion 0.385 0.583 0.002 0.006 0.002 0.004 0.002 0.002 0.002 0.002
##
## Value 26.0 28.5 33.0 47.5
## Frequency 1 1 1 1
## Proportion 0.002 0.002 0.002 0.002
## ---------------------------------------------------------------------------The predictors consist of 16 variables:
ICD-10 codes for primary and secondary as well multiple tumours if any; Spirometric values (FEV1, FVC and Tiffneau index= FEV1/FVC), Performance status on Zubrod scale, Pain before surgery (T,F), Haemoptysis before surgery (T,F), Dyspnoea before surgery (T,F), Cough before surgery (T,F), Weakness before surgery (T,F), T in clinical TNM - size of the original tumour, from OC11 (smallest) to OC14 (largest), Type 2 diabetes mellitus (T,F), MI up to 6 months (T,F), peripheral arterial diseases (T,F), Smoking (T,F), Asthma (T,F) and Age at surgery (numeric).
The target outcome is highly imbalanced with a ratio of 400 survived / 70 dead (about 1:6)
1.4 Data visualising
library(gridExtra)
a1=df%>%ggplot(aes(x=Survival,fill=Diagnosis))+geom_bar(position="fill",color="black",alpha=0.8,show.legend = T)+scale_fill_manual(values=mycolors)+coord_flip()+ggtitle("Diagnosis")
a2=df%>%ggplot(aes(x=Diagnosis,y=..count..,fill=Diagnosis))+geom_bar(color="black",alpha=0.8,show.legend =F)+scale_fill_manual(values=mycolors)+coord_flip()+facet_grid(Survival~.)
grid.arrange(a1,a2,ncol=1)
b1=df%>%ggplot(aes(x=Survival,fill=T_grade))+geom_bar(position="fill",color="black",alpha=0.8,show.legend = T)+scale_fill_manual(values=mycolors)+coord_flip()+ggtitle("T_grade")
b2=df%>%ggplot(aes(x=T_grade,y=..count..,fill=T_grade))+geom_bar(color="black",alpha=0.8,show.legend =F)+scale_fill_manual(values=mycolors)+coord_flip()+facet_grid(Survival~.)
grid.arrange(b1,b2,ncol=1)
df%>%gather(Pain:Weakness,DBtype2:Asthma,key="Features",value="Value")%>%ggplot(aes(x=Survival,y=..count..,fill=Value))+geom_bar(alpha=0.8,color="black")+facet_wrap(~Features,ncol=5)+scale_fill_manual(values=mycolors)
df%>%gather(Age,FEV1,FVC,Tiffneau,key="Features",value="Value")%>%ggplot(aes(x=Survival,y=Value,fill=Survival))+geom_boxplot(alpha=0.8,color="black")+coord_flip()+facet_wrap(~Features,ncol=1,scales="free")+scale_fill_manual(values=mycolors)
df%>%gather(Age,FEV1,FVC,Tiffneau,key="Features",value="Value")%>%ggplot(aes(x=Value,fill=Survival))+geom_density(alpha=0.6,color="black")+facet_wrap(~Features,ncol=2,scales="free")+scale_fill_manual(values=mycolors)
2 Machine learning experiment
The first step consists of initialising our h2o package in R. The caret package will be used for data splitting, as it’s the only package that warrant the identical proportion of classes across the training and testing subset. By keeping the same proportion of target outcome, we hope that imbalanced data would not effect the validation of trained model.
library(h2o)
h2o.init(nthreads = -1,max_mem_size ="4g")##
## H2O is not running yet, starting it now...
##
## Note: In case of errors look at the following log files:
## /var/folders/5p/hr5_8rfs2h30_vbr9xwbvpg40000gn/T//RtmpgthBoE/h2o_olegbaydakov_started_from_r.out
## /var/folders/5p/hr5_8rfs2h30_vbr9xwbvpg40000gn/T//RtmpgthBoE/h2o_olegbaydakov_started_from_r.err
##
##
## Starting H2O JVM and connecting: .. Connection successful!
##
## R is connected to the H2O cluster:
## H2O cluster uptime: 2 seconds 298 milliseconds
## H2O cluster version: 3.16.0.1
## H2O cluster version age: 8 days
## H2O cluster name: H2O_started_from_R_olegbaydakov_tau002
## H2O cluster total nodes: 1
## H2O cluster total memory: 3.56 GB
## H2O cluster total cores: 8
## H2O cluster allowed cores: 8
## H2O cluster healthy: TRUE
## H2O Connection ip: localhost
## H2O Connection port: 54321
## H2O Connection proxy: NA
## H2O Internal Security: FALSE
## H2O API Extensions: XGBoost, Algos, AutoML, Core V3, Core V4
## R Version: R version 3.4.1 (2017-06-30)library(caret)
set.seed(123)
idTrain=caret::createDataPartition(y=df$Survival,p=369/470,list=FALSE)
trainset=df[idTrain,]
testset=df[-idTrain,]
sp1=df%>%ggplot(aes(x=Survival,fill=Survival))+stat_count(color="black",alpha=0.7,show.legend = F)+scale_fill_manual(values=c("#f32440","#ffd700"))+coord_flip()+ggtitle("Origin")
sp2=trainset%>%ggplot(aes(x=Survival,fill=Survival))+stat_count(color="black",alpha=0.7,show.legend = F)+scale_fill_manual(values=c("#f32440","#ffd700"))+coord_flip()+ggtitle("Train")
sp3=testset%>%ggplot(aes(x=Survival,fill=Survival))+stat_count(color="black",alpha=0.7,show.legend = F)+scale_fill_manual(values=c("#f32440","#ffd700"))+coord_flip()+ggtitle("Test")
grid.arrange(sp1,sp2,sp3,ncol=1)
wtrain=as.h2o(trainset)##
|
| | 0%
|
|=================================================================| 100%wtest=as.h2o(testset)##
|
| | 0%
|
|=================================================================| 100%response="Survival"
features=setdiff(colnames(wtrain),response)Our experiment consists of evaluating the effect of 4 possible setting combinations between the Fold assignment and Balance classes parameters:
Model RF with Randomised fold assignment without classes balancing
Model RF with Randomised fold assignment with classes balancing
Model RF with Stratified fold assignment without classes balancing
Model RF with Stratified fold assignment with classes balancing
#RF learner
#Balanced + stratified
rfmod1=h2o.randomForest(x = features,
y = response,
training_frame = wtrain,nfolds=10,
fold_assignment = "Stratified",
balance_classes = TRUE,class_sampling_factors=c(1.17,6.66),
ntrees = 100, max_depth = 50,
stopping_metric = "logloss",
stopping_tolerance = 0.01,
stopping_rounds = 3,
keep_cross_validation_fold_assignment = TRUE,
keep_cross_validation_predictions=TRUE,
score_each_iteration = TRUE,
seed=12345)##
|
| | 0%
|
|= | 2%
|
|==== | 6%
|
|=========================================================== | 91%
|
|=================================================================| 100%#Balanced + Not stratified
rfmod2=h2o.randomForest(x = features,
y = response,
training_frame = wtrain,nfolds=10,
fold_assignment = "AUTO",
balance_classes = TRUE,class_sampling_factors=c(1.17,6.66),
ntrees = 100, max_depth = 50,
stopping_metric = "logloss",
stopping_tolerance = 0.01,
stopping_rounds = 3,
keep_cross_validation_fold_assignment = TRUE,
keep_cross_validation_predictions=TRUE,
score_each_iteration = TRUE,
seed=12345)##
|
| | 0%
|
|=================================================================| 100%#Unbalanced + stratified
rfmod3=h2o.randomForest(x = features,
y = response,
training_frame = wtrain,nfolds=10,
fold_assignment = "Stratified",
balance_classes = FALSE,
ntrees = 100, max_depth = 50,
stopping_metric = "logloss",
stopping_tolerance = 0.01,
stopping_rounds = 3,
keep_cross_validation_fold_assignment = TRUE,
keep_cross_validation_predictions=TRUE,
score_each_iteration = TRUE,
seed=12345)##
|
| | 0%
|
|= | 1%
|
|=================================================================| 100%#Unbalanced + not stratified
rfmod0=h2o.randomForest(x = features,
y = response,
training_frame = wtrain,nfolds=10,
balance_classes = F,
ntrees = 100, max_depth = 50,
stopping_metric = "logloss",
stopping_tolerance = 0.01,
stopping_rounds = 3,
keep_cross_validation_fold_assignment = TRUE,
keep_cross_validation_predictions=TRUE,
score_each_iteration = TRUE,
seed=12345)##
|
| | 0%
|
|= | 2%
|
|=================================================================| 100%3 Confusion matrix and performance of 4 models on tests subset
h2o.performance(rfmod0,wtest)## H2OBinomialMetrics: drf
##
## MSE: 0.1314281
## RMSE: 0.3625301
## LogLoss: 1.035559
## Mean Per-Class Error: 0.4666667
## AUC: 0.647451
## Gini: 0.294902
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## Dead Survived Error Rate
## Dead 1 14 0.933333 =14/15
## Survived 0 85 0.000000 =0/85
## Totals 1 99 0.140000 =14/100
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.510417 0.923913 61
## 2 max f2 0.510417 0.968109 61
## 3 max f0point5 0.510417 0.883576 61
## 4 max accuracy 0.510417 0.860000 61
## 5 max precision 0.913889 0.954545 20
## 6 max recall 0.510417 1.000000 61
## 7 max specificity 1.000000 0.866667 0
## 8 max absolute_mcc 0.862500 0.285831 29
## 9 max min_per_class_accuracy 0.862500 0.658824 29
## 10 max mean_per_class_accuracy 0.862500 0.696078 29
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`h2o.performance(rfmod1,wtest)## H2OBinomialMetrics: drf
##
## MSE: 0.1830306
## RMSE: 0.4278207
## LogLoss: 1.241207
## Mean Per-Class Error: 0.4666667
## AUC: 0.5968627
## Gini: 0.1937255
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## Dead Survived Error Rate
## Dead 1 14 0.933333 =14/15
## Survived 0 85 0.000000 =0/85
## Totals 1 99 0.140000 =14/100
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.080790 0.923913 52
## 2 max f2 0.080790 0.968109 52
## 3 max f0point5 0.462663 0.890736 39
## 4 max accuracy 0.080790 0.860000 52
## 5 max precision 0.584416 0.907895 37
## 6 max recall 0.080790 1.000000 52
## 7 max specificity 1.000000 0.866667 0
## 8 max absolute_mcc 0.584416 0.288526 37
## 9 max min_per_class_accuracy 0.659121 0.533333 23
## 10 max mean_per_class_accuracy 0.584416 0.672549 37
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`h2o.performance(rfmod2,wtest)## H2OBinomialMetrics: drf
##
## MSE: 0.1830306
## RMSE: 0.4278207
## LogLoss: 1.241207
## Mean Per-Class Error: 0.4666667
## AUC: 0.5968627
## Gini: 0.1937255
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## Dead Survived Error Rate
## Dead 1 14 0.933333 =14/15
## Survived 0 85 0.000000 =0/85
## Totals 1 99 0.140000 =14/100
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.080790 0.923913 52
## 2 max f2 0.080790 0.968109 52
## 3 max f0point5 0.462663 0.890736 39
## 4 max accuracy 0.080790 0.860000 52
## 5 max precision 0.584416 0.907895 37
## 6 max recall 0.080790 1.000000 52
## 7 max specificity 1.000000 0.866667 0
## 8 max absolute_mcc 0.584416 0.288526 37
## 9 max min_per_class_accuracy 0.659121 0.533333 23
## 10 max mean_per_class_accuracy 0.584416 0.672549 37
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`h2o.performance(rfmod3,wtest)## H2OBinomialMetrics: drf
##
## MSE: 0.1328032
## RMSE: 0.3644218
## LogLoss: 1.031488
## Mean Per-Class Error: 0.5
## AUC: 0.6631373
## Gini: 0.3262745
##
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
## Dead Survived Error Rate
## Dead 0 15 1.000000 =15/15
## Survived 0 85 0.000000 =0/85
## Totals 0 100 0.150000 =15/100
##
## Maximum Metrics: Maximum metrics at their respective thresholds
## metric threshold value idx
## 1 max f1 0.450000 0.918919 51
## 2 max f2 0.450000 0.965909 51
## 3 max f0point5 0.450000 0.876289 51
## 4 max accuracy 0.450000 0.850000 51
## 5 max precision 0.907407 0.960784 16
## 6 max recall 0.450000 1.000000 51
## 7 max specificity 1.000000 0.866667 0
## 8 max absolute_mcc 0.907407 0.316527 16
## 9 max min_per_class_accuracy 0.861111 0.658824 24
## 10 max mean_per_class_accuracy 0.907407 0.721569 16
##
## Gains/Lift Table: Extract with `h2o.gainsLift(<model>, <data>)` or `h2o.gainsLift(<model>, valid=<T/F>, xval=<T/F>)`Suppose that “Survived” is the negative and “Dead” is positive outcomes (as we try to identify the patients with higher risk of post-operative mortality): It seems that all 4 models have the same performance on test subset: they provided correct survival prognosis, but all failed to identify Mortality risk. Each model maded 14 to 15 mistakes (all of them were false negatives). We don’t know whether our settings could have any effect on each one of these 4 models or not ?.
To find the answer for this question, we must resample the validation on replicated dataset by bootstrapping
4 MLR package - resample the validation on replicated dataset by bootstrapping
First, we call up the mlr package, and train a dummy h2o random forest model in h2o. The real purpose of this dummy model is for cloning exactly the 4 models trained in h2o. This trick has been well explained in the previous tutorials.
library(mlr)## Loading required package: ParamHelpers##
## Attaching package: 'mlr'## The following object is masked from 'package:caret':
##
## traintaskTS=mlr::makeClassifTask(id="Thorac",data=df,target="Survival",positive = "Dead")
learnerH2ORF=makeLearner(id="h2oRF","classif.h2o.randomForest", predict.type = "prob")
mlrRF=train(learner = learnerH2ORF, task=taskTS)##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|============ | 18%
|
|=================================================================| 100%Neither accuracy nor absolute misclassification rate are right metric to use when confronting the imbalanced data. They could be misleading, as mentioned above. Other metrics are more appropriate, including True/False Negative or Positive rate, Balanced accuracy or ballanced error rate, Recall (or specificity), Cohen’s Kappa coefficient or F1 score. We will adopt some of those metrics for out benchmark study.
bootmlrPERF=function(h2omodel,data,i){
d=data[i,]
predmlr=predict(mlrRF,newdata=d)
predh2o=predict(get(h2omodel),as.h2o(d))
predmlr$data$response<-as.vector(predh2o$predict)
predmlr$data$prob.Dead<-as.vector(predh2o$Dead)
predmlr$data$prob.Dead<-as.vector(predh2o$Survived)
mets=list(bac,f1,tpr,tnr,fpr,fnr)
p=mlr::performance(predmlr,mets)
BAC=p[[1]]
F1=p[[2]]
TPR=p[[3]]
TNR=p[[4]]
FPR=p[[5]]
FNR=p[[6]]
return=cbind(BAC,F1,TPR,TNR,FPR,FNR)
}
set.seed(123)
library(boot)
perfmod0=boot(statistic=bootmlrPERF,h2omodel="rfmod0",data=df,R=2)%>%.$t%>%as_tibble()%>%mutate(Mode="Unbalanced_Random",iter=as.numeric(rownames(.)))##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%perfmod1=boot(statistic=bootmlrPERF,h2omodel="rfmod1",data=df,R=2)%>%.$t%>%as_tibble()%>%mutate(Mode="Balanced_Stratified",iter=as.numeric(rownames(.)))##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%perfmod2=boot(statistic=bootmlrPERF,h2omodel="rfmod2",data=df,R=2)%>%.$t%>%as_tibble()%>%mutate(Mode="Balanced_Random",iter=as.numeric(rownames(.)))##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%perfmod3=boot(statistic=bootmlrPERF,h2omodel="rfmod3",data=df,R=2)%>%.$t%>%as_tibble()%>%mutate(Mode="Unbalanced_Stratified",iter=as.numeric(rownames(.)))##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%
##
|
| | 0%
|
|=================================================================| 100%bootperf=rbind(perfmod0,perfmod1,perfmod2,perfmod3)
names(bootperf)=c("BAC","F1","TPR","TNR","FPR","FNR","Mode","Iteration")
bootperf[,c(1:6)]%>%psych::describeBy(.,bootperf$Mode)##
## Descriptive statistics by group
## group: Balanced_Random
## vars n mean sd median trimmed mad min max range skew kurtosis
## BAC 1 2 0.92 0.03 0.92 0.92 0.04 0.90 0.94 0.05 0 -2.75
## F1 2 2 0.84 0.05 0.84 0.84 0.05 0.81 0.88 0.07 0 -2.75
## TPR 3 2 0.88 0.07 0.88 0.88 0.07 0.83 0.93 0.10 0 -2.75
## TNR 4 2 0.96 0.00 0.96 0.96 0.00 0.96 0.96 0.00 0 -2.75
## FPR 5 2 0.04 0.00 0.04 0.04 0.00 0.04 0.04 0.00 0 -2.75
## FNR 6 2 0.12 0.07 0.12 0.12 0.07 0.07 0.17 0.10 0 -2.75
## se
## BAC 0.02
## F1 0.04
## TPR 0.05
## TNR 0.00
## FPR 0.00
## FNR 0.05
## --------------------------------------------------------
## group: Balanced_Stratified
## vars n mean sd median trimmed mad min max range skew kurtosis
## BAC 1 2 0.91 0.06 0.91 0.91 0.07 0.87 0.96 0.09 0 -2.75
## F1 2 2 0.81 0.09 0.81 0.81 0.09 0.75 0.87 0.13 0 -2.75
## TPR 3 2 0.87 0.13 0.87 0.87 0.13 0.78 0.96 0.18 0 -2.75
## TNR 4 2 0.96 0.00 0.96 0.96 0.00 0.96 0.96 0.00 0 -2.75
## FPR 5 2 0.04 0.00 0.04 0.04 0.00 0.04 0.04 0.00 0 -2.75
## FNR 6 2 0.13 0.13 0.13 0.13 0.13 0.04 0.22 0.18 0 -2.75
## se
## BAC 0.05
## F1 0.06
## TPR 0.09
## TNR 0.00
## FPR 0.00
## FNR 0.09
## --------------------------------------------------------
## group: Unbalanced_Random
## vars n mean sd median trimmed mad min max range skew kurtosis se
## BAC 1 2 0.5 0 0.5 0.5 0 0.5 0.5 0 NaN NaN 0
## F1 2 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## TPR 3 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## TNR 4 2 1.0 0 1.0 1.0 0 1.0 1.0 0 NaN NaN 0
## FPR 5 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## FNR 6 2 1.0 0 1.0 1.0 0 1.0 1.0 0 NaN NaN 0
## --------------------------------------------------------
## group: Unbalanced_Stratified
## vars n mean sd median trimmed mad min max range skew kurtosis se
## BAC 1 2 0.5 0 0.5 0.5 0 0.5 0.5 0 NaN NaN 0
## F1 2 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## TPR 3 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## TNR 4 2 1.0 0 1.0 1.0 0 1.0 1.0 0 NaN NaN 0
## FPR 5 2 0.0 0 0.0 0.0 0 0.0 0.0 0 NaN NaN 0
## FNR 6 2 1.0 0 1.0 1.0 0 1.0 1.0 0 NaN NaN 0pairwise.wilcox.test(x=bootperf$FNR,g=bootperf$Mode,p.adjust.method="bonferroni",n=6,paired=T)## Warning in wilcox.test.default(xi, xj, paired = paired, ...): cannot
## compute exact p-value with zeroes##
## Pairwise comparisons using Wilcoxon signed rank test
##
## data: bootperf$FNR and bootperf$Mode
##
## Balanced_Random Balanced_Stratified
## Balanced_Stratified 1 -
## Unbalanced_Random 1 1
## Unbalanced_Stratified 1 1
## Unbalanced_Random
## Balanced_Stratified -
## Unbalanced_Random -
## Unbalanced_Stratified -
##
## P value adjustment method: bonferronipairwise.wilcox.test(x=bootperf$FPR,g=bootperf$Mode,p.adjust.method="bonferroni",n=6,paired=T)## Warning in wilcox.test.default(xi, xj, paired = paired, ...): cannot
## compute exact p-value with zeroes##
## Pairwise comparisons using Wilcoxon signed rank test
##
## data: bootperf$FPR and bootperf$Mode
##
## Balanced_Random Balanced_Stratified
## Balanced_Stratified 1 -
## Unbalanced_Random 1 1
## Unbalanced_Stratified 1 1
## Unbalanced_Random
## Balanced_Stratified -
## Unbalanced_Random -
## Unbalanced_Stratified -
##
## P value adjustment method: bonferronibootlong=bootperf%>%gather(BAC:FNR,key="Metric",value="Score")
bootlong%>%ggplot(aes(x=Score,fill=Mode))+geom_histogram(alpha=0.6,color="black")+facet_grid(Mode~Metric,scales="free")+scale_fill_manual(values=mycolors)## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
bootlong%>%ggplot(aes(x=Metric,y=Score,fill=Mode))+geom_boxplot(alpha=0.6)+facet_wrap(~Metric,scales="free",ncol=2)+scale_fill_manual(values=mycolors)+coord_flip()
bootperf%>%ggplot(aes(x=Iteration,y=F1,color=Mode,fill=Mode))+geom_path(alpha=0.8,size=1.2)+geom_point(shape=21,size=4,color="black")+scale_color_manual(values=c("red4","purple","orange","blue"))+scale_fill_manual(values=c("red","purple1","gold","skyblue"))+geom_hline(yintercept=0.95,linetype=2,size=1)+facet_wrap(~Mode,ncol=1)
bootperf%>%ggplot(aes(x=Iteration,y=BAC,color=Mode,fill=Mode))+geom_path(alpha=0.8,size=1.2)+geom_point(shape=21,size=4,color="black")+scale_color_manual(values=c("red4","purple","orange","blue"))+scale_fill_manual(values=c("red","purple1","gold","skyblue"))+geom_hline(yintercept=0.9,linetype=2,size=1)+facet_wrap(~Mode,ncol=1)
bootperf%>%ggplot(aes(x=Iteration,y=FNR,color=Mode,fill=Mode))+geom_path(alpha=0.8,size=1.2)+geom_point(shape=21,size=4,color="black")+scale_color_manual(values=c("red4","purple","orange","blue"))+scale_fill_manual(values=c("red","purple1","gold","skyblue"))+geom_hline(yintercept=0.1,linetype=2,size=1)+facet_wrap(~Mode,ncol=1)
The bootstraped validation on 30 random testing subsets revealed something more interesting:
The model with stand-alone classes weighting or combined with stratified fold assignment produced the best performance (in red and yellow). Those two models had the best F1 score, highest True Positive rate and lowest false negative rate. Only model trained by such protocole can make sense when applied to random imbalanced validation sets.
The difference in False negative rate between Random/Balanced model and Random/Unbalanced model is significative (Wilcoxon test p value =0.0000091).There was no difference between Balanced/Random assignment and Balanced/stratified models. This indicates that applying Balance class parameter alone (without Fold assignment Stratification) can already improve the model’s performance on imbalanced data.Folds stratification might be not enough to resolve the data imbalance problem. Combining Classes balancing and Stratified assignment could optomise our model’s performance.
5 Conclusion
Classes imbalance is a common and frustrating problem in machine learning practice. Class balancing is an useful feature in h2o that could be used alone or in combination with Fold assignment Stratification for dealing with such problem.
LS0tCnRpdGxlOiA8Y2VudGVyPiBNTE0gQ2FzZSBzdHVkeSBYOCA8L2NlbnRlcj4Kc3VidGl0bGU6IDxjZW50ZXI+IERlYWxpbmcgd2l0aCBpbWJhbGFuY2VkIGRhdGEgaW4gaDJvIDwvY2VudGVyPgphdXRob3I6IDxjZW50ZXI+IE9sZWcgQmF5ZGFrb3YgPC9jZW50ZXI+CmRhdGU6IDxjZW50ZXI+IERlY2VtYmVyIDAyLCAyMDE3IDwvY2VudGVyPgpvdXRwdXQ6IAogIGh0bWxfZG9jdW1lbnQ6IAogICAgY29kZV9kb3dubG9hZDogeWVzCiAgICBjb2RlX2ZvbGRpbmc6IHNob3cKICAgIGRmX3ByaW50OiBrYWJsZQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMKICAgIHRoZW1lOiBmbGF0bHkKICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwotLS0KCjxjZW50ZXI+IVtdKGJhbGFuY2VkX2RhdGFfcmZfaDIwLnBuZyl7IHdpZHRoPTUwJX08L2NlbnRlcj4KCiMgSW50cm9kdWN0aW9uIHsudGFic2V0fQoKIyMgT2JqZWN0aXZlClRoZSBtYWluIG9iamVjdGl2ZSBvZiB0aGlzIGNhc2Ugc3R1ZHkgWDggaXMgdG8gZXZhbHVhdGUgdGhlIGVmZmVjdCBvZiBkaWZmZXJlbnQgZm9sZF9hc3NpZ25tZW50IGFuZCBiYWxhbmNlIGNsYXNzZXMgc2V0dGluZ3Mgb24gdGhlIHBlcmZvcm1hbmNlIG9mIFJhbmRvbSBGb3Jlc3QgYWxnb3JpdGhtIGFwcGxpZWQgdG8gYSBiaW5hcnkgY2xhc3NmaWNhdGlvbiBwcm9ibGVtLiBPdXIgc3R1ZHkgd2lsbCBpbXBseSB0aGUgVGhvcmFjaWMgc3VyZ2VyeSBkYXRhc2V0IChXcm9jbGF3IE1lZGljYWwgVW5pdmVyc2l0eSwgUG9sYW5kKS4gVGhpcyBkYXRhc2V0IHdhcyBjb2xsZWN0ZWQgcmV0cm9zcGVjdGl2ZWx5IGF0IFdyb2NsYXcgVGhvcmFjaWMgU3VyZ2VyeSBDZW50cmUgZm9yIHBhdGllbnRzIHdobyB1bmRlcndlbnQgbWFqb3IgbHVuZyByZXNlY3Rpb25zIGZvciBwcmltYXJ5IGx1bmcgY2FuY2VyIGluIHRoZSBwZXJpb2Qgb2YgMjAwNyB0byAyMDExLiBPdXIgY2xhc3NmaWNhdGlvbiB0YXNrIGFpbXMgdG8gbWFrZSAxIHllYXIgU3Vydml2YWwgcHJvZ25vc2lzIG9mIHRob3NlIHBhdGllbnRzLCBiYXNlZCBvbiB0aGVpciBwcmVvcGVyYXRpdmUgcGh5c2ljYWwgYW5kIGZ1bmN0aW9uYWwgY2hhcmFjdGVyaXN0aWNzLgoKIyMgTWF0ZXJpYWxzIGFuZCBtZXRob2QKRmlyc3QsIHdlIHdpbGwgcHJlcGFyZSB0aGUgZ2dwbG90IHRoZW1lIGZvciBvdXIgZXhwZXJpbWVudApgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKCm15X3RoZW1lIDwtIGZ1bmN0aW9uKGJhc2Vfc2l6ZSA9IDEwLCBiYXNlX2ZhbWlseSA9ICJzYW5zIil7CiAgdGhlbWVfbWluaW1hbChiYXNlX3NpemUgPSBiYXNlX3NpemUsIGJhc2VfZmFtaWx5ID0gYmFzZV9mYW1pbHkpICsKICAgIHRoZW1lKAogICAgICBheGlzLnRleHQgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDEwKSwKICAgICAgYXhpcy50ZXh0LnggPSBlbGVtZW50X3RleHQoYW5nbGUgPSAwLCB2anVzdCA9IDAuNSwgaGp1c3QgPSAwLjUpLAogICAgICBheGlzLnRpdGxlID0gZWxlbWVudF90ZXh0KHNpemUgPSAxMiksCiAgICAgIHBhbmVsLmdyaWQubWFqb3IgPSBlbGVtZW50X2xpbmUoY29sb3IgPSAiZ3JleSIpLAogICAgICBwYW5lbC5ncmlkLm1pbm9yID0gZWxlbWVudF9ibGFuaygpLAogICAgICBwYW5lbC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSAiI2ZmZmZlZiIpLAogICAgICBzdHJpcC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSAiI2ZmYmIwMCIsIGNvbG9yID0gImJsYWNrIiwgc2l6ZSA9MC41KSwKICAgICAgc3RyaXAudGV4dCA9IGVsZW1lbnRfdGV4dChmYWNlID0gImJvbGQiLCBzaXplID0gMTAsIGNvbG9yID0gImJsYWNrIiksCiAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgICBsZWdlbmQuanVzdGlmaWNhdGlvbiA9ICJjZW50ZXIiLAogICAgICBsZWdlbmQuYmFja2dyb3VuZCA9IGVsZW1lbnRfYmxhbmsoKSwKICAgICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9yZWN0KGNvbG9yID0gImdyZXkzMCIsIGZpbGwgPSBOQSwgc2l6ZSA9IDAuNSkKICAgICkKfQp0aGVtZV9zZXQobXlfdGhlbWUoKSkKCm15Y29sb3JzPWMoIiNmMzI0NDAiLCIjZmZkNzAwIiwiI2ZmOGMwMCIsIiNjOWUxMDEiLCIjYzEwMGU2IiwiIzM5ZDNkNiIsIiNlODQ0MTIiKQpgYGAKCiMjIERhdGEgbG9hZApOb3cgd2UgbG9hZCB0aGUgZGF0YXNldCBmcm9tIFVDSSB3ZWJzaXRlIGFuZCBwZXJmb3JtIGEgZGVzY3JpcHRpdmUgYW5hbHlzaXMKYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KcmVxdWlyZShmb3JlaWduKQpkZj1yZWFkLmFyZmYoImh0dHBzOi8vYXJjaGl2ZS5pY3MudWNpLmVkdS9tbC9tYWNoaW5lLWxlYXJuaW5nLWRhdGFiYXNlcy8wMDI3Ny9UaG9yYXJpY1N1cmdlcnkuYXJmZiIpJT4lYXNfdGliYmxlKCkKCm5hbWVzKGRmKT1jKCJEaWFnbm9zaXMiLCJGVkMiLCJGRVYxIiwiWnVicm9kIiwiUGFpbiIsIkhhZW1vcHR5c2lzIiwiRHlzcG5vZWEiLCJDb3VnaCIsIldlYWtuZXNzIiwiVF9ncmFkZSIsIkRCdHlwZTIiLCJNSSIsIlBBRCIsIlNtb2tpbmciLCJBc3RobWEiLCJBZ2UiLCJTdXJ2aXZhbCIpCgpkZiRTdXJ2aXZhbD1kZiRTdXJ2aXZhbCU+JXJlY29kZV9mYWN0b3IoLixgRmAgPSAiU3Vydml2ZWQiLCBgVGAgPSAiRGVhZCIpCgpkZiRUaWZmbmVhdT1kZiRGRVYxL2RmJEZWQwoKSG1pc2M6OmRlc2NyaWJlKGRmKQpgYGAKPC9icj4KClRoZSBwcmVkaWN0b3JzIGNvbnNpc3Qgb2YgMTYgdmFyaWFibGVzOgoKSUNELTEwIGNvZGVzIGZvciBwcmltYXJ5IGFuZCBzZWNvbmRhcnkgYXMgd2VsbCBtdWx0aXBsZSB0dW1vdXJzIGlmIGFueTsgU3Bpcm9tZXRyaWMgdmFsdWVzIChGRVYxLCBGVkMgYW5kIFRpZmZuZWF1IGluZGV4PSBGRVYxL0ZWQyksIFBlcmZvcm1hbmNlIHN0YXR1cyBvbiBadWJyb2Qgc2NhbGUsIFBhaW4gYmVmb3JlIHN1cmdlcnkgKFQsRiksIEhhZW1vcHR5c2lzIGJlZm9yZSBzdXJnZXJ5IChULEYpLCBEeXNwbm9lYSBiZWZvcmUgc3VyZ2VyeSAoVCxGKSwgQ291Z2ggYmVmb3JlIHN1cmdlcnkgKFQsRiksIFdlYWtuZXNzIGJlZm9yZSBzdXJnZXJ5IChULEYpLCBUIGluIGNsaW5pY2FsIFROTSAtIHNpemUgb2YgdGhlIG9yaWdpbmFsIHR1bW91ciwgZnJvbSBPQzExIChzbWFsbGVzdCkgdG8gT0MxNCAobGFyZ2VzdCksIFR5cGUgMiBkaWFiZXRlcyBtZWxsaXR1cyAoVCxGKSwgTUkgdXAgdG8gNiBtb250aHMgKFQsRiksIHBlcmlwaGVyYWwgYXJ0ZXJpYWwgZGlzZWFzZXMgKFQsRiksIFNtb2tpbmcgKFQsRiksIEFzdGhtYSAoVCxGKSBhbmQgQWdlIGF0IHN1cmdlcnkgKG51bWVyaWMpLgoKVGhlIHRhcmdldCBvdXRjb21lIGlzIGhpZ2hseSBpbWJhbGFuY2VkIHdpdGggYSByYXRpbyBvZiA0MDAgc3Vydml2ZWQgLyA3MCBkZWFkIChhYm91dCAxOjYpCgojIyBEYXRhIHZpc3VhbGlzaW5nCgpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KGdyaWRFeHRyYSkKCmExPWRmJT4lZ2dwbG90KGFlcyh4PVN1cnZpdmFsLGZpbGw9RGlhZ25vc2lzKSkrZ2VvbV9iYXIocG9zaXRpb249ImZpbGwiLGNvbG9yPSJibGFjayIsYWxwaGE9MC44LHNob3cubGVnZW5kID0gVCkrc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzPW15Y29sb3JzKStjb29yZF9mbGlwKCkrZ2d0aXRsZSgiRGlhZ25vc2lzIikKCmEyPWRmJT4lZ2dwbG90KGFlcyh4PURpYWdub3Npcyx5PS4uY291bnQuLixmaWxsPURpYWdub3NpcykpK2dlb21fYmFyKGNvbG9yPSJibGFjayIsYWxwaGE9MC44LHNob3cubGVnZW5kID1GKStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9bXljb2xvcnMpK2Nvb3JkX2ZsaXAoKStmYWNldF9ncmlkKFN1cnZpdmFsfi4pCgpncmlkLmFycmFuZ2UoYTEsYTIsbmNvbD0xKQpgYGAKYGBge3J9CmIxPWRmJT4lZ2dwbG90KGFlcyh4PVN1cnZpdmFsLGZpbGw9VF9ncmFkZSkpK2dlb21fYmFyKHBvc2l0aW9uPSJmaWxsIixjb2xvcj0iYmxhY2siLGFscGhhPTAuOCxzaG93LmxlZ2VuZCA9IFQpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1teWNvbG9ycykrY29vcmRfZmxpcCgpK2dndGl0bGUoIlRfZ3JhZGUiKQpiMj1kZiU+JWdncGxvdChhZXMoeD1UX2dyYWRlLHk9Li5jb3VudC4uLGZpbGw9VF9ncmFkZSkpK2dlb21fYmFyKGNvbG9yPSJibGFjayIsYWxwaGE9MC44LHNob3cubGVnZW5kID1GKStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9bXljb2xvcnMpK2Nvb3JkX2ZsaXAoKStmYWNldF9ncmlkKFN1cnZpdmFsfi4pCgpncmlkLmFycmFuZ2UoYjEsYjIsbmNvbD0xKQpgYGAKCmBgYHtyfQpkZiU+JWdhdGhlcihQYWluOldlYWtuZXNzLERCdHlwZTI6QXN0aG1hLGtleT0iRmVhdHVyZXMiLHZhbHVlPSJWYWx1ZSIpJT4lZ2dwbG90KGFlcyh4PVN1cnZpdmFsLHk9Li5jb3VudC4uLGZpbGw9VmFsdWUpKStnZW9tX2JhcihhbHBoYT0wLjgsY29sb3I9ImJsYWNrIikrZmFjZXRfd3JhcCh+RmVhdHVyZXMsbmNvbD01KStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9bXljb2xvcnMpCmBgYAoKYGBge3J9CmRmJT4lZ2F0aGVyKEFnZSxGRVYxLEZWQyxUaWZmbmVhdSxrZXk9IkZlYXR1cmVzIix2YWx1ZT0iVmFsdWUiKSU+JWdncGxvdChhZXMoeD1TdXJ2aXZhbCx5PVZhbHVlLGZpbGw9U3Vydml2YWwpKStnZW9tX2JveHBsb3QoYWxwaGE9MC44LGNvbG9yPSJibGFjayIpK2Nvb3JkX2ZsaXAoKStmYWNldF93cmFwKH5GZWF0dXJlcyxuY29sPTEsc2NhbGVzPSJmcmVlIikrc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzPW15Y29sb3JzKQpgYGAKCmBgYHtyfQpkZiU+JWdhdGhlcihBZ2UsRkVWMSxGVkMsVGlmZm5lYXUsa2V5PSJGZWF0dXJlcyIsdmFsdWU9IlZhbHVlIiklPiVnZ3Bsb3QoYWVzKHg9VmFsdWUsZmlsbD1TdXJ2aXZhbCkpK2dlb21fZGVuc2l0eShhbHBoYT0wLjYsY29sb3I9ImJsYWNrIikrZmFjZXRfd3JhcCh+RmVhdHVyZXMsbmNvbD0yLHNjYWxlcz0iZnJlZSIpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1teWNvbG9ycykKYGBgCgojIE1hY2hpbmUgbGVhcm5pbmcgZXhwZXJpbWVudAoKVGhlIGZpcnN0IHN0ZXAgY29uc2lzdHMgb2YgaW5pdGlhbGlzaW5nIG91ciBoMm8gcGFja2FnZSBpbiBSLiBUaGUgY2FyZXQgcGFja2FnZSB3aWxsIGJlIHVzZWQgZm9yIGRhdGEgc3BsaXR0aW5nLCBhcyBpdOKAmXMgdGhlIG9ubHkgcGFja2FnZSB0aGF0IHdhcnJhbnQgdGhlIGlkZW50aWNhbCBwcm9wb3J0aW9uIG9mIGNsYXNzZXMgYWNyb3NzIHRoZSB0cmFpbmluZyBhbmQgdGVzdGluZyBzdWJzZXQuIEJ5IGtlZXBpbmcgdGhlIHNhbWUgcHJvcG9ydGlvbiBvZiB0YXJnZXQgb3V0Y29tZSwgd2UgaG9wZSB0aGF0IGltYmFsYW5jZWQgZGF0YSB3b3VsZCBub3QgZWZmZWN0IHRoZSB2YWxpZGF0aW9uIG9mIHRyYWluZWQgbW9kZWwuCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CmxpYnJhcnkoaDJvKQoKaDJvLmluaXQobnRocmVhZHMgPSAtMSxtYXhfbWVtX3NpemUgPSI0ZyIpCgpsaWJyYXJ5KGNhcmV0KQpzZXQuc2VlZCgxMjMpCgppZFRyYWluPWNhcmV0OjpjcmVhdGVEYXRhUGFydGl0aW9uKHk9ZGYkU3Vydml2YWwscD0zNjkvNDcwLGxpc3Q9RkFMU0UpCnRyYWluc2V0PWRmW2lkVHJhaW4sXQp0ZXN0c2V0PWRmWy1pZFRyYWluLF0KCgpzcDE9ZGYlPiVnZ3Bsb3QoYWVzKHg9U3Vydml2YWwsZmlsbD1TdXJ2aXZhbCkpK3N0YXRfY291bnQoY29sb3I9ImJsYWNrIixhbHBoYT0wLjcsc2hvdy5sZWdlbmQgPSBGKStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9YygiI2YzMjQ0MCIsIiNmZmQ3MDAiKSkrY29vcmRfZmxpcCgpK2dndGl0bGUoIk9yaWdpbiIpCnNwMj10cmFpbnNldCU+JWdncGxvdChhZXMoeD1TdXJ2aXZhbCxmaWxsPVN1cnZpdmFsKSkrc3RhdF9jb3VudChjb2xvcj0iYmxhY2siLGFscGhhPTAuNyxzaG93LmxlZ2VuZCA9IEYpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1jKCIjZjMyNDQwIiwiI2ZmZDcwMCIpKStjb29yZF9mbGlwKCkrZ2d0aXRsZSgiVHJhaW4iKQpzcDM9dGVzdHNldCU+JWdncGxvdChhZXMoeD1TdXJ2aXZhbCxmaWxsPVN1cnZpdmFsKSkrc3RhdF9jb3VudChjb2xvcj0iYmxhY2siLGFscGhhPTAuNyxzaG93LmxlZ2VuZCA9IEYpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1jKCIjZjMyNDQwIiwiI2ZmZDcwMCIpKStjb29yZF9mbGlwKCkrZ2d0aXRsZSgiVGVzdCIpCgpncmlkLmFycmFuZ2Uoc3AxLHNwMixzcDMsbmNvbD0xKQoKd3RyYWluPWFzLmgybyh0cmFpbnNldCkKCnd0ZXN0PWFzLmgybyh0ZXN0c2V0KQoKcmVzcG9uc2U9IlN1cnZpdmFsIgpmZWF0dXJlcz1zZXRkaWZmKGNvbG5hbWVzKHd0cmFpbikscmVzcG9uc2UpCmBgYApPdXIgZXhwZXJpbWVudCBjb25zaXN0cyBvZiBldmFsdWF0aW5nIHRoZSBlZmZlY3Qgb2YgNCBwb3NzaWJsZSBzZXR0aW5nIGNvbWJpbmF0aW9ucyBiZXR3ZWVuIHRoZSBGb2xkIGFzc2lnbm1lbnQgYW5kIEJhbGFuY2UgY2xhc3NlcyBwYXJhbWV0ZXJzOgoKKyBNb2RlbCBSRiB3aXRoIFJhbmRvbWlzZWQgZm9sZCBhc3NpZ25tZW50IHdpdGhvdXQgY2xhc3NlcyBiYWxhbmNpbmcKCisgTW9kZWwgUkYgd2l0aCBSYW5kb21pc2VkIGZvbGQgYXNzaWdubWVudCB3aXRoIGNsYXNzZXMgYmFsYW5jaW5nCgorIE1vZGVsIFJGIHdpdGggU3RyYXRpZmllZCBmb2xkIGFzc2lnbm1lbnQgd2l0aG91dCBjbGFzc2VzIGJhbGFuY2luZwoKKyBNb2RlbCBSRiB3aXRoIFN0cmF0aWZpZWQgZm9sZCBhc3NpZ25tZW50IHdpdGggY2xhc3NlcyBiYWxhbmNpbmcKCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CiNSRiBsZWFybmVyCgojQmFsYW5jZWQgKyBzdHJhdGlmaWVkCgpyZm1vZDE9aDJvLnJhbmRvbUZvcmVzdCh4ID0gZmVhdHVyZXMsCiAgICAgICAgICAgICAgeSA9IHJlc3BvbnNlLAogICAgICAgICAgICAgIHRyYWluaW5nX2ZyYW1lID0gd3RyYWluLG5mb2xkcz0xMCwKICAgICAgICAgICAgICBmb2xkX2Fzc2lnbm1lbnQgPSAiU3RyYXRpZmllZCIsCiAgICAgICAgICAgICAgYmFsYW5jZV9jbGFzc2VzID0gVFJVRSxjbGFzc19zYW1wbGluZ19mYWN0b3JzPWMoMS4xNyw2LjY2KSwKICAgICAgICAgICAgICBudHJlZXMgPSAxMDAsIG1heF9kZXB0aCA9IDUwLAogICAgICAgICAgICAgIHN0b3BwaW5nX21ldHJpYyA9ICJsb2dsb3NzIiwKICAgICAgICAgICAgICBzdG9wcGluZ190b2xlcmFuY2UgPSAwLjAxLAogICAgICAgICAgICAgIHN0b3BwaW5nX3JvdW5kcyA9IDMsCiAgICAgICAgICAgICAga2VlcF9jcm9zc192YWxpZGF0aW9uX2ZvbGRfYXNzaWdubWVudCA9IFRSVUUsCiAgICAgICAgICAgICAga2VlcF9jcm9zc192YWxpZGF0aW9uX3ByZWRpY3Rpb25zPVRSVUUsCiAgICAgICAgICAgICAgc2NvcmVfZWFjaF9pdGVyYXRpb24gPSBUUlVFLAogICAgICAgICAgICAgIHNlZWQ9MTIzNDUpCgojQmFsYW5jZWQgKyBOb3Qgc3RyYXRpZmllZAoKcmZtb2QyPWgyby5yYW5kb21Gb3Jlc3QoeCA9IGZlYXR1cmVzLAogICAgICAgICAgICAgICB5ID0gcmVzcG9uc2UsCiAgICAgICAgICAgICAgIHRyYWluaW5nX2ZyYW1lID0gd3RyYWluLG5mb2xkcz0xMCwKICAgICAgICAgICAgICAgZm9sZF9hc3NpZ25tZW50ID0gIkFVVE8iLAogICAgICAgICAgICAgICBiYWxhbmNlX2NsYXNzZXMgPSBUUlVFLGNsYXNzX3NhbXBsaW5nX2ZhY3RvcnM9YygxLjE3LDYuNjYpLAogICAgICAgICAgICAgICBudHJlZXMgPSAxMDAsIG1heF9kZXB0aCA9IDUwLAogICAgICAgICAgICAgICBzdG9wcGluZ19tZXRyaWMgPSAibG9nbG9zcyIsCiAgICAgICAgICAgICAgIHN0b3BwaW5nX3RvbGVyYW5jZSA9IDAuMDEsCiAgICAgICAgICAgICAgIHN0b3BwaW5nX3JvdW5kcyA9IDMsCiAgICAgICAgICAgICAgIGtlZXBfY3Jvc3NfdmFsaWRhdGlvbl9mb2xkX2Fzc2lnbm1lbnQgPSBUUlVFLAogICAgICAgICAgICAgICBrZWVwX2Nyb3NzX3ZhbGlkYXRpb25fcHJlZGljdGlvbnM9VFJVRSwKICAgICAgICAgICAgICAgc2NvcmVfZWFjaF9pdGVyYXRpb24gPSBUUlVFLAogICAgICAgICAgICAgICBzZWVkPTEyMzQ1KQoKI1VuYmFsYW5jZWQgKyBzdHJhdGlmaWVkCgpyZm1vZDM9aDJvLnJhbmRvbUZvcmVzdCh4ID0gZmVhdHVyZXMsCiAgICAgICAgICAgICAgIHkgPSByZXNwb25zZSwKICAgICAgICAgICAgICAgdHJhaW5pbmdfZnJhbWUgPSB3dHJhaW4sbmZvbGRzPTEwLAogICAgICAgICAgICAgICBmb2xkX2Fzc2lnbm1lbnQgPSAiU3RyYXRpZmllZCIsCiAgICAgICAgICAgICAgIGJhbGFuY2VfY2xhc3NlcyA9IEZBTFNFLAogICAgICAgICAgICAgICBudHJlZXMgPSAxMDAsIG1heF9kZXB0aCA9IDUwLAogICAgICAgICAgICAgICBzdG9wcGluZ19tZXRyaWMgPSAibG9nbG9zcyIsCiAgICAgICAgICAgICAgIHN0b3BwaW5nX3RvbGVyYW5jZSA9IDAuMDEsCiAgICAgICAgICAgICAgIHN0b3BwaW5nX3JvdW5kcyA9IDMsCiAgICAgICAgICAgICAgIGtlZXBfY3Jvc3NfdmFsaWRhdGlvbl9mb2xkX2Fzc2lnbm1lbnQgPSBUUlVFLAogICAgICAgICAgICAgICBrZWVwX2Nyb3NzX3ZhbGlkYXRpb25fcHJlZGljdGlvbnM9VFJVRSwKICAgICAgICAgICAgICAgc2NvcmVfZWFjaF9pdGVyYXRpb24gPSBUUlVFLAogICAgICAgICAgICAgICBzZWVkPTEyMzQ1KQoKI1VuYmFsYW5jZWQgKyBub3Qgc3RyYXRpZmllZAoKcmZtb2QwPWgyby5yYW5kb21Gb3Jlc3QoeCA9IGZlYXR1cmVzLAogICAgICAgICAgICAgICB5ID0gcmVzcG9uc2UsCiAgICAgICAgICAgICAgIHRyYWluaW5nX2ZyYW1lID0gd3RyYWluLG5mb2xkcz0xMCwKICAgICAgICAgICAgICAgYmFsYW5jZV9jbGFzc2VzID0gRiwKICAgICAgICAgICAgICAgbnRyZWVzID0gMTAwLCBtYXhfZGVwdGggPSA1MCwKICAgICAgICAgICAgICAgc3RvcHBpbmdfbWV0cmljID0gImxvZ2xvc3MiLAogICAgICAgICAgICAgICBzdG9wcGluZ190b2xlcmFuY2UgPSAwLjAxLAogICAgICAgICAgICAgICBzdG9wcGluZ19yb3VuZHMgPSAzLAogICAgICAgICAgICAgICBrZWVwX2Nyb3NzX3ZhbGlkYXRpb25fZm9sZF9hc3NpZ25tZW50ID0gVFJVRSwKICAgICAgICAgICAgICAga2VlcF9jcm9zc192YWxpZGF0aW9uX3ByZWRpY3Rpb25zPVRSVUUsCiAgICAgICAgICAgICAgIHNjb3JlX2VhY2hfaXRlcmF0aW9uID0gVFJVRSwKICAgICAgICAgICAgICAgc2VlZD0xMjM0NSkKYGBgCgojIENvbmZ1c2lvbiBtYXRyaXggYW5kIHBlcmZvcm1hbmNlIG9mIDQgbW9kZWxzIG9uIHRlc3RzIHN1YnNldApgYGB7ciBtZXNzYWdlPVRSVUUsIHdhcm5pbmc9RkFMU0V9Cmgyby5wZXJmb3JtYW5jZShyZm1vZDAsd3Rlc3QpCmBgYApgYGB7ciBtZXNzYWdlPVRSVUUsIHdhcm5pbmc9RkFMU0V9Cmgyby5wZXJmb3JtYW5jZShyZm1vZDEsd3Rlc3QpCmBgYApgYGB7ciBtZXNzYWdlPVRSVUUsIHdhcm5pbmc9RkFMU0V9Cmgyby5wZXJmb3JtYW5jZShyZm1vZDIsd3Rlc3QpCmBgYApgYGB7ciBtZXNzYWdlPVRSVUUsIHdhcm5pbmc9RkFMU0V9Cmgyby5wZXJmb3JtYW5jZShyZm1vZDMsd3Rlc3QpCmBgYAoKU3VwcG9zZSB0aGF0IOKAnFN1cnZpdmVk4oCdIGlzIHRoZSBuZWdhdGl2ZSBhbmQg4oCcRGVhZOKAnSBpcyBwb3NpdGl2ZSBvdXRjb21lcyAoYXMgd2UgdHJ5IHRvIGlkZW50aWZ5IHRoZSBwYXRpZW50cyB3aXRoIGhpZ2hlciByaXNrIG9mIHBvc3Qtb3BlcmF0aXZlIG1vcnRhbGl0eSk6IEl0IHNlZW1zIHRoYXQgYWxsIDQgbW9kZWxzIGhhdmUgdGhlIHNhbWUgcGVyZm9ybWFuY2Ugb24gdGVzdCBzdWJzZXQ6IHRoZXkgcHJvdmlkZWQgY29ycmVjdCBzdXJ2aXZhbCBwcm9nbm9zaXMsIGJ1dCBhbGwgZmFpbGVkIHRvIGlkZW50aWZ5IE1vcnRhbGl0eSByaXNrLiBFYWNoIG1vZGVsIG1hZGVkIDE0IHRvIDE1IG1pc3Rha2VzIChhbGwgb2YgdGhlbSB3ZXJlIGZhbHNlIG5lZ2F0aXZlcykuIFdlIGRvbuKAmXQga25vdyB3aGV0aGVyIG91ciBzZXR0aW5ncyBjb3VsZCBoYXZlIGFueSBlZmZlY3Qgb24gZWFjaCBvbmUgb2YgdGhlc2UgNCBtb2RlbHMgb3Igbm90ID8uCgpUbyBmaW5kIHRoZSBhbnN3ZXIgZm9yIHRoaXMgcXVlc3Rpb24sIHdlIG11c3QgcmVzYW1wbGUgdGhlIHZhbGlkYXRpb24gb24gcmVwbGljYXRlZCBkYXRhc2V0IGJ5IGJvb3RzdHJhcHBpbmcKCiMgTUxSIHBhY2thZ2UgLSByZXNhbXBsZSB0aGUgdmFsaWRhdGlvbiBvbiByZXBsaWNhdGVkIGRhdGFzZXQgYnkgYm9vdHN0cmFwcGluZyAKCkZpcnN0LCB3ZSBjYWxsIHVwIHRoZSBtbHIgcGFja2FnZSwgYW5kIHRyYWluIGEgZHVtbXkgaDJvIHJhbmRvbSBmb3Jlc3QgbW9kZWwgaW4gaDJvLiBUaGUgcmVhbCBwdXJwb3NlIG9mIHRoaXMgZHVtbXkgbW9kZWwgaXMgZm9yIGNsb25pbmcgZXhhY3RseSB0aGUgNCBtb2RlbHMgdHJhaW5lZCBpbiBoMm8uIFRoaXMgdHJpY2sgaGFzIGJlZW4gd2VsbCBleHBsYWluZWQgaW4gdGhlIHByZXZpb3VzIHR1dG9yaWFscy4KCmBgYHtyIG1lc3NhZ2U9VFJVRSwgd2FybmluZz1GQUxTRX0KbGlicmFyeShtbHIpCgp0YXNrVFM9bWxyOjptYWtlQ2xhc3NpZlRhc2soaWQ9IlRob3JhYyIsZGF0YT1kZix0YXJnZXQ9IlN1cnZpdmFsIixwb3NpdGl2ZSA9ICJEZWFkIikKCmxlYXJuZXJIMk9SRj1tYWtlTGVhcm5lcihpZD0iaDJvUkYiLCJjbGFzc2lmLmgyby5yYW5kb21Gb3Jlc3QiLCBwcmVkaWN0LnR5cGUgPSAicHJvYiIpCgptbHJSRj10cmFpbihsZWFybmVyID0gbGVhcm5lckgyT1JGLCB0YXNrPXRhc2tUUykKYGBgCk5laXRoZXIgYWNjdXJhY3kgbm9yIGFic29sdXRlIG1pc2NsYXNzaWZpY2F0aW9uIHJhdGUgYXJlIHJpZ2h0IG1ldHJpYyB0byB1c2Ugd2hlbiBjb25mcm9udGluZyB0aGUgaW1iYWxhbmNlZCBkYXRhLiBUaGV5IGNvdWxkIGJlIG1pc2xlYWRpbmcsIGFzIG1lbnRpb25lZCBhYm92ZS4gT3RoZXIgbWV0cmljcyBhcmUgbW9yZSBhcHByb3ByaWF0ZSwgaW5jbHVkaW5nIFRydWUvRmFsc2UgTmVnYXRpdmUgb3IgUG9zaXRpdmUgcmF0ZSwgQmFsYW5jZWQgYWNjdXJhY3kgb3IgYmFsbGFuY2VkIGVycm9yIHJhdGUsIFJlY2FsbCAob3Igc3BlY2lmaWNpdHkpLCBDb2hlbuKAmXMgS2FwcGEgY29lZmZpY2llbnQgb3IgRjEgc2NvcmUuIFdlIHdpbGwgYWRvcHQgc29tZSBvZiB0aG9zZSBtZXRyaWNzIGZvciBvdXQgYmVuY2htYXJrIHN0dWR5LgoKYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KYm9vdG1sclBFUkY9ZnVuY3Rpb24oaDJvbW9kZWwsZGF0YSxpKXsKICBkPWRhdGFbaSxdCiAgcHJlZG1scj1wcmVkaWN0KG1sclJGLG5ld2RhdGE9ZCkKICBwcmVkaDJvPXByZWRpY3QoZ2V0KGgyb21vZGVsKSxhcy5oMm8oZCkpCiAgcHJlZG1sciRkYXRhJHJlc3BvbnNlPC1hcy52ZWN0b3IocHJlZGgybyRwcmVkaWN0KQogIHByZWRtbHIkZGF0YSRwcm9iLkRlYWQ8LWFzLnZlY3RvcihwcmVkaDJvJERlYWQpCiAgcHJlZG1sciRkYXRhJHByb2IuRGVhZDwtYXMudmVjdG9yKHByZWRoMm8kU3Vydml2ZWQpCiAgbWV0cz1saXN0KGJhYyxmMSx0cHIsdG5yLGZwcixmbnIpCiAgcD1tbHI6OnBlcmZvcm1hbmNlKHByZWRtbHIsbWV0cykKICBCQUM9cFtbMV1dCiAgRjE9cFtbMl1dCiAgVFBSPXBbWzNdXQogIFROUj1wW1s0XV0KICBGUFI9cFtbNV1dCiAgRk5SPXBbWzZdXQogIHJldHVybj1jYmluZChCQUMsRjEsVFBSLFROUixGUFIsRk5SKQp9CgoKc2V0LnNlZWQoMTIzKQpsaWJyYXJ5KGJvb3QpCgpwZXJmbW9kMD1ib290KHN0YXRpc3RpYz1ib290bWxyUEVSRixoMm9tb2RlbD0icmZtb2QwIixkYXRhPWRmLFI9MiklPiUuJHQlPiVhc190aWJibGUoKSU+JW11dGF0ZShNb2RlPSJVbmJhbGFuY2VkX1JhbmRvbSIsaXRlcj1hcy5udW1lcmljKHJvd25hbWVzKC4pKSkKCnBlcmZtb2QxPWJvb3Qoc3RhdGlzdGljPWJvb3RtbHJQRVJGLGgyb21vZGVsPSJyZm1vZDEiLGRhdGE9ZGYsUj0yKSU+JS4kdCU+JWFzX3RpYmJsZSgpJT4lbXV0YXRlKE1vZGU9IkJhbGFuY2VkX1N0cmF0aWZpZWQiLGl0ZXI9YXMubnVtZXJpYyhyb3duYW1lcyguKSkpCgpwZXJmbW9kMj1ib290KHN0YXRpc3RpYz1ib290bWxyUEVSRixoMm9tb2RlbD0icmZtb2QyIixkYXRhPWRmLFI9MiklPiUuJHQlPiVhc190aWJibGUoKSU+JW11dGF0ZShNb2RlPSJCYWxhbmNlZF9SYW5kb20iLGl0ZXI9YXMubnVtZXJpYyhyb3duYW1lcyguKSkpCgpwZXJmbW9kMz1ib290KHN0YXRpc3RpYz1ib290bWxyUEVSRixoMm9tb2RlbD0icmZtb2QzIixkYXRhPWRmLFI9MiklPiUuJHQlPiVhc190aWJibGUoKSU+JW11dGF0ZShNb2RlPSJVbmJhbGFuY2VkX1N0cmF0aWZpZWQiLGl0ZXI9YXMubnVtZXJpYyhyb3duYW1lcyguKSkpCmBgYApgYGB7ciBtZXNzYWdlPVRSVUUsIHdhcm5pbmc9RkFMU0V9CmJvb3RwZXJmPXJiaW5kKHBlcmZtb2QwLHBlcmZtb2QxLHBlcmZtb2QyLHBlcmZtb2QzKQoKbmFtZXMoYm9vdHBlcmYpPWMoIkJBQyIsIkYxIiwiVFBSIiwiVE5SIiwiRlBSIiwiRk5SIiwiTW9kZSIsIkl0ZXJhdGlvbiIpCgpib290cGVyZlssYygxOjYpXSU+JXBzeWNoOjpkZXNjcmliZUJ5KC4sYm9vdHBlcmYkTW9kZSkKYGBgCgpgYGB7cn0KcGFpcndpc2Uud2lsY294LnRlc3QoeD1ib290cGVyZiRGTlIsZz1ib290cGVyZiRNb2RlLHAuYWRqdXN0Lm1ldGhvZD0iYm9uZmVycm9uaSIsbj02LHBhaXJlZD1UKQpgYGAKCmBgYHtyfQpwYWlyd2lzZS53aWxjb3gudGVzdCh4PWJvb3RwZXJmJEZQUixnPWJvb3RwZXJmJE1vZGUscC5hZGp1c3QubWV0aG9kPSJib25mZXJyb25pIixuPTYscGFpcmVkPVQpCmBgYAoKYGBge3J9CmJvb3Rsb25nPWJvb3RwZXJmJT4lZ2F0aGVyKEJBQzpGTlIsa2V5PSJNZXRyaWMiLHZhbHVlPSJTY29yZSIpCgpib290bG9uZyU+JWdncGxvdChhZXMoeD1TY29yZSxmaWxsPU1vZGUpKStnZW9tX2hpc3RvZ3JhbShhbHBoYT0wLjYsY29sb3I9ImJsYWNrIikrZmFjZXRfZ3JpZChNb2Rlfk1ldHJpYyxzY2FsZXM9ImZyZWUiKStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9bXljb2xvcnMpCmBgYAoKYGBge3J9CmJvb3Rsb25nJT4lZ2dwbG90KGFlcyh4PU1ldHJpYyx5PVNjb3JlLGZpbGw9TW9kZSkpK2dlb21fYm94cGxvdChhbHBoYT0wLjYpK2ZhY2V0X3dyYXAofk1ldHJpYyxzY2FsZXM9ImZyZWUiLG5jb2w9Mikrc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzPW15Y29sb3JzKStjb29yZF9mbGlwKCkKYGBgCgpgYGB7cn0KYm9vdHBlcmYlPiVnZ3Bsb3QoYWVzKHg9SXRlcmF0aW9uLHk9RjEsY29sb3I9TW9kZSxmaWxsPU1vZGUpKStnZW9tX3BhdGgoYWxwaGE9MC44LHNpemU9MS4yKStnZW9tX3BvaW50KHNoYXBlPTIxLHNpemU9NCxjb2xvcj0iYmxhY2siKStzY2FsZV9jb2xvcl9tYW51YWwodmFsdWVzPWMoInJlZDQiLCJwdXJwbGUiLCJvcmFuZ2UiLCJibHVlIikpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1jKCJyZWQiLCJwdXJwbGUxIiwiZ29sZCIsInNreWJsdWUiKSkrZ2VvbV9obGluZSh5aW50ZXJjZXB0PTAuOTUsbGluZXR5cGU9MixzaXplPTEpK2ZhY2V0X3dyYXAofk1vZGUsbmNvbD0xKQpgYGAKCmBgYHtyfQpib290cGVyZiU+JWdncGxvdChhZXMoeD1JdGVyYXRpb24seT1CQUMsY29sb3I9TW9kZSxmaWxsPU1vZGUpKStnZW9tX3BhdGgoYWxwaGE9MC44LHNpemU9MS4yKStnZW9tX3BvaW50KHNoYXBlPTIxLHNpemU9NCxjb2xvcj0iYmxhY2siKStzY2FsZV9jb2xvcl9tYW51YWwodmFsdWVzPWMoInJlZDQiLCJwdXJwbGUiLCJvcmFuZ2UiLCJibHVlIikpK3NjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcz1jKCJyZWQiLCJwdXJwbGUxIiwiZ29sZCIsInNreWJsdWUiKSkrZ2VvbV9obGluZSh5aW50ZXJjZXB0PTAuOSxsaW5ldHlwZT0yLHNpemU9MSkrZmFjZXRfd3JhcCh+TW9kZSxuY29sPTEpCmBgYApgYGB7cn0KYm9vdHBlcmYlPiVnZ3Bsb3QoYWVzKHg9SXRlcmF0aW9uLHk9Rk5SLGNvbG9yPU1vZGUsZmlsbD1Nb2RlKSkrZ2VvbV9wYXRoKGFscGhhPTAuOCxzaXplPTEuMikrZ2VvbV9wb2ludChzaGFwZT0yMSxzaXplPTQsY29sb3I9ImJsYWNrIikrc2NhbGVfY29sb3JfbWFudWFsKHZhbHVlcz1jKCJyZWQ0IiwicHVycGxlIiwib3JhbmdlIiwiYmx1ZSIpKStzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXM9YygicmVkIiwicHVycGxlMSIsImdvbGQiLCJza3libHVlIikpK2dlb21faGxpbmUoeWludGVyY2VwdD0wLjEsbGluZXR5cGU9MixzaXplPTEpK2ZhY2V0X3dyYXAofk1vZGUsbmNvbD0xKQpgYGAKVGhlIGJvb3RzdHJhcGVkIHZhbGlkYXRpb24gb24gMzAgcmFuZG9tIHRlc3Rpbmcgc3Vic2V0cyByZXZlYWxlZCBzb21ldGhpbmcgbW9yZSBpbnRlcmVzdGluZzoKClRoZSBtb2RlbCB3aXRoIHN0YW5kLWFsb25lIGNsYXNzZXMgd2VpZ2h0aW5nIG9yIGNvbWJpbmVkIHdpdGggc3RyYXRpZmllZCBmb2xkIGFzc2lnbm1lbnQgcHJvZHVjZWQgdGhlIGJlc3QgcGVyZm9ybWFuY2UgKGluIHJlZCBhbmQgeWVsbG93KS4gVGhvc2UgdHdvIG1vZGVscyBoYWQgdGhlIGJlc3QgRjEgc2NvcmUsIGhpZ2hlc3QgVHJ1ZSBQb3NpdGl2ZSByYXRlIGFuZCBsb3dlc3QgZmFsc2UgbmVnYXRpdmUgcmF0ZS4gT25seSBtb2RlbCB0cmFpbmVkIGJ5IHN1Y2ggcHJvdG9jb2xlIGNhbiBtYWtlIHNlbnNlIHdoZW4gYXBwbGllZCB0byByYW5kb20gaW1iYWxhbmNlZCB2YWxpZGF0aW9uIHNldHMuCgpUaGUgZGlmZmVyZW5jZSBpbiBGYWxzZSBuZWdhdGl2ZSByYXRlIGJldHdlZW4gUmFuZG9tL0JhbGFuY2VkIG1vZGVsIGFuZCBSYW5kb20vVW5iYWxhbmNlZCBtb2RlbCBpcyBzaWduaWZpY2F0aXZlIChXaWxjb3hvbiB0ZXN0IHAgdmFsdWUgPTAuMDAwMDA5MSkuVGhlcmUgd2FzIG5vIGRpZmZlcmVuY2UgYmV0d2VlbiBCYWxhbmNlZC9SYW5kb20gYXNzaWdubWVudCBhbmQgQmFsYW5jZWQvc3RyYXRpZmllZCBtb2RlbHMuIFRoaXMgaW5kaWNhdGVzIHRoYXQgYXBwbHlpbmcgQmFsYW5jZSBjbGFzcyBwYXJhbWV0ZXIgYWxvbmUgKHdpdGhvdXQgRm9sZCBhc3NpZ25tZW50IFN0cmF0aWZpY2F0aW9uKSBjYW4gYWxyZWFkeSBpbXByb3ZlIHRoZSBtb2RlbOKAmXMgcGVyZm9ybWFuY2Ugb24gaW1iYWxhbmNlZCBkYXRhLkZvbGRzIHN0cmF0aWZpY2F0aW9uIG1pZ2h0IGJlIG5vdCBlbm91Z2ggdG8gcmVzb2x2ZSB0aGUgZGF0YSBpbWJhbGFuY2UgcHJvYmxlbS4gQ29tYmluaW5nIENsYXNzZXMgYmFsYW5jaW5nIGFuZCBTdHJhdGlmaWVkIGFzc2lnbm1lbnQgY291bGQgb3B0b21pc2Ugb3VyIG1vZGVs4oCZcyBwZXJmb3JtYW5jZS4KCiMgQ29uY2x1c2lvbgpDbGFzc2VzIGltYmFsYW5jZSBpcyBhIGNvbW1vbiBhbmQgZnJ1c3RyYXRpbmcgcHJvYmxlbSBpbiBtYWNoaW5lIGxlYXJuaW5nIHByYWN0aWNlLiBDbGFzcyBiYWxhbmNpbmcgaXMgYW4gdXNlZnVsIGZlYXR1cmUgaW4gaDJvIHRoYXQgY291bGQgYmUgdXNlZCBhbG9uZSBvciBpbiBjb21iaW5hdGlvbiB3aXRoIEZvbGQgYXNzaWdubWVudCBTdHJhdGlmaWNhdGlvbiBmb3IgZGVhbGluZyB3aXRoIHN1Y2ggcHJvYmxlbS4K