About Automated Machine Learning

Việc xây dựng các mô hình Machine Learning là một quá trình và hầu hết nguồn lực được tập trung vào hai giai đoạn là Feature Engineering và Turning Hyperparameters. Nhưng hai khâu này có thể là một thách thức với những người mà không thành thạo Machine Learning (xem thêm ở đây). Trong những tình huống như thế thì Automated Machine Learning có thể là một giải pháp hiệu quả nhất là trong những tình huống mà nguồn lực thời gian không còn nhiều như deadline đã sắp đến.

trong bài này chúng ta sẽ so sánh hiệu quả của sử dụng Automated Machine Learning để giải quyết một bài toán cụ thể, một vấn đề cụ thể là Credit Scoring / Credit Classification với bộ số liệu hmeq.csv đã được sử dụng trong post trước. Cụ thể hơn với Xgboost mặc định thì AUC trên test data là 0.9507 còn nếu áp dụng Bayesian Optimization để tinh chỉnh các tham số thì chúng ta có thể đạt được một kết quả tốt hơn với AUC cao hơn là 0.95489. Câu hỏi là: nếu vẫn sử dụng AUC làm tiêu chuẩn đánh giá và so sánh thì Automated Machine Learning có tốt hơn Xgboost tinh chỉnh hay không?.

Automated Machine Learning using h2o

Automated Machine Learning có thể được thực hiện bằng h2o là một open-source tool cho Machine Learning/Big Data được có thể chạy trên cả R lẫn Python với cú phát như nhau và cho nhiều thuật toán cả unsupervised lẫn supervised algorithms và dữ liệu đầu vào cho các thuật toán thì h2o nhận cả biến numeric, categorical lẫn ordinal. Đây là một điểm thuận lợi vì chúng ta không cần phải thực hiện, ví dụ, cái gọi là thủ tục one-hot encoding.

Trước hết đọc dữ liệu và xử lí missing data bằng Random Forest như đã trình bày trong post trước:

## 
## Missing value imputation by random forests
## 
##   Variables to impute:       MORTDUE, VALUE, REASON, JOB, YOJ, DEROG, DELINQ, CLAGE, NINQ, CLNO, DEBTINC
##   Variables used to impute:  BAD, LOAN, MORTDUE, VALUE, REASON, JOB, YOJ, DEROG, DELINQ, CLAGE, NINQ, CLNO, DEBTINC
## iter 1:  ...........
## iter 2:  ...........
## iter 3:  ...........

Với dữ liệu sau khi đã xử lí missing chúng ta phân chia thành Training, Validation và Testing data theo tỉ lệ 50% - 20% - 30%:

## 
## H2O is not running yet, starting it now...
## 
## Note:  In case of errors look at the following log files:
##     /tmp/RtmpwprFz2/h2o_chidung_started_from_r.out
##     /tmp/RtmpwprFz2/h2o_chidung_started_from_r.err
## 
## 
## Starting H2O JVM and connecting: . Connection successful!
## 
## R is connected to the H2O cluster: 
##     H2O cluster uptime:         1 seconds 846 milliseconds 
##     H2O cluster timezone:       Asia/Ho_Chi_Minh 
##     H2O data parsing timezone:  UTC 
##     H2O cluster version:        3.28.0.4 
##     H2O cluster version age:    6 days  
##     H2O cluster name:           H2O_started_from_R_chidung_dkb320 
##     H2O cluster total nodes:    1 
##     H2O cluster total memory:   32.00 GB 
##     H2O cluster total cores:    40 
##     H2O cluster allowed cores:  40 
##     H2O cluster healthy:        TRUE 
##     H2O Connection ip:          localhost 
##     H2O Connection port:        54321 
##     H2O Connection proxy:       NA 
##     H2O Internal Security:      FALSE 
##     H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4 
##     R Version:                  R version 3.6.2 (2019-12-12)

Với dữ liệu đã chuẩn bị chúng ta có thể thực hiện huấn luyện Automated Machine Learning bằng hàm h2o.automl() (nếu cần có thể tham khảo thêm ở đây) như sau:

## 
## 00:01:22.973: Stopping tolerance set by the user is < 70% of the recommended default of 0.01830630064845531, so models may take a long time to converge or may not converge at all.

Trong số 45 mô hình thì StackedEnsemble_AllModels_AutoML_20200229_142838 có AUC trên Validation Data là cao nhất và nó xếp ở đầu bảng như ta có thể thấy:

Table 1: AUC on Validation Data
model_id AUC_Val Rank
StackedEnsemble_AllModels_AutoML_20200301_000122 0.9720 1
StackedEnsemble_BestOfFamily_AutoML_20200301_000122 0.9713 2
GBM_grid__1_AutoML_20200301_000122_model_2 0.9712 3
DeepLearning_grid__2_AutoML_20200301_000122_model_2 0.9639 4
GBM_4_AutoML_20200301_000122 0.9636 5
GBM_grid__1_AutoML_20200301_000122_model_8 0.9606 6
GBM_grid__1_AutoML_20200301_000122_model_9 0.9590 7
DRF_1_AutoML_20200301_000122 0.9589 8
DeepLearning_grid__1_AutoML_20200301_000122_model_1 0.9578 9
GBM_3_AutoML_20200301_000122 0.9572 10
GBM_1_AutoML_20200301_000122 0.9570 11
DeepLearning_grid__3_AutoML_20200301_000122_model_2 0.9541 12
XRT_1_AutoML_20200301_000122 0.9539 13
GBM_2_AutoML_20200301_000122 0.9530 14
XGBoost_grid__1_AutoML_20200301_000122_model_13 0.9468 15
XGBoost_grid__1_AutoML_20200301_000122_model_7 0.9451 16
GBM_grid__1_AutoML_20200301_000122_model_4 0.9439 17
DeepLearning_grid__2_AutoML_20200301_000122_model_1 0.9404 18
XGBoost_grid__1_AutoML_20200301_000122_model_15 0.9395 19
DeepLearning_grid__1_AutoML_20200301_000122_model_2 0.9388 20
XGBoost_grid__1_AutoML_20200301_000122_model_9 0.9383 21
XGBoost_grid__1_AutoML_20200301_000122_model_18 0.9381 22
GBM_5_AutoML_20200301_000122 0.9360 23
GBM_grid__1_AutoML_20200301_000122_model_6 0.9359 24
GBM_grid__1_AutoML_20200301_000122_model_3 0.9355 25
GBM_grid__1_AutoML_20200301_000122_model_7 0.9342 26
XGBoost_grid__1_AutoML_20200301_000122_model_11 0.9320 27
DeepLearning_grid__1_AutoML_20200301_000122_model_3 0.9285 28
XGBoost_grid__1_AutoML_20200301_000122_model_4 0.9264 29
XGBoost_grid__1_AutoML_20200301_000122_model_6 0.9247 30
XGBoost_grid__1_AutoML_20200301_000122_model_12 0.9247 31
DeepLearning_grid__3_AutoML_20200301_000122_model_1 0.9234 32
XGBoost_grid__1_AutoML_20200301_000122_model_8 0.9219 33
XGBoost_1_AutoML_20200301_000122 0.9157 34
XGBoost_3_AutoML_20200301_000122 0.9135 35
XGBoost_grid__1_AutoML_20200301_000122_model_16 0.9124 36
XGBoost_2_AutoML_20200301_000122 0.9051 37
XGBoost_grid__1_AutoML_20200301_000122_model_3 0.8966 38
XGBoost_grid__1_AutoML_20200301_000122_model_17 0.8911 39
XGBoost_grid__1_AutoML_20200301_000122_model_14 0.8909 40
XGBoost_grid__1_AutoML_20200301_000122_model_1 0.8907 41
XGBoost_grid__1_AutoML_20200301_000122_model_2 0.8880 42
DeepLearning_grid__2_AutoML_20200301_000122_model_3 0.8816 43
XGBoost_grid__1_AutoML_20200301_000122_model_5 0.8813 44
DeepLearning_grid__1_AutoML_20200301_000122_model_4 0.8809 45
XGBoost_grid__1_AutoML_20200301_000122_model_10 0.8791 46
GBM_grid__1_AutoML_20200301_000122_model_1 0.8772 47
DeepLearning_1_AutoML_20200301_000122 0.8749 48
GBM_grid__1_AutoML_20200301_000122_model_5 0.8651 49
GBM_grid__1_AutoML_20200301_000122_model_10 0.8558 50
GLM_1_AutoML_20200301_000122 0.8484 51
DeepLearning_grid__3_AutoML_20200301_000122_model_3 0.8465 52

Còn AUC trên Test Data của tất cả các mô hình này:

Table 2: AUC on Test Data
model_id AUC_Test
StackedEnsemble_AllModels_AutoML_20200301_000122 0.9843819
StackedEnsemble_BestOfFamily_AutoML_20200301_000122 0.9831996
GBM_grid__1_AutoML_20200301_000122_model_2 0.9743221
DeepLearning_grid__2_AutoML_20200301_000122_model_2 0.9585590
GBM_4_AutoML_20200301_000122 0.9744070
GBM_grid__1_AutoML_20200301_000122_model_8 0.9683310
GBM_grid__1_AutoML_20200301_000122_model_9 0.9598262
DRF_1_AutoML_20200301_000122 0.9703508
DeepLearning_grid__1_AutoML_20200301_000122_model_1 0.9477725
GBM_3_AutoML_20200301_000122 0.9681218
GBM_1_AutoML_20200301_000122 0.9606316
DeepLearning_grid__3_AutoML_20200301_000122_model_2 0.9500418
XRT_1_AutoML_20200301_000122 0.9621121
GBM_2_AutoML_20200301_000122 0.9586605
XGBoost_grid__1_AutoML_20200301_000122_model_13 0.9522407
XGBoost_grid__1_AutoML_20200301_000122_model_7 0.9508245
GBM_grid__1_AutoML_20200301_000122_model_4 0.9550401
DeepLearning_grid__2_AutoML_20200301_000122_model_1 0.9492209
XGBoost_grid__1_AutoML_20200301_000122_model_15 0.9461947
DeepLearning_grid__1_AutoML_20200301_000122_model_2 0.9261560
XGBoost_grid__1_AutoML_20200301_000122_model_9 0.9480613
XGBoost_grid__1_AutoML_20200301_000122_model_18 0.9381486
GBM_5_AutoML_20200301_000122 0.9505036
GBM_grid__1_AutoML_20200301_000122_model_6 0.9431935
GBM_grid__1_AutoML_20200301_000122_model_3 0.9393982
GBM_grid__1_AutoML_20200301_000122_model_7 0.9440331
XGBoost_grid__1_AutoML_20200301_000122_model_11 0.9400173
DeepLearning_grid__1_AutoML_20200301_000122_model_3 0.9331990
XGBoost_grid__1_AutoML_20200301_000122_model_4 0.9318624
XGBoost_grid__1_AutoML_20200301_000122_model_6 0.9291490
XGBoost_grid__1_AutoML_20200301_000122_model_12 0.9317692
DeepLearning_grid__3_AutoML_20200301_000122_model_1 0.9206711
XGBoost_grid__1_AutoML_20200301_000122_model_8 0.9283684
XGBoost_1_AutoML_20200301_000122 0.9223824
XGBoost_3_AutoML_20200301_000122 0.9100295
XGBoost_grid__1_AutoML_20200301_000122_model_16 0.9133942
XGBoost_2_AutoML_20200301_000122 0.9089114
XGBoost_grid__1_AutoML_20200301_000122_model_3 0.8961714
XGBoost_grid__1_AutoML_20200301_000122_model_17 0.8964457
XGBoost_grid__1_AutoML_20200301_000122_model_14 0.8946526
XGBoost_grid__1_AutoML_20200301_000122_model_1 0.8934155
XGBoost_grid__1_AutoML_20200301_000122_model_2 0.8902082
DeepLearning_grid__2_AutoML_20200301_000122_model_3 0.8762030
XGBoost_grid__1_AutoML_20200301_000122_model_5 0.8845752
DeepLearning_grid__1_AutoML_20200301_000122_model_4 0.8625094
XGBoost_grid__1_AutoML_20200301_000122_model_10 0.8779112
GBM_grid__1_AutoML_20200301_000122_model_1 0.8753623
DeepLearning_1_AutoML_20200301_000122 0.8727773
GBM_grid__1_AutoML_20200301_000122_model_5 0.8642746
GBM_grid__1_AutoML_20200301_000122_model_10 0.8590785
GLM_1_AutoML_20200301_000122 0.8286777
DeepLearning_grid__3_AutoML_20200301_000122_model_3 0.8412118

Chúng ta kì vọng rằng AUC trên Validation Data cao thì sẽ dẫn đến AUC trên Test Data cũng cao. Tuy nhiên thực tế thì vẫn có ngoại lệ, mô hình có AUC cao thứ hai trên Validation Data nhưng AUC trên Test Data lại thua kém mô hình xếp thứ 2 (so sánh Table 1 và Table 2) nhưng xu hướng chung thì AUC trên Validation và Test data là tương quan dương rất cao và tương quan này có ý nghĩa thống kê:

## 
##  Pearson's product-moment correlation
## 
## data:  df_results$AUC_Val and auc_on_testData$AUC_Test
## t = 42.297, df = 50, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.9761603 0.9921584
## sample estimates:
##       cor 
## 0.9863124

Chốt lại chúng ta sẽ sử dụng StackedEnsemble_AllModels_AutoML_20200229_142838 - là mô hình tốt nhất và sử dụng mô hình này chúng ta có AUC là 0.9843819 trên Test Data. Sử dụng mô hình này để tính PD (Probability of Default) như sau:

Để so sánh chúng ta huấn luyện Xgboost rồi đánh giá chất lượng phân loại của mô hình này trên Test Data. Kết quả cho thấy AUC trên Test Data là thấp hơn (Table 3):

#=========================
#  Compare with Xgboost
#=========================

# Convert to data frame: 

bind_rows(as.data.frame(train), as.data.frame(valid)) -> df_train
as.data.frame(test) -> df_test


# Function conducts one-hot encoding: 

library(caret)

one_hotEncoding <- function(df) {
  
  dummies <- dummyVars("~.", data = df)
  
  predict(dummies, df) %>% as.data.frame() -> df_oneHot
  
  df_oneHot %>% 
    select(-BAD.Good) %>% 
    rename(BAD = BAD.Bad) %>% 
    return()
}


# Use function: 
df_train %>% one_hotEncoding() -> df_train
df_test %>% one_hotEncoding() -> df_test

# Convert features to DMatrix form: 

X_train <- df_train %>% 
  select(-BAD) %>% 
  as.matrix()

Y_train <- df_train %>% 
  pull(BAD)

X_test <- df_test %>% 
  select(-BAD) %>% 
  as.matrix()

Y_test <- df_test %>% 
  pull(BAD)

#------------------------------------------
#   Train XGBoost with default parameters
#------------------------------------------
library(xgboost)

# Convert to DMatrix form for train data: 
dtrain <- xgb.DMatrix(data = X_train, label = Y_train)

# Train a default XGBoost: 
set.seed(29)
xgb1 <- xgboost(data = dtrain, 
                objective = "binary:logistic", 
                eval_metric = "auc", 
                verbose = 0, 
                nround = 30)

# Use Xgboost for predicting PD: 
pd_xgb <- predict(xgb1, X_test)


# AUC on test data by Xgboost: 
library(pROC)
auc_xgb <- roc(Y_test, pd_xgb)$auc %>% as.numeric()


# AUC on test data by 1-th model: 
auc_best <- auc_on_testData$AUC_Test[1]

# Compare AUC by the two approaches: 
data.frame(Model = c("BestAutoML", "Xgboost"), AUC = c(auc_best, auc_xgb)) %>% 
  knitr::kable(caption = "Table 3: AUC on Test Data")
Table 3: AUC on Test Data
Model AUC
BestAutoML 0.9843819
Xgboost 0.9526290

Đến đây chúng ta có bằng chứng để tin rằng BestAutoML nên được sử dụng. Tuy nhiên để đánh giá chi tiết hơn nữa chúng ta có thể khảo sát và so sánh khả năng phân loại của hai mô hình này tương ứng với một loạt các ngưỡng phân loại (cutoff đã trình bày trong post trước). Trước hết chúng ta viết hai hàm tính ra các chỉ tiêu đánh giá mô hình phân loại tương ứng với một ngưỡng lựa chọn cho cả hai mô hình:

Rồi sử dụng hai hàm này để tính Model Performance trên một loạt giá trị của Cutoff cho cả hai mô hình:

Qua Figure 1 chúng ta thấy rằng ở cả 4 tiêu chí thì BestAutoML cao hơn Xgboost. Ngoài ra có hai kết quả đáng chú ý là:

  1. Ở các ngưỡng thấp hơn 0.2 thì Recall - tức là khả năng mô hình dự báo chính xác các trường hợp xấu (Bad, hay kí hiệu là B với hàm ý rằng khách hàng sẽ default) thì hai mô hình không có sự khác biệt đáng kể nhưng từ sau ngưỡng này thì sự chênh lệch là đáng kể và càng nới rộng. Điều này là bằng chứng chắc chắn rằng BestAutoML nếu được sử dụng cho mục đích phân loại thì sẽ là mô hình mang lại lợi nhuận cao hơn.

  2. Accuracy là có hình chữ U ngược hàm ý rằng với cả hai mô hình thì sẽ tồn tại một ngưỡng phân loại mà tối đa hóa chỉ tiêu này. Điều này là nhất quán với nguyên lí sự đánh đổi của các chỉ tiêu. Cụ thể hơn chúng ta có thể thấy rằng Recall và Specificity có xu hướng biến đổi ngược nhau khi thay đổi ngưỡng phân loại.

Chúng ta có thể diễn đạt bằng một cách truyền tải khác dễ hiểu hơn với những người không chuyên về dữ liệu và thống kê (thường họ là bậc quản lí - boss và thường họ thường có tiếng nói quyết định trong việc sử dụng / lựa chọn) bằng công cụ hình ảnh sau (Figure 2):

Ở đây, ví dụ, BB hàm ý là các khách hàng xấu (Bad) được phân loại đúng thành khách hàng xấu (Bad). Các kí hiệu khác như BG, GB, và GG được diễn giải tương tự. Rõ ràng từ Figure 2 chúng ta thấy BestAutoML phân loại và dự báo nhóm khách hàng xấu chính xác hơn ở mọi ngưỡng so với Xgboost.

And More about h2o

Ở trên chúng sử dụng thư viện xgboost để thực hiện huận luyện Xgboost. Có thể thấy hàm xgboost() đòi hỏi rằng dữ liệu bắt buộc phải ở dạng numeric. Đó là lí do mà chúng ta cần phải thực hiện thêm thủ tục one-hot encoding. Chúng ta có thể sử dụng hàm h2o.xgboost() để thực hiện thuật toán này mà không cần phải cần đến one-hot encoding. Hướng dẫn chi tiết về việc thực hiện thuật toán này có thể tham khảo ở đây cho R và ở đây cho Python.

R Environment and OS

## R version 3.6.2 (2019-12-12)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.4 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8    LC_NUMERIC=C           
##  [3] LC_TIME=vi_VN           LC_COLLATE=en_US.UTF-8 
##  [5] LC_MONETARY=vi_VN       LC_MESSAGES=en_US.UTF-8
##  [7] LC_PAPER=vi_VN          LC_NAME=C              
##  [9] LC_ADDRESS=C            LC_TELEPHONE=C         
## [11] LC_MEASUREMENT=vi_VN    LC_IDENTIFICATION=C    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] plotly_4.9.1     pROC_1.14.0      xgboost_0.90.0.2 caret_6.0-84    
##  [5] lattice_0.20-40  h2o_3.28.0.4     missRanger_2.1.0 forcats_0.4.0   
##  [9] stringr_1.4.0    dplyr_0.8.0.1    purrr_0.3.3      readr_1.3.1     
## [13] tidyr_0.8.3      tibble_2.1.3     ggplot2_3.2.1    tidyverse_1.2.1 
## 
## loaded via a namespace (and not attached):
##  [1] nlme_3.1-144       bitops_1.0-6       lubridate_1.7.4   
##  [4] httr_1.4.0         tools_3.6.2        backports_1.1.5   
##  [7] R6_2.4.1           rpart_4.1-15       lazyeval_0.2.2    
## [10] colorspace_1.4-1   nnet_7.3-12        withr_2.1.2       
## [13] tidyselect_0.2.5   curl_3.3           compiler_3.6.2    
## [16] cli_2.0.0          rvest_0.3.3        xml2_1.2.0        
## [19] labeling_0.3       scales_1.1.0       digest_0.6.23     
## [22] rmarkdown_1.12     base64enc_0.1-3    pkgconfig_2.0.3   
## [25] htmltools_0.3.6    highr_0.8          htmlwidgets_1.3   
## [28] rlang_0.4.2        readxl_1.3.1       rstudioapi_0.10   
## [31] FNN_1.1.3          shiny_1.3.2        generics_0.0.2    
## [34] jsonlite_1.6       crosstalk_1.0.0    ModelMetrics_1.2.2
## [37] RCurl_1.95-4.12    magrittr_1.5       Matrix_1.2-18     
## [40] Rcpp_1.0.3         munsell_0.5.0      fansi_0.4.0       
## [43] lifecycle_0.1.0    stringi_1.4.3      yaml_2.2.0        
## [46] MASS_7.3-51.5      plyr_1.8.5         recipes_0.1.5     
## [49] grid_3.6.2         promises_1.0.1     crayon_1.3.4      
## [52] haven_2.1.0        splines_3.6.2      hms_0.4.2         
## [55] knitr_1.22         pillar_1.4.3       ranger_0.11.2     
## [58] reshape2_1.4.3     codetools_0.2-16   stats4_3.6.2      
## [61] glue_1.3.1         evaluate_0.13      data.table_1.12.2 
## [64] modelr_0.1.4       httpuv_1.5.1       foreach_1.4.4     
## [67] cellranger_1.1.0   gtable_0.3.0       assertthat_0.2.1  
## [70] xfun_0.6           gower_0.2.0        mime_0.6          
## [73] prodlim_2018.04.18 xtable_1.8-4       broom_0.5.2       
## [76] e1071_1.7-1        later_0.8.0        class_7.3-15      
## [79] survival_3.1-8     viridisLite_0.3.0  timeDate_3043.102 
## [82] iterators_1.0.10   lava_1.6.5         ipred_0.9-9
LS0tCnRpdGxlOiAnQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmcgZm9yIENyZWRpdCBTY29yaW5nIFByb2JsZW0nCmF1dGhvcjogJ0F1dGhvcjogTmd1eWVuIENoaSBEdW5nJwpzdWJ0aXRsZTogIlIgTWFjaGluZSBMZWFybmluZyBTZXJpZXMiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OiAKICAgIGNvZGVfZG93bmxvYWQ6IHRydWUKICAgICNjb2RlX2ZvbGRpbmc6IGhpZGUKICAgIGhpZ2hsaWdodDogemVuYnVybgogICAgIyBudW1iZXJfc2VjdGlvbnM6IHllcwogICAgdGhlbWU6ICJmbGF0bHkiCiAgICB0b2M6IFRSVUUKICAgIHRvY19mbG9hdDogVFJVRQotLS0KCmBgYHtyIHNldHVwLGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwgd2FybmluZyA9IEZBTFNFLCBtZXNzYWdlID0gRkFMU0UsIGZpZy53aWR0aCA9IDEwLCBmaWcuaGVpZ2h0ID0gNikKYGBgCgohW10oL2hvbWUvY2hpZHVuZy9Eb2N1bWVudHMvYXV0b21sLnBuZykKCiMgQWJvdXQgQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmcKClZp4buHYyB4w6J5IGThu7FuZyBjw6FjIG3DtCBow6xuaCBNYWNoaW5lIExlYXJuaW5nIGzDoCBt4buZdCBxdcOhIHRyw6xuaCB2w6AgaOG6p3UgaOG6v3Qgbmd14buTbiBs4buxYyDEkcaw4bujYyB04bqtcCB0cnVuZyB2w6BvIGhhaSBnaWFpIMSRb+G6oW4gbMOgIEZlYXR1cmUgRW5naW5lZXJpbmcgdsOgIFR1cm5pbmcgSHlwZXJwYXJhbWV0ZXJzLiBOaMawbmcgaGFpIGtow6J1IG7DoHkgY8OzIHRo4buDIGzDoCBt4buZdCB0aMOhY2ggdGjhu6ljIHbhu5tpIG5o4buvbmcgbmfGsOG7nWkgbcOgICpraMO0bmcgdGjDoG5oIHRo4bqhbyBNYWNoaW5lIExlYXJuaW5nKiAoeGVtIHRow6ptIFvhu58gxJHDonldKGh0dHBzOi8vd3d3LnVkZW15LmNvbS9jb3Vyc2UvYXV0b21hdGVkLW1hY2hpbmUtbGVhcm5pbmcvKSkuIFRyb25nIG5o4buvbmcgdMOsbmggaHXhu5FuZyBuaMawIHRo4bq/IHRow6wgQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmcgY8OzIHRo4buDIGzDoCBt4buZdCBnaeG6o2kgcGjDoXAgaGnhu4d1IHF14bqjIG5o4bqldCBsw6AgdHJvbmcgbmjhu69uZyB0w6xuaCBodeG7kW5nIG3DoCBuZ3Xhu5NuIGzhu7FjIHRo4budaSBnaWFuIGtow7RuZyBjw7JuIG5oaeG7gXUgbmjGsCBkZWFkbGluZSDEkcOjIHPhuq9wIMSR4bq/bi4gCgp0cm9uZyBiw6BpIG7DoHkgY2jDum5nIHRhIHPhur0gc28gc8OhbmggaGnhu4d1IHF14bqjIGPhu6dhIHPhu60gZOG7pW5nIEF1dG9tYXRlZCBNYWNoaW5lIExlYXJuaW5nIMSR4buDIGdp4bqjaSBxdXnhur90IG3hu5l0IGLDoGkgdG/DoW4gY+G7pSB0aOG7gywgbeG7mXQgduG6pW4gxJHhu4EgY+G7pSB0aOG7gyBsw6AgQ3JlZGl0IFNjb3JpbmcgLyBDcmVkaXQgQ2xhc3NpZmljYXRpb24gduG7m2kgYuG7mSBz4buRIGxp4buHdSBobWVxLmNzdiDEkcOjIMSRxrDhu6NjIHPhu60gZOG7pW5nIHRyb25nIFtwb3N0IHRyxrDhu5tjXShodHRwczovL3JwdWJzLmNvbS9jaGlkdW5na3QvNTc3MzM0KS4gQ+G7pSB0aOG7gyBoxqFuIHbhu5tpIFhnYm9vc3QgbeG6t2MgxJHhu4tuaCB0aMOsIEFVQyB0csOqbiB0ZXN0IGRhdGEgbMOgIDAuOTUwNyBjw7JuIG7hur91IMOhcCBk4bulbmcgQmF5ZXNpYW4gT3B0aW1pemF0aW9uIMSR4buDIHRpbmggY2jhu4luaCBjw6FjIHRoYW0gc+G7kSB0aMOsIGNow7puZyB0YSBjw7MgdGjhu4MgxJHhuqF0IMSRxrDhu6NjIG3hu5l0IGvhur90IHF14bqjIHThu5F0IGjGoW4gduG7m2kgQVVDIGNhbyBoxqFuIGzDoCAwLjk1NDg5LiBDw6J1IGjhu49pIGzDoDogbuG6v3UgduG6q24gc+G7rSBk4bulbmcgQVVDIGzDoG0gdGnDqnUgY2h14bqpbiDEkcOhbmggZ2nDoSB2w6Agc28gc8OhbmggdGjDrCAqKkF1dG9tYXRlZCBNYWNoaW5lIExlYXJuaW5nIGPDsyB04buRdCBoxqFuIFhnYm9vc3QgdGluaCBjaOG7iW5oIGhheSBraMO0bmc/KiouIAoKIyBBdXRvbWF0ZWQgTWFjaGluZSBMZWFybmluZyB1c2luZyBoMm8KCkF1dG9tYXRlZCBNYWNoaW5lIExlYXJuaW5nIGPDsyB0aOG7gyDEkcaw4bujYyB0aOG7sWMgaGnhu4duIGLhurFuZyBbaDJvXShodHRwczovL3d3dy5oMm8uYWkvKSBsw6AgbeG7mXQgb3Blbi1zb3VyY2UgdG9vbCBjaG8gTWFjaGluZSBMZWFybmluZy9CaWcgRGF0YSDEkcaw4bujYyBjw7MgdGjhu4MgY2jhuqF5IHRyw6puIGPhuqMgUiBs4bqrbiBQeXRob24gduG7m2kgY8O6IHBow6F0IG5oxrAgbmhhdSB2w6AgY2hvIG5oaeG7gXUgdGh14bqtdCB0b8OhbiBj4bqjIHVuc3VwZXJ2aXNlZCBs4bqrbiBzdXBlcnZpc2VkIGFsZ29yaXRobXMgdsOgIGThu68gbGnhu4d1IMSR4bqndSB2w6BvIGNobyBjw6FjIHRodeG6rXQgdG/DoW4gdGjDrCBoMm8gbmjhuq1uIGPhuqMgYmnhur9uIG51bWVyaWMsIGNhdGVnb3JpY2FsIGzhuqtuIG9yZGluYWwuIMSQw6J5IGzDoCBt4buZdCDEkWnhu4NtIHRodeG6rW4gbOG7o2kgdsOsIGNow7puZyB0YSBraMO0bmcgY+G6p24gcGjhuqNpIHRo4buxYyBoaeG7h24sIHbDrSBk4bulLCBjw6FpIGfhu41pIGzDoCB0aOG7pyB04bulYyBvbmUtaG90IGVuY29kaW5nLiAKCgpUcsaw4bubYyBo4bq/dCDEkeG7jWMgZOG7ryBsaeG7h3UgdsOgIHjhu60gbMOtIG1pc3NpbmcgZGF0YSBi4bqxbmcgUmFuZG9tIEZvcmVzdCBuaMawIMSRw6MgdHLDrG5oIGLDoHkgdHJvbmcgcG9zdCB0csaw4bubYzogCgpgYGB7cn0KCiM9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KIyAgICAgICBEYXRhIFByZS1wcm9jZXNzaW5nCiM9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KCiMgQ2xlYXIgd29ya3NwYWNlOiAKcm0obGlzdCA9IGxzKCkpCgojIEltcG9ydCBkYXRhOiAKbGlicmFyeSh0aWR5dmVyc2UpCmhtZXEgPC0gcmVhZF9jc3YoImh0dHA6Ly93d3cuY3JlZGl0cmlza2FuYWx5dGljcy5uZXQvdXBsb2Fkcy8xLzkvNS8xLzE5NTExNjAxL2htZXEuY3N2IikKCiMgU3RhZ2UgMSAtIEltcHV0ZSBtaXNzaW5nIGRhdGE6IApsaWJyYXJ5KG1pc3NSYW5nZXIpCgptaXNzUmFuZ2VyKGhtZXEsIHNlZWQgPSAyOSkgLT4gaG1lcV9pbXB1dGVkCgojIFN0YWdlIDIgLSBOb3JtYWxpemUgMC0xIGZlYXR1cmVzOiAKCmhtZXFfaW1wdXRlZCAlPiUgCiAgbXV0YXRlKEJBRCA9IGNhc2Vfd2hlbihCQUQgPT0gMSB+ICJCYWQiLCBUUlVFIH4gIkdvb2QiKSkgJT4lIAogIG11dGF0ZV9pZihpcy5jaGFyYWN0ZXIsIGFzLmZhY3RvcikgJT4lIAogIG11dGF0ZV9pZihpcy5udW1lcmljLCBmdW5jdGlvbih4KSB7KHggLSBtaW4oeCkpIC8gKG1heCh4KSAtIG1pbih4KSl9KSAtPiBkZl9maW5hbApgYGAKClbhu5tpIGThu68gbGnhu4d1IHNhdSBraGkgxJHDoyB44butIGzDrSBtaXNzaW5nIGNow7puZyB0YSBwaMOibiBjaGlhIHRow6BuaCBUcmFpbmluZywgVmFsaWRhdGlvbiB2w6AgVGVzdGluZyBkYXRhIHRoZW8gdOG7iSBs4buHIDUwJSAtIDIwJSAtIDMwJToKCmBgYHtyfQojIFN0YWdlIC0gU3BsaXQgZGF0YSBmb3IgdHJhaW5pbmcsIHZhbGlkYXRpb24gYW5kIHRlc3Rpbmc6IAoKbGlicmFyeShoMm8pCmgyby5pbml0KG50aHJlYWRzID0gNDAsIG1heF9tZW1fc2l6ZSA9ICIzMmciKQpoMm8ubm9fcHJvZ3Jlc3MoKQoKYXMuaDJvKGRmX2ZpbmFsKSAtPiBoMm9fZnJhbWUKc3BsaXRzIDwtIGgyby5zcGxpdEZyYW1lKGgyb19mcmFtZSwgcmF0aW9zID0gYygwLjUsIDAuMiksIHNlZWQgPSAyOSkKCnRyYWluIDwtIHNwbGl0c1tbMV1dCnZhbGlkIDwtIHNwbGl0c1tbMl1dCnRlc3QgPC0gc3BsaXRzW1szXV0KCiMgRGVmaW5lIHByZWRpY3RvcnMgYW5kIHRhcmdldDogCnkgPC0gIkJBRCIKeCA8LSBzZXRkaWZmKG5hbWVzKHRyYWluKSwgeSkKCmBgYAoKVuG7m2kgZOG7ryBsaeG7h3UgxJHDoyBjaHXhuqluIGLhu4sgY2jDum5nIHRhIGPDsyB0aOG7gyB0aOG7sWMgaGnhu4duIGh14bqlbiBsdXnhu4duIEF1dG9tYXRlZCBNYWNoaW5lIExlYXJuaW5nIGLhurFuZyBow6BtICoqaDJvLmF1dG9tbCgpKiogKG7hur91IGPhuqduIGPDsyB0aOG7gyB0aGFtIGto4bqjbyB0aMOqbSBb4bufIMSRw6J5XShodHRwOi8vZG9jcy5oMm8uYWkvaDJvL2xhdGVzdC1zdGFibGUvaDJvLWRvY3MvYXV0b21sLmh0bWwpKSBuaMawIHNhdTogCgpgYGB7cn0KCiM9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQojICBUcmFpbmluZyBBdXRvIE1hY2hpbmUgTGVhcm5pbmcKIz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09CgojIFRyYWluIEF1dG8gTWFjaGluZSBMZWFybmluZzogCgphdXRvTUwgPC0gaDJvLmF1dG9tbCh4ID0geCwgCiAgICAgICAgICAgICAgICAgICAgIHkgPSB5LCAKICAgICAgICAgICAgICAgICAgICAgdHJhaW5pbmdfZnJhbWUgPSB0cmFpbiwgCiAgICAgICAgICAgICAgICAgICAgIGxlYWRlcmJvYXJkX2ZyYW1lID0gdmFsaWQsIAogICAgICAgICAgICAgICAgICAgICBzdG9wcGluZ19tZXRyaWMgPSAiQVVDIiwgCiAgICAgICAgICAgICAgICAgICAgIHN0b3BwaW5nX3JvdW5kcyA9IDEwLCAKICAgICAgICAgICAgICAgICAgICAgc3RvcHBpbmdfdG9sZXJhbmNlID0gMC4wMSwgCiAgICAgICAgICAgICAgICAgICAgIG1heF9tb2RlbHMgPSA1MCwgCiAgICAgICAgICAgICAgICAgICAgIG1heF9ydW50aW1lX3NlY3MgPSA2MCo2MCwgCiAgICAgICAgICAgICAgICAgICAgIHNlZWQgPSAxLCAKICAgICAgICAgICAgICAgICAgICAgc29ydF9tZXRyaWMgPSAiQVVDIikKCmBgYAoKVHJvbmcgc+G7kSA0NSBtw7QgaMOsbmggdGjDrCAqU3RhY2tlZEVuc2VtYmxlX0FsbE1vZGVsc19BdXRvTUxfMjAyMDAyMjlfMTQyODM4KiBjw7MgQVVDIHRyw6puIFZhbGlkYXRpb24gRGF0YSBsw6AgY2FvIG5o4bqldCB2w6AgbsOzIHjhur9wIOG7nyDEkeG6p3UgYuG6o25nICBuaMawIHRhIGPDsyB0aOG7gyB0aOG6pXk6IAoKCmBgYHtyfQojIE1vZGVsIHBlcmZvcm1hbmNlIGJ5IEFVQzogCgphdXRvTUxAbGVhZGVyYm9hcmQgJT4lIAogIGFzLmRhdGEuZnJhbWUoKSAlPiUgCiAgc2VsZWN0KG1vZGVsX2lkLCBhdWMpICU+JSAKICBtdXRhdGUoUmFuayA9IDE6bnJvdyguKSwgYXVjID0gcm91bmQoYXVjLCA0KSkgJT4lIAogIHJlbmFtZShBVUNfVmFsID0gYXVjKSAtPiBkZl9yZXN1bHRzCgpkZl9yZXN1bHRzICU+JSAKICBrbml0cjo6a2FibGUoY2FwdGlvbiA9ICJUYWJsZSAxOiBBVUMgb24gVmFsaWRhdGlvbiBEYXRhIikKCmBgYAoKQ8OybiBBVUMgdHLDqm4gVGVzdCBEYXRhIGPhu6dhIHThuqV0IGPhuqMgY8OhYyBtw7QgaMOsbmggbsOgeTogCgpgYGB7cn0KIyBBVUMgb24gdGVzdCBkYXRhIGJ5IGktdGggbW9kZWw6IAoKZ2V0QVVDX29uVGVzdERhdGEgPC0gZnVuY3Rpb24oaSkgewogIAogICMgRXh0cmFjdCBpLXRoIG1vZGVsOiAKICBoMm8uZ2V0TW9kZWwoYXV0b01MQGxlYWRlcmJvYXJkW2ksIDFdKSAtPiBiZXN0X2l0aAogIAogICMgTW9kZWwgcGVyZm9ybWFuY2UgYnkgaXRoIG1vZGVsIGJ5IEFVQyBvbiBUZXN0IGRhdGE6ICAKICBoMm8ucGVyZm9ybWFuY2UobW9kZWwgPSBiZXN0X2l0aCwgbmV3ZGF0YSA9IHRlc3QpIC0+IG1ldHJpY3NfaXRoCiAgCiAgIyBSZXR1cm4gb3V0cHV0OiAKICByZXR1cm4oZGF0YS5mcmFtZShBVUNfVGVzdCA9IG1ldHJpY3NfaXRoQG1ldHJpY3MkQVVDLCBtb2RlbF9pZCA9IGJlc3RfaXRoQG1vZGVsX2lkKSkKICAKfQoKIyBDYWxjdWxhdGUgQVVDIGZvciBhbGwgbW9kZWxzOiAKCmxhcHBseSgxOm5yb3coZGZfcmVzdWx0cyksIGdldEFVQ19vblRlc3REYXRhKSAtPiBhdWNfb25fdGVzdERhdGEKZG8uY2FsbCgiYmluZF9yb3dzIiwgYXVjX29uX3Rlc3REYXRhKSAtPiBhdWNfb25fdGVzdERhdGEKCiMgQVVDIGJ5IGFsbCBtb2RlbHMgb24gdGVzdCBkYXRhOiAKYXVjX29uX3Rlc3REYXRhICU+JSAKICBzZWxlY3QobW9kZWxfaWQsIEFVQ19UZXN0KSAlPiUgCiAga25pdHI6OmthYmxlKGNhcHRpb24gPSAiVGFibGUgMjogQVVDIG9uIFRlc3QgRGF0YSIpCmBgYAoKQ2jDum5nIHRhIGvDrCB24buNbmcgcuG6sW5nIEFVQyB0csOqbiBWYWxpZGF0aW9uIERhdGEgY2FvIHRow6wgc+G6vSBk4bqrbiDEkeG6v24gQVVDIHRyw6puIFRlc3QgRGF0YSBjxaluZyBjYW8uIFR1eSBuaGnDqm4gdGjhu7FjIHThur8gdGjDrCB24bqrbiBjw7Mgbmdv4bqhaSBs4buHLCBtw7QgaMOsbmggY8OzIEFVQyBjYW8gdGjhu6kgaGFpIHRyw6puIFZhbGlkYXRpb24gRGF0YSBuaMawbmcgQVVDIHRyw6puIFRlc3QgRGF0YSBs4bqhaSB0aHVhIGvDqW0gbcO0IGjDrG5oIHjhur9wIHRo4bupIDIgKHNvIHPDoW5oIFRhYmxlIDEgdsOgIFRhYmxlIDIpIG5oxrBuZyB4dSBoxrDhu5tuZyBjaHVuZyB0aMOsIEFVQyB0csOqbiBWYWxpZGF0aW9uIHbDoCBUZXN0IGRhdGEgbMOgIHTGsMahbmcgcXVhbiBkxrDGoW5nIHLhuqV0IGNhbyB2w6AgdMawxqFuZyBxdWFuIG7DoHkgY8OzIMO9IG5naMSpYSB0aOG7kW5nIGvDqjogIAoKYGBge3J9CgojIENvcnJlbGF0aW9uIGJldHdlZW4gVmFsaWRhdGF0aW9uIGFuZCB0ZXN0IEFVQzogCmNvci50ZXN0KGRmX3Jlc3VsdHMkQVVDX1ZhbCwgYXVjX29uX3Rlc3REYXRhJEFVQ19UZXN0KQoKYGBgCgpDaOG7kXQgbOG6oWkgY2jDum5nIHRhIHPhur0gc+G7rSBk4bulbmcgKlN0YWNrZWRFbnNlbWJsZV9BbGxNb2RlbHNfQXV0b01MXzIwMjAwMjI5XzE0MjgzOCogLSBsw6AgbcO0IGjDrG5oIHThu5F0IG5o4bqldCB2w6Agc+G7rSBk4bulbmcgbcO0IGjDrG5oIG7DoHkgY2jDum5nIHRhIGPDsyBBVUMgbMOgIGByIGF1Y19vbl90ZXN0RGF0YSRBVUNfVGVzdFsxXWAgdHLDqm4gVGVzdCBEYXRhLiBT4butIGThu6VuZyBtw7QgaMOsbmggbsOgeSDEkeG7gyB0w61uaCBQRCAoUHJvYmFiaWxpdHkgb2YgRGVmYXVsdCkgbmjGsCBzYXU6IAoKYGBge3J9CiMgVXNlIGJlc3QgbW9kZWwgZm9yIHByZWRpY3RpbmcgUEQ6IApwZF9iZXN0IDwtIGgyby5wcmVkaWN0KGF1dG9NTEBsZWFkZXIsIHRlc3QpICU+JSAKICBhcy5kYXRhLmZyYW1lKCkgJT4lIAogIHB1bGwoQmFkKQoKYGBgCgrEkOG7gyBzbyBzw6FuaCBjaMO6bmcgdGEgaHXhuqVuIGx1eeG7h24gWGdib29zdCBy4buTaSDEkcOhbmggZ2nDoSBjaOG6pXQgbMaw4bujbmcgcGjDom4gbG/huqFpIGPhu6dhIG3DtCBow6xuaCBuw6B5IHRyw6puIFRlc3QgRGF0YS4gS+G6v3QgcXXhuqMgY2hvIHRo4bqleSBBVUMgdHLDqm4gVGVzdCBEYXRhIGzDoCB0aOG6pXAgaMahbiAoVGFibGUgMyk6IAoKYGBge3J9CgojPT09PT09PT09PT09PT09PT09PT09PT09PQojICBDb21wYXJlIHdpdGggWGdib29zdAojPT09PT09PT09PT09PT09PT09PT09PT09PQoKIyBDb252ZXJ0IHRvIGRhdGEgZnJhbWU6IAoKYmluZF9yb3dzKGFzLmRhdGEuZnJhbWUodHJhaW4pLCBhcy5kYXRhLmZyYW1lKHZhbGlkKSkgLT4gZGZfdHJhaW4KYXMuZGF0YS5mcmFtZSh0ZXN0KSAtPiBkZl90ZXN0CgoKIyBGdW5jdGlvbiBjb25kdWN0cyBvbmUtaG90IGVuY29kaW5nOiAKCmxpYnJhcnkoY2FyZXQpCgpvbmVfaG90RW5jb2RpbmcgPC0gZnVuY3Rpb24oZGYpIHsKICAKICBkdW1taWVzIDwtIGR1bW15VmFycygifi4iLCBkYXRhID0gZGYpCiAgCiAgcHJlZGljdChkdW1taWVzLCBkZikgJT4lIGFzLmRhdGEuZnJhbWUoKSAtPiBkZl9vbmVIb3QKICAKICBkZl9vbmVIb3QgJT4lIAogICAgc2VsZWN0KC1CQUQuR29vZCkgJT4lIAogICAgcmVuYW1lKEJBRCA9IEJBRC5CYWQpICU+JSAKICAgIHJldHVybigpCn0KCgojIFVzZSBmdW5jdGlvbjogCmRmX3RyYWluICU+JSBvbmVfaG90RW5jb2RpbmcoKSAtPiBkZl90cmFpbgpkZl90ZXN0ICU+JSBvbmVfaG90RW5jb2RpbmcoKSAtPiBkZl90ZXN0CgojIENvbnZlcnQgZmVhdHVyZXMgdG8gRE1hdHJpeCBmb3JtOiAKClhfdHJhaW4gPC0gZGZfdHJhaW4gJT4lIAogIHNlbGVjdCgtQkFEKSAlPiUgCiAgYXMubWF0cml4KCkKCllfdHJhaW4gPC0gZGZfdHJhaW4gJT4lIAogIHB1bGwoQkFEKQoKWF90ZXN0IDwtIGRmX3Rlc3QgJT4lIAogIHNlbGVjdCgtQkFEKSAlPiUgCiAgYXMubWF0cml4KCkKCllfdGVzdCA8LSBkZl90ZXN0ICU+JSAKICBwdWxsKEJBRCkKCiMtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyAgIFRyYWluIFhHQm9vc3Qgd2l0aCBkZWZhdWx0IHBhcmFtZXRlcnMKIy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpsaWJyYXJ5KHhnYm9vc3QpCgojIENvbnZlcnQgdG8gRE1hdHJpeCBmb3JtIGZvciB0cmFpbiBkYXRhOiAKZHRyYWluIDwtIHhnYi5ETWF0cml4KGRhdGEgPSBYX3RyYWluLCBsYWJlbCA9IFlfdHJhaW4pCgojIFRyYWluIGEgZGVmYXVsdCBYR0Jvb3N0OiAKc2V0LnNlZWQoMjkpCnhnYjEgPC0geGdib29zdChkYXRhID0gZHRyYWluLCAKICAgICAgICAgICAgICAgIG9iamVjdGl2ZSA9ICJiaW5hcnk6bG9naXN0aWMiLCAKICAgICAgICAgICAgICAgIGV2YWxfbWV0cmljID0gImF1YyIsIAogICAgICAgICAgICAgICAgdmVyYm9zZSA9IDAsIAogICAgICAgICAgICAgICAgbnJvdW5kID0gMzApCgojIFVzZSBYZ2Jvb3N0IGZvciBwcmVkaWN0aW5nIFBEOiAKcGRfeGdiIDwtIHByZWRpY3QoeGdiMSwgWF90ZXN0KQoKCiMgQVVDIG9uIHRlc3QgZGF0YSBieSBYZ2Jvb3N0OiAKbGlicmFyeShwUk9DKQphdWNfeGdiIDwtIHJvYyhZX3Rlc3QsIHBkX3hnYikkYXVjICU+JSBhcy5udW1lcmljKCkKCgojIEFVQyBvbiB0ZXN0IGRhdGEgYnkgMS10aCBtb2RlbDogCmF1Y19iZXN0IDwtIGF1Y19vbl90ZXN0RGF0YSRBVUNfVGVzdFsxXQoKIyBDb21wYXJlIEFVQyBieSB0aGUgdHdvIGFwcHJvYWNoZXM6IApkYXRhLmZyYW1lKE1vZGVsID0gYygiQmVzdEF1dG9NTCIsICJYZ2Jvb3N0IiksIEFVQyA9IGMoYXVjX2Jlc3QsIGF1Y194Z2IpKSAlPiUgCiAga25pdHI6OmthYmxlKGNhcHRpb24gPSAiVGFibGUgMzogQVVDIG9uIFRlc3QgRGF0YSIpCgpgYGAKCsSQ4bq/biDEkcOieSBjaMO6bmcgdGEgY8OzIGLhurFuZyBjaOG7qW5nIMSR4buDIHRpbiBy4bqxbmcgQmVzdEF1dG9NTCBuw6puIMSRxrDhu6NjIHPhu60gZOG7pW5nLiBUdXkgbmhpw6puIMSR4buDIMSRw6FuaCBnacOhIGNoaSB0aeG6v3QgaMahbiBu4buvYSBjaMO6bmcgdGEgY8OzIHRo4buDIGto4bqjbyBzw6F0IHbDoCBzbyBzw6FuaCBraOG6oyBuxINuZyBwaMOibiBsb+G6oWkgY+G7p2EgaGFpIG3DtCBow6xuaCBuw6B5IHTGsMahbmcg4bupbmcgduG7m2kgbeG7mXQgbG/huqF0IGPDoWMgbmfGsOG7oW5nIHBow6JuIGxv4bqhaSAoY3V0b2ZmIMSRw6MgdHLDrG5oIGLDoHkgdHJvbmcgW3Bvc3QgdHLGsOG7m2NdKGh0dHBzOi8vcnB1YnMuY29tL2NoaWR1bmdrdC81Nzc1MDUpKS4gVHLGsOG7m2MgaOG6v3QgY2jDum5nIHRhIHZp4bq/dCBoYWkgaMOgbSB0w61uaCByYSBjw6FjIGNo4buJIHRpw6p1IMSRw6FuaCBnacOhIG3DtCBow6xuaCBwaMOibiBsb+G6oWkgdMawxqFuZyDhu6luZyB24bubaSBt4buZdCBuZ8aw4buhbmcgbOG7sWEgY2jhu41uIGNobyBj4bqjIGhhaSBtw7QgaMOsbmg6IAoKCmBgYHtyfQojIENhbGN1bGF0ZSBtb2RlbCBwZXJmb3JtYW5jZSBieSBjdXRvZmYgc2VsZWN0ZWQgZm9yIFhnYm9vc3Q6IAoKYnlDdXRvZmZfeGdiIDwtIGZ1bmN0aW9uKGN1dG9mZikgewoKICBwcmVkIDwtIGNhc2Vfd2hlbihwZF94Z2IgPj0gY3V0b2ZmIH4gIkJhZCIsIFRSVUUgfiAiR29vZCIpICU+JSBhcy5mYWN0b3IoKQogIAogIHRodWNfdGUgPC0gY2FzZV93aGVuKFlfdGVzdCA9PSAxIH4gIkJhZCIsIFlfdGVzdCA9PSAwIH4gIkdvb2QiKSAlPiUgYXMuZmFjdG9yKCkKICAKICBjb25mdXNpb25NYXRyaXgocHJlZCwgdGh1Y190ZSwgcG9zaXRpdmUgPSAiQmFkIikgLT4gY20KICAKICBjbSR0YWJsZSAlPiUgYXMudmVjdG9yKCkgLT4gYmcKICBjbSRvdmVyYWxsICU+JSBhcy52ZWN0b3IoKSAtPiBhY2MKICBjbSRieUNsYXNzICU+JSBhcy52ZWN0b3IoKSAtPiBzZW4KICAKICBkYXRhLmZyYW1lKEJCID0gYmdbMV0sIEJHID0gYmdbMl0sIEdCID0gYmdbM10sIEdHID0gYmdbNF0sIEFjY3VyYWN5ID0gYWNjWzFdLAogICAgICAgICAgICAgS2FwcGEgPSBhY2NbMl0sIFJlY2FsbCA9IHNlblsxXSwgU3BlY2lmaWNpdHkgPSBzZW5bMl0sIEN1dG9mZiA9IGN1dG9mZikgLT4gbW9kZWxfcGVyQ3V0b2ZmCiAgCiAgcmV0dXJuKG1vZGVsX3BlckN1dG9mZikKICAKfQoKIyBDYWxjdWxhdGUgbW9kZWwgcGVyZm9ybWFuY2UgYnkgY3V0b2ZmIHNlbGVjdGVkIGZvciBiZXN0IEF1dG8gTUw6IAoKCgpieUN1dG9mZl9iZXN0IDwtIGZ1bmN0aW9uKGN1dG9mZikgewogIAogIHByZWQgPC0gY2FzZV93aGVuKHBkX2Jlc3QgPj0gY3V0b2ZmIH4gIkJhZCIsIFRSVUUgfiAiR29vZCIpICU+JSBhcy5mYWN0b3IoKQogIAogIHRodWNfdGUgPC0gY2FzZV93aGVuKFlfdGVzdCA9PSAxIH4gIkJhZCIsIFlfdGVzdCA9PSAwIH4gIkdvb2QiKSAlPiUgYXMuZmFjdG9yKCkKICAKICBjb25mdXNpb25NYXRyaXgocHJlZCwgdGh1Y190ZSwgcG9zaXRpdmUgPSAiQmFkIikgLT4gY20KICAKICBjbSR0YWJsZSAlPiUgYXMudmVjdG9yKCkgLT4gYmcKICBjbSRvdmVyYWxsICU+JSBhcy52ZWN0b3IoKSAtPiBhY2MKICBjbSRieUNsYXNzICU+JSBhcy52ZWN0b3IoKSAtPiBzZW4KICAKICBkYXRhLmZyYW1lKEJCID0gYmdbMV0sIEJHID0gYmdbMl0sIEdCID0gYmdbM10sIEdHID0gYmdbNF0sIEFjY3VyYWN5ID0gYWNjWzFdLAogICAgICAgICAgICAgS2FwcGEgPSBhY2NbMl0sIFJlY2FsbCA9IHNlblsxXSwgU3BlY2lmaWNpdHkgPSBzZW5bMl0sIEN1dG9mZiA9IGN1dG9mZikgLT4gbW9kZWxfcGVyQ3V0b2ZmCiAgCiAgcmV0dXJuKG1vZGVsX3BlckN1dG9mZikKICAKfQpgYGAKClLhu5NpIHPhu60gZOG7pW5nIGhhaSBow6BtIG7DoHkgxJHhu4MgdMOtbmggTW9kZWwgUGVyZm9ybWFuY2UgdHLDqm4gbeG7mXQgbG/huqF0IGdpw6EgdHLhu4sgY+G7p2EgQ3V0b2ZmIGNobyBj4bqjIGhhaSBtw7QgaMOsbmg6IAoKYGBge3J9CiMgQSByYW5nZSBvZiBjdXRvZmZzOiAKY3V0b2ZmcyA8LSBzZXEoMC4wNSwgMC45NSwgMC4wNSkKCiMgTW9kZWwgcGVyZm9ybWFuY2UgYnkgY3V0b2ZmIGZvciB0aGUgdHdvIG1vZGVsczogCmxhcHBseShjdXRvZmZzLCBieUN1dG9mZl94Z2IpIC0+IHBlcmZvcm1hbmNlX2N1dG9mZl94Z2IKbGFwcGx5KGN1dG9mZnMsIGJ5Q3V0b2ZmX2Jlc3QpIC0+IHBlcmZvcm1hbmNlX2N1dG9mZl9iZXN0CgojIENvbnZlcnQgdG8gREYgYW5kIGNvbWJpbmUgcmVzdWx0czogCmRvLmNhbGwoImJpbmRfcm93cyIsIHBlcmZvcm1hbmNlX2N1dG9mZl94Z2IpIC0+IHBlcmZvcm1hbmNlX2N1dG9mZl94Z2IKZG8uY2FsbCgiYmluZF9yb3dzIiwgcGVyZm9ybWFuY2VfY3V0b2ZmX2Jlc3QpIC0+IHBlcmZvcm1hbmNlX2N1dG9mZl9iZXN0CgoKYmluZF9yb3dzKHBlcmZvcm1hbmNlX2N1dG9mZl9iZXN0ICU+JSBtdXRhdGUoTW9kZWwgPSAiQmVzdEF1dG9NTCIpLCAKICAgICAgICAgIHBlcmZvcm1hbmNlX2N1dG9mZl94Z2IgJT4lIG11dGF0ZShNb2RlbCA9ICJYZ2Jvb3N0IikpIC0+IGRmX2NvbXBhcmlzaW9uCgoKIyBDb21wYXJlIG1vZGVsIHBlcmZvcm1hbmNlIGJ5IHBsb3Q6IAoKCm15X2NvbG9ycyA8LSBjKCIjZTQxYTFjIiwgIiMzNzdlYjgiKQp0aGVtZV9zZXQodGhlbWVfZ3JheSgpKQoKZGZfY29tcGFyaXNpb24gJT4lIAogIHNlbGVjdCg1OjEwKSAlPiUgCiAgZ2F0aGVyKE1ldHJpYywgVmFsdWUsIC1DdXRvZmYsIC1Nb2RlbCkgJT4lIAogIGdncGxvdChhZXMoQ3V0b2ZmLCBWYWx1ZSwgY29sb3IgPSBNb2RlbCkpICsgCiAgZ2VvbV9saW5lKCkgKyAKICBnZW9tX3BvaW50KCkgKyAKICBzY2FsZV9jb2xvcl9tYW51YWwodmFsdWVzID0gbXlfY29sb3JzKSArIAogIGZhY2V0X3dyYXAofiBNZXRyaWMsIHNjYWxlcyA9ICJmcmVlIikgKyAKICB0aGVtZShsZWdlbmQucG9zaXRpb24gPSAidG9wIikgKyAKICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzID0gc2NhbGVzOjpwZXJjZW50KSArIAogIGxhYnMoeCA9IE5VTEwsIHkgPSBOVUxMLCAKICAgICAgIHRpdGxlID0gIkZpZ3VyZSAxOiBNb2RlbCBQZXJmb3JtYW5jZSBiZXR3ZWVuIEF1dG9NTCBhbmQgWGdib29zdCBieSBDdXRvZmYiLCAKICAgICAgIHN1YnRpdGxlID0gIkRhdGEgU291cmNlOiBodHRwOi8vd3d3LmNyZWRpdHJpc2thbmFseXRpY3MubmV0IikgLT4gZjEKCgpsaWJyYXJ5KHBsb3RseSkKZ2dwbG90bHkoZjEpCmBgYAoKUXVhIEZpZ3VyZSAxIGNow7puZyB0YSB0aOG6pXkgcuG6sW5nIOG7nyBj4bqjIDQgdGnDqnUgY2jDrSB0aMOsIEJlc3RBdXRvTUwgY2FvIGjGoW4gWGdib29zdC4gTmdvw6BpIHJhIGPDsyBoYWkga+G6v3QgcXXhuqMgxJHDoW5nIGNow7ogw70gbMOgOiAKCjEuIOG7niBjw6FjIG5nxrDhu6FuZyB0aOG6pXAgaMahbiAwLjIgdGjDrCBSZWNhbGwgLSB04bupYyBsw6Aga2jhuqMgbsSDbmcgbcO0IGjDrG5oIGThu7EgYsOhbyBjaMOtbmggeMOhYyBjw6FjIHRyxrDhu51uZyBo4bujcCB44bqldSAoQmFkLCBoYXkga8OtIGhp4buHdSBsw6AgQiB24bubaSBow6BtIMO9IHLhurFuZyBraMOhY2ggaMOgbmcgc+G6vSBkZWZhdWx0KSB0aMOsIGhhaSBtw7QgaMOsbmgga2jDtG5nIGPDsyBz4buxIGtow6FjIGJp4buHdCDEkcOhbmcga+G7gyBuaMawbmcgdOG7qyBzYXUgbmfGsOG7oW5nIG7DoHkgdGjDrCBz4buxIGNow6puaCBs4buHY2ggbMOgIMSRw6FuZyBr4buDIHbDoCBjw6BuZyBu4bubaSBy4buZbmcuIMSQaeG7gXUgbsOgeSBsw6AgYuG6sW5nIGNo4bupbmcgY2jhuq9jIGNo4bqvbiBy4bqxbmcgQmVzdEF1dG9NTCBu4bq/dSDEkcaw4bujYyBz4butIGThu6VuZyBjaG8gbeG7pWMgxJHDrWNoIHBow6JuIGxv4bqhaSB0aMOsIHPhur0gbMOgIG3DtCBow6xuaCBtYW5nIGzhuqFpIGzhu6NpIG5odeG6rW4gY2FvIGjGoW4uIAoKMi4gQWNjdXJhY3kgbMOgIGPDsyBow6xuaCBjaOG7ryBVIG5nxrDhu6NjIGjDoG0gw70gcuG6sW5nIHbhu5tpIGPhuqMgaGFpIG3DtCBow6xuaCB0aMOsIHPhur0gdOG7k24gdOG6oWkgbeG7mXQgbmfGsOG7oW5nIHBow6JuIGxv4bqhaSBtw6AgdOG7kWkgxJFhIGjDs2EgY2jhu4kgdGnDqnUgbsOgeS4gxJBp4buBdSBuw6B5IGzDoCBuaOG6pXQgcXXDoW4gduG7m2kgbmd1ecOqbiBsw60gKipz4buxIMSRw6FuaCDEkeG7lWkgY+G7p2EgY8OhYyBjaOG7iSB0acOqdSoqLiBD4bulIHRo4buDIGjGoW4gY2jDum5nIHRhIGPDsyB0aOG7gyB0aOG6pXkgcuG6sW5nIFJlY2FsbCB2w6AgU3BlY2lmaWNpdHkgY8OzIHh1IGjGsOG7m25nIGJp4bq/biDEkeG7lWkgbmfGsOG7o2MgbmhhdSBraGkgdGhheSDEkeG7lWkgbmfGsOG7oW5nIHBow6JuIGxv4bqhaS4gCgpDaMO6bmcgdGEgY8OzIHRo4buDIGRp4buFbiDEkeG6oXQgYuG6sW5nIG3hu5l0IGPDoWNoIHRydXnhu4FuIHThuqNpIGtow6FjIGThu4UgaGnhu4N1IGjGoW4gduG7m2kgbmjhu69uZyBuZ8aw4budaSBraMO0bmcgY2h1ecOqbiB24buBIGThu68gbGnhu4d1IHbDoCB0aOG7kW5nIGvDqiAodGjGsOG7nW5nIGjhu40gbMOgIGLhuq1jIHF14bqjbiBsw60gLSBib3NzIHbDoCB0aMaw4budbmcgaOG7jSB0aMaw4budbmcgY8OzIHRp4bq/bmcgbsOzaSBxdXnhur90IMSR4buLbmggdHJvbmcgdmnhu4djIHPhu60gZOG7pW5nIC8gbOG7sWEgY2jhu41uKSBi4bqxbmcgY8O0bmcgY+G7pSBow6xuaCDhuqNuaCBzYXUgKEZpZ3VyZSAyKTogCgoKYGBge3J9CgpkZl9jb21wYXJpc2lvbiAlPiUgCiAgc2VsZWN0KC1jKDU6OCkpICU+JSAKICBnYXRoZXIoTWV0cmljLCBWYWx1ZSwgLUN1dG9mZiwgLU1vZGVsKSAlPiUgCiAgZ2dwbG90KGFlcyhDdXRvZmYsIFZhbHVlLCBjb2xvciA9IE1vZGVsKSkgKyAKICBnZW9tX2xpbmUoKSArIAogIGdlb21fcG9pbnQoKSArIAogIHNjYWxlX2NvbG9yX21hbnVhbCh2YWx1ZXMgPSBteV9jb2xvcnMpICsgCiAgZmFjZXRfd3JhcCh+IE1ldHJpYywgc2NhbGVzID0gImZyZWUiKSArIAogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJ0b3AiKSArIAogIGxhYnMoeCA9IE5VTEwsIHkgPSBOVUxMLCAKICAgICAgIHRpdGxlID0gIkZpZ3VyZSAyOiBNb2RlbCBQZXJmb3JtYW5jZSBiZXR3ZWVuIEF1dG9NTCBhbmQgWGdib29zdCBieSBDdXRvZmYiLCAKICAgICAgIHN1YnRpdGxlID0gIkRhdGEgU291cmNlOiBodHRwOi8vd3d3LmNyZWRpdHJpc2thbmFseXRpY3MubmV0IikgLT4gZjIKCmdncGxvdGx5KGYyKQoKCmBgYAoK4bueIMSRw6J5LCB2w60gZOG7pSwgQkIgaMOgbSDDvSBsw6AgY8OhYyBraMOhY2ggaMOgbmcgeOG6pXUgKEJhZCkgxJHGsOG7o2MgcGjDom4gbG/huqFpIMSRw7puZyB0aMOgbmgga2jDoWNoIGjDoG5nIHjhuqV1IChCYWQpLiBDw6FjIGvDrSBoaeG7h3Uga2jDoWMgbmjGsCBCRywgR0IsIHbDoCBHRyDEkcaw4bujYyBkaeG7hW4gZ2nhuqNpIHTGsMahbmcgdOG7sS4gUsO1IHLDoG5nIHThu6sgRmlndXJlIDIgY2jDum5nIHRhIHRo4bqleSBCZXN0QXV0b01MIHBow6JuIGxv4bqhaSB2w6AgZOG7sSBiw6FvIG5ow7NtIGtow6FjaCBow6BuZyB44bqldSBjaMOtbmggeMOhYyBoxqFuIOG7nyBt4buNaSBuZ8aw4buhbmcgc28gduG7m2kgWGdib29zdC4gCgojIEFuZCBNb3JlIGFib3V0IGgybwoK4bueIHRyw6puIGNow7puZyBz4butIGThu6VuZyB0aMawIHZp4buHbiAqKnhnYm9vc3QqKiDEkeG7gyB0aOG7sWMgaGnhu4duIGh14bqtbiBsdXnhu4duIFhnYm9vc3QuIEPDsyB0aOG7gyB0aOG6pXkgaMOgbSAqKnhnYm9vc3QoKSoqIMSRw7JpIGjhu49pIHLhurFuZyBk4buvIGxp4buHdSBi4bqvdCBideG7mWMgcGjhuqNpIOG7nyBk4bqhbmcgbnVtZXJpYy4gxJDDsyBsw6AgbMOtIGRvIG3DoCBjaMO6bmcgdGEgY+G6p24gcGjhuqNpIHRo4buxYyBoaeG7h24gdGjDqm0gdGjhu6cgdOG7pWMgb25lLWhvdCBlbmNvZGluZy4gQ2jDum5nIHRhIGPDsyB0aOG7gyBz4butIGThu6VuZyBow6BtICoqaDJvLnhnYm9vc3QoKSoqIMSR4buDIHRo4buxYyBoaeG7h24gdGh14bqtdCB0b8OhbiBuw6B5IG3DoCBraMO0bmcgY+G6p24gcGjhuqNpIGPhuqduIMSR4bq/biBvbmUtaG90IGVuY29kaW5nLiBIxrDhu5tuZyBk4bqrbiBjaGkgdGnhur90IHbhu4Egdmnhu4djIHRo4buxYyBoaeG7h24gdGh14bqtdCB0b8OhbiBuw6B5IGPDsyB0aOG7gyB0aGFtIGto4bqjbyBb4bufIMSRw6J5XShodHRwczovL2dpdGh1Yi5jb20vaDJvYWkvaDJvLXR1dG9yaWFscy9ibG9iL21hc3Rlci90dXRvcmlhbHMvZW5zZW1ibGVzLXN0YWNraW5nL3N0YWNrZWRfZW5zZW1ibGVfaDJvX3hnYm9vc3QuUm1kKSBjaG8gUiB2w6AgW+G7nyDEkcOieV0oaHR0cHM6Ly93d3cuaDJvLmFpL2Jsb2cveGdib29zdC1pbi1oMm8tbWFjaGluZS1sZWFybmluZy1wbGF0Zm9ybS8pIGNobyBQeXRob24uIAoKCiMgUmVmZXJlbmNlcwoxLiBbQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmc6IE1ldGhvZHMsIFN5c3RlbXMsIENoYWxsZW5nZXNdKGh0dHBzOi8vd3d3LmFtYXpvbi5jb20vQXV0b21hdGVkLU1hY2hpbmUtTGVhcm5pbmctQ2hhbGxlbmdlcy1TcHJpbmdlci9kcC8zMDMwMDUzMTcyKS4gCjIuIFtQcmFjdGljYWwgQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmcgb24gQXp1cmU6IFVzaW5nIEF6dXJlIE1hY2hpbmUgTGVhcm5pbmcgdG8gUXVpY2tseSBCdWlsZCBBSSBTb2x1dGlvbnNdKGh0dHA6Ly9zaG9wLm9yZWlsbHkuY29tL3Byb2R1Y3QvMDYzNjkyMDI2OTg4NS5kbykuIAozLiBbSGFuZHMtT24gQXV0b21hdGVkIE1hY2hpbmUgTGVhcm5pbmc6IEEgYmVnaW5uZXIncyBndWlkZSB0byBidWlsZGluZyBhdXRvbWF0ZWQgbWFjaGluZSBsZWFybmluZyBzeXN0ZW1zIHVzaW5nIEF1dG9NTCBhbmQgUHl0aG9uXShodHRwczovL3d3dy5hbWF6b24uY29tL0hhbmRzLUF1dG9tYXRlZC1NYWNoaW5lLUxlYXJuaW5nLWJlZ2lubmVycy1lYm9vay9kcC9CMDdCV0tUTUtHKS4gCgoKCiMgUiBFbnZpcm9ubWVudCBhbmQgT1MKCmBgYHtyfQpzZXNzaW9uSW5mbygpCgpgYGAKCgo=