Churn predictions with keras, rsample and recipes
# Load libraries
library(keras)
library(lime)
library(tidyquant)
library(rsample)
library(recipes)
library(yardstick)
library(corrr)Preprocess Data
Prune the data
Load data and look at the structure
churn_data_raw <- readxl::read_excel("/Users/andreymarkin/Downloads/WA_Fn-UseC_-Telco-Customer-Churn.xlsx")
glimpse(churn_data_raw)Deselect customerID and drop NAs
churn_data_tbl <- churn_data_raw %>%
select(-customerID) %>%
drop_na() %>%
select(Churn, everything())Split into train/test splits
We have a new package, rsample, which is very useful for sampling methods. It has the initial_split() function for splitting data sets into training and testing sets. The return is a special rsplit object.
set.seed(100)
train_test_split <- initial_split(churn_data_tbl, prop = 0.8)
train_test_split<5626/1406/7032>We can retrieve our training and testing sets using training() and testing() functions
train_tbl <- training(train_test_split)
test_tbl <- training(train_test_split)Exploration: what other trainsformations are needed?
DISCRETIZE THE “TENURE” FEATURE
churn_data_tbl %>%
ggplot(aes(x = tenure)) +
geom_histogram()
churn_data_tbl %>%
ggplot(aes(x = tenure)) +
geom_histogram(bins = 6)TRANSFORM THE “TOTALCHARGES” FEATURE
churn_data_tbl %>%
ggplot(aes(x = TotalCharges)) +
geom_histogram(bins = 100)
churn_data_tbl %>%
ggplot(aes(x = log(TotalCharges))) +
geom_histogram(bins = 100)Pro Tip: A quick test is to see if the log transformation increases the magnitude of the correlation between “TotalCharges” and “Churn”. We’ll use a few dplyr operations along with the corrr package to perform a quick correlation.
correlate(): performs tidy correlations on numeric datafocus(): Similar toselect(). Takes columns and focuses on only the rows/columns of importance.fashion(): Makes the formatting easier to read.
ONE-HOT ENCODING
FEATURE SCALING
Preprocessing with recipies
Create a recipe
A “recipe” is nothing more than a series of steps you would like to perform on the training, testing and/or validation sets.
We use the recipe() function to implement our preprocessing steps. The function takes two arguments object = Churn ~ . and data argument data = train_tbl.
A recipe is not very useful until we add “steps”, which are used to transform the data. For our model we’ll use:
step_discretize()withoptions = list(cut = 6)to cut continuous data into six binsstep_log()to log transform “TotalCharges”step_dummy()to one-hot encode categorical features.step_center()to mean-center datastep_scale()to scale the data
The last steps is to prepare recipe with prep() function. This step is used to “estimate the required parameters from a training set that can later be applied to other data sets”. This is important for centering and scaling and other functions that use parameters defined from the training set.
#create recipe
rec_obj <- recipe(Churn ~ ., data = train_tbl) %>%
step_discretize(tenure, options = list(cuts = 6)) %>%
step_log(TotalCharges) %>%
step_dummy(all_nominal(), -all_outcomes()) %>%
step_center(all_predictors(), -all_outcomes()) %>%
step_scale(all_predictors(), -all_outcomes()) %>%
prep(data = train_tbl)rec_objData Recipe
Inputs:
role #variables
outcome 1
predictor 19
Training data contained 5626 data points and no missing data.
Operations:
Dummy variables from tenure [trained]
Log transformation on TotalCharges [trained]
Dummy variables from gender, Partner, Dependents, tenure, PhoneService, MultipleLines, ... [trained]
Centering for SeniorCitizen, MonthlyCharges, TotalCharges, gender_Male, ... [trained]
Scaling for SeniorCitizen, MonthlyCharges, TotalCharges, gender_Male, ... [trained]Baking with your recipe
We can apply the recipe to any dataset with bake() function and it processes the data following our recipe steps.
glimpse(x_train_tbl)
Observations: 5,626
Variables: 35
$ SeniorCitizen <dbl> -0.4351959, -0.4351959, -0.4351959, -0.4351959, -0.4351959...
$ MonthlyCharges <dbl> -1.157597168, -0.260133339, -0.745293674, 0.195221925, 1.1...
$ TotalCharges <dbl> -2.275819130, 0.389259098, 0.372464296, -1.231480878, -0.1...
$ gender_Male <dbl> -1.0016900, 0.9981354, 0.9981354, -1.0016900, -1.0016900, ...
$ Partner_Yes <dbl> 1.0262054, -0.9742906, -0.9742906, -0.9742906, -0.9742906,...
$ Dependents_Yes <dbl> -0.6507747, -0.6507747, -0.6507747, -0.6507747, -0.6507747...
$ tenure_bin1 <dbl> 2.1677790, -0.4612196, -0.4612196, 2.1677790, -0.4612196, ...
$ tenure_bin2 <dbl> -0.4389453, -0.4389453, -0.4389453, -0.4389453, 2.2777833,...
$ tenure_bin3 <dbl> -0.4481273, -0.4481273, -0.4481273, -0.4481273, -0.4481273...
$ tenure_bin4 <dbl> -0.4509837, 2.2169809, 2.2169809, -0.4509837, -0.4509837, ...
$ tenure_bin5 <dbl> -0.4498419, -0.4498419, -0.4498419, -0.4498419, -0.4498419...
$ tenure_bin6 <dbl> -0.4337508, -0.4337508, -0.4337508, -0.4337508, -0.4337508...
$ PhoneService_Yes <dbl> -3.0407367, 0.3288092, -3.0407367, 0.3288092, 0.3288092, 0...
$ MultipleLines_No.phone.service <dbl> 3.0407367, -0.3288092, 3.0407367, -0.3288092, -0.3288092, ...
$ MultipleLines_Yes <dbl> -0.8571364, -0.8571364, -0.8571364, -0.8571364, 1.1664681,...
$ InternetService_Fiber.optic <dbl> -0.8884255, -0.8884255, -0.8884255, 1.1253867, 1.1253867, ...
$ InternetService_No <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ OnlineSecurity_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ OnlineSecurity_Yes <dbl> -0.6369654, 1.5696648, 1.5696648, -0.6369654, -0.6369654, ...
$ OnlineBackup_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ OnlineBackup_Yes <dbl> 1.3771987, -0.7259826, -0.7259826, -0.7259826, -0.7259826,...
$ DeviceProtection_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ DeviceProtection_Yes <dbl> -0.7259826, 1.3771987, 1.3771987, -0.7259826, 1.3771987, -...
$ TechSupport_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ TechSupport_Yes <dbl> -0.6358628, -0.6358628, 1.5723867, -0.6358628, -0.6358628,...
$ StreamingTV_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ StreamingTV_Yes <dbl> -0.7917326, -0.7917326, -0.7917326, -0.7917326, 1.2628282,...
$ StreamingMovies_No.internet.service <dbl> -0.5272627, -0.5272627, -0.5272627, -0.5272627, -0.5272627...
$ StreamingMovies_Yes <dbl> -0.797388, -0.797388, -0.797388, -0.797388, 1.253872, -0.7...
$ Contract_One.year <dbl> -0.5156834, 1.9388298, 1.9388298, -0.5156834, -0.5156834, ...
$ Contract_Two.year <dbl> -0.5618358, -0.5618358, -0.5618358, -0.5618358, -0.5618358...
$ PaperlessBilling_Yes <dbl> 0.8330334, -1.2002187, -1.2002187, 0.8330334, 0.8330334, 0...
$ PaymentMethod_Credit.card..automatic. <dbl> -0.5231315, -0.5231315, -0.5231315, -0.5231315, -0.5231315...
$ PaymentMethod_Electronic.check <dbl> 1.4154085, -0.7063842, -0.7063842, 1.4154085, 1.4154085, -...
$ PaymentMethod_Mailed.check <dbl> -0.5517013, 1.8122527, -0.5517013, -0.5517013, -0.5517013,...Don’t forget the target
y_train_vec <- ifelse(pull(train_tbl, Churn) == "Yes", 1, 0)
y_test_vec <- ifelse(pull(test_tbl, Churn) == "Yes", 1, 0)#Model Customer Churn in Keras
model_keras <- keras_model_sequential() %>%
layer_dense(
units = 64,
kernel_initializer = "uniform",
activation = "relu",
input_shape = ncol(x_train_tbl)
) %>%
layer_dropout(rate = 0.3) %>%
layer_dense(
units = 64,
kernel_initializer = "uniform",
activation = "relu"
) %>%
layer_dropout(rate = 0.3) %>%
layer_dense(
units = 1,
kernel_initializer = "uniform",
activation = "sigmoid")
model_keras %>% compile(
loss = "binary_crossentropy",
optimizer = "adam",
metrics = "accuracy"
)history <- model_keras %>% fit(
as.matrix(x_train_tbl),
y_train_vec,
batch_size = 32,
epochs = 30,
validation_split = 0.3
)
Train on 3938 samples, validate on 1688 samples
Epoch 1/30
32/3938 [..............................] - ETA: 137s - loss: 0.6927 - acc: 0.5312
256/3938 [>.............................] - ETA: 16s - loss: 0.6868 - acc: 0.6914
480/3938 [==>...........................] - ETA: 8s - loss: 0.6790 - acc: 0.7063
736/3938 [====>.........................] - ETA: 5s - loss: 0.6655 - acc: 0.7079
992/3938 [======>.......................] - ETA: 3s - loss: 0.6479 - acc: 0.7026
1248/3938 [========>.....................] - ETA: 2s - loss: 0.6170 - acc: 0.7139
1536/3938 [==========>...................] - ETA: 2s - loss: 0.5892 - acc: 0.7194
1824/3938 [============>.................] - ETA: 1s - loss: 0.5630 - acc: 0.7270
2080/3938 [==============>...............] - ETA: 1s - loss: 0.5482 - acc: 0.7279
2368/3938 [=================>............] - ETA: 1s - loss: 0.5357 - acc: 0.7302
2656/3938 [===================>..........] - ETA: 0s - loss: 0.5220 - acc: 0.7372
2912/3938 [=====================>........] - ETA: 0s - loss: 0.5140 - acc: 0.7411
3136/3938 [======================>.......] - ETA: 0s - loss: 0.5189 - acc: 0.7414
3424/3938 [=========================>....] - ETA: 0s - loss: 0.5146 - acc: 0.7427
3712/3938 [===========================>..] - ETA: 0s - loss: 0.5075 - acc: 0.7470
3938/3938 [==============================] - 2s - loss: 0.5015 - acc: 0.7499 - val_loss: 0.4425 - val_acc: 0.7950
Epoch 2/30
32/3938 [..............................] - ETA: 0s - loss: 0.6743 - acc: 0.6875
256/3938 [>.............................] - ETA: 0s - loss: 0.4387 - acc: 0.7969
512/3938 [==>...........................] - ETA: 0s - loss: 0.3935 - acc: 0.8184
768/3938 [====>.........................] - ETA: 0s - loss: 0.4096 - acc: 0.8099
992/3938 [======>.......................] - ETA: 0s - loss: 0.4089 - acc: 0.8115
1248/3938 [========>.....................] - ETA: 0s - loss: 0.4158 - acc: 0.8053
1536/3938 [==========>...................] - ETA: 0s - loss: 0.4316 - acc: 0.7995
1824/3938 [============>.................] - ETA: 0s - loss: 0.4364 - acc: 0.7977
2080/3938 [==============>...............] - ETA: 0s - loss: 0.4344 - acc: 0.8005
2336/3938 [================>.............] - ETA: 0s - loss: 0.4335 - acc: 0.7992
2592/3938 [==================>...........] - ETA: 0s - loss: 0.4297 - acc: 0.8032
2848/3938 [====================>.........] - ETA: 0s - loss: 0.4248 - acc: 0.8041
3104/3938 [======================>.......] - ETA: 0s - loss: 0.4239 - acc: 0.8032
3360/3938 [========================>.....] - ETA: 0s - loss: 0.4259 - acc: 0.8027
3712/3938 [===========================>..] - ETA: 0s - loss: 0.4254 - acc: 0.8025
3936/3938 [============================>.] - ETA: 0s - loss: 0.4276 - acc: 0.8011
3938/3938 [==============================] - 1s - loss: 0.4277 - acc: 0.8009 - val_loss: 0.4255 - val_acc: 0.7992
Epoch 3/30
32/3938 [..............................] - ETA: 0s - loss: 0.3049 - acc: 0.9688
288/3938 [=>............................] - ETA: 0s - loss: 0.3856 - acc: 0.8611
544/3938 [===>..........................] - ETA: 0s - loss: 0.3964 - acc: 0.8327
800/3938 [=====>........................] - ETA: 0s - loss: 0.4053 - acc: 0.8187
1088/3938 [=======>......................] - ETA: 0s - loss: 0.4052 - acc: 0.8153
1376/3938 [=========>....................] - ETA: 0s - loss: 0.4155 - acc: 0.8067
1632/3938 [===========>..................] - ETA: 0s - loss: 0.4140 - acc: 0.8094
1888/3938 [=============>................] - ETA: 0s - loss: 0.4119 - acc: 0.8093
2176/3938 [===============>..............] - ETA: 0s - loss: 0.4152 - acc: 0.8061
2464/3938 [=================>............] - ETA: 0s - loss: 0.4121 - acc: 0.8056
2720/3938 [===================>..........] - ETA: 0s - loss: 0.4119 - acc: 0.8051
3008/3938 [=====================>........] - ETA: 0s - loss: 0.4125 - acc: 0.8068
3296/3938 [========================>.....] - ETA: 0s - loss: 0.4146 - acc: 0.8061
3552/3938 [==========================>...] - ETA: 0s - loss: 0.4161 - acc: 0.8052
3840/3938 [============================>.] - ETA: 0s - loss: 0.4190 - acc: 0.8018
3938/3938 [==============================] - 0s - loss: 0.4187 - acc: 0.8012 - val_loss: 0.4240 - val_acc: 0.8081
Epoch 4/30
32/3938 [..............................] - ETA: 0s - loss: 0.5363 - acc: 0.7188
288/3938 [=>............................] - ETA: 0s - loss: 0.4336 - acc: 0.7778
512/3938 [==>...........................] - ETA: 0s - loss: 0.3953 - acc: 0.8145
736/3938 [====>.........................] - ETA: 0s - loss: 0.3926 - acc: 0.8261
992/3938 [======>.......................] - ETA: 0s - loss: 0.4044 - acc: 0.8175
1248/3938 [========>.....................] - ETA: 0s - loss: 0.4082 - acc: 0.8125
1504/3938 [==========>...................] - ETA: 0s - loss: 0.4044 - acc: 0.8138
1792/3938 [============>.................] - ETA: 0s - loss: 0.4130 - acc: 0.8047
2048/3938 [==============>...............] - ETA: 0s - loss: 0.4103 - acc: 0.8057
2080/3938 [==============>...............] - ETA: 0s - loss: 0.4123 - acc: 0.8043
2368/3938 [=================>............] - ETA: 0s - loss: 0.4173 - acc: 0.8032
2624/3938 [==================>...........] - ETA: 0s - loss: 0.4198 - acc: 0.8011
2880/3938 [====================>.........] - ETA: 0s - loss: 0.4181 - acc: 0.8035
3136/3938 [======================>.......] - ETA: 0s - loss: 0.4136 - acc: 0.8071
3424/3938 [=========================>....] - ETA: 0s - loss: 0.4140 - acc: 0.8055
3680/3938 [===========================>..] - ETA: 0s - loss: 0.4137 - acc: 0.8049
3938/3938 [==============================] - 1s - loss: 0.4148 - acc: 0.8040 - val_loss: 0.4241 - val_acc: 0.8051
Epoch 5/30
32/3938 [..............................] - ETA: 0s - loss: 0.2608 - acc: 0.9062
288/3938 [=>............................] - ETA: 0s - loss: 0.4168 - acc: 0.7917
576/3938 [===>..........................] - ETA: 0s - loss: 0.4082 - acc: 0.7969
832/3938 [=====>........................] - ETA: 0s - loss: 0.4101 - acc: 0.7993
1088/3938 [=======>......................] - ETA: 0s - loss: 0.4135 - acc: 0.8015
1344/3938 [=========>....................] - ETA: 0s - loss: 0.4071 - acc: 0.8058
1632/3938 [===========>..................] - ETA: 0s - loss: 0.4049 - acc: 0.8076
1952/3938 [=============>................] - ETA: 0s - loss: 0.4162 - acc: 0.8017
2208/3938 [===============>..............] - ETA: 0s - loss: 0.4151 - acc: 0.8025
2528/3938 [==================>...........] - ETA: 0s - loss: 0.4098 - acc: 0.8066
2784/3938 [====================>.........] - ETA: 0s - loss: 0.4162 - acc: 0.8014
3072/3938 [======================>.......] - ETA: 0s - loss: 0.4143 - acc: 0.8034
3360/3938 [========================>.....] - ETA: 0s - loss: 0.4154 - acc: 0.8012
3648/3938 [==========================>...] - ETA: 0s - loss: 0.4142 - acc: 0.8035
3904/3938 [============================>.] - ETA: 0s - loss: 0.4139 - acc: 0.8030
3938/3938 [==============================] - 0s - loss: 0.4125 - acc: 0.8035 - val_loss: 0.4221 - val_acc: 0.8004
Epoch 6/30
32/3938 [..............................] - ETA: 0s - loss: 0.4348 - acc: 0.8750
288/3938 [=>............................] - ETA: 0s - loss: 0.4199 - acc: 0.7917
544/3938 [===>..........................] - ETA: 0s - loss: 0.3904 - acc: 0.8180
800/3938 [=====>........................] - ETA: 0s - loss: 0.3933 - acc: 0.8150
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3976 - acc: 0.8125
1344/3938 [=========>....................] - ETA: 0s - loss: 0.4058 - acc: 0.8118
1632/3938 [===========>..................] - ETA: 0s - loss: 0.4045 - acc: 0.8094
1920/3938 [=============>................] - ETA: 0s - loss: 0.4038 - acc: 0.8068
2176/3938 [===============>..............] - ETA: 0s - loss: 0.4062 - acc: 0.8065
2464/3938 [=================>............] - ETA: 0s - loss: 0.4113 - acc: 0.8060
2816/3938 [====================>.........] - ETA: 0s - loss: 0.4095 - acc: 0.8058
3104/3938 [======================>.......] - ETA: 0s - loss: 0.4161 - acc: 0.8019
3392/3938 [========================>.....] - ETA: 0s - loss: 0.4142 - acc: 0.8016
3616/3938 [==========================>...] - ETA: 0s - loss: 0.4156 - acc: 0.8001
3904/3938 [============================>.] - ETA: 0s - loss: 0.4108 - acc: 0.8030
3938/3938 [==============================] - 0s - loss: 0.4115 - acc: 0.8024 - val_loss: 0.4228 - val_acc: 0.8004
Epoch 7/30
32/3938 [..............................] - ETA: 0s - loss: 0.4770 - acc: 0.8125
256/3938 [>.............................] - ETA: 0s - loss: 0.4304 - acc: 0.8047
480/3938 [==>...........................] - ETA: 0s - loss: 0.4251 - acc: 0.8083
704/3938 [====>.........................] - ETA: 0s - loss: 0.4542 - acc: 0.7983
960/3938 [======>.......................] - ETA: 0s - loss: 0.4449 - acc: 0.7990
1248/3938 [========>.....................] - ETA: 0s - loss: 0.4311 - acc: 0.8069
1536/3938 [==========>...................] - ETA: 0s - loss: 0.4276 - acc: 0.8086
1824/3938 [============>.................] - ETA: 0s - loss: 0.4199 - acc: 0.8098
2112/3938 [===============>..............] - ETA: 0s - loss: 0.4175 - acc: 0.8068
2400/3938 [=================>............] - ETA: 0s - loss: 0.4141 - acc: 0.8083
2688/3938 [===================>..........] - ETA: 0s - loss: 0.4129 - acc: 0.8080
2976/3938 [=====================>........] - ETA: 0s - loss: 0.4097 - acc: 0.8091
3264/3938 [=======================>......] - ETA: 0s - loss: 0.4084 - acc: 0.8097
3520/3938 [=========================>....] - ETA: 0s - loss: 0.4117 - acc: 0.8088
3808/3938 [============================>.] - ETA: 0s - loss: 0.4118 - acc: 0.8070
3938/3938 [==============================] - 0s - loss: 0.4118 - acc: 0.8078 - val_loss: 0.4247 - val_acc: 0.8021
Epoch 8/30
32/3938 [..............................] - ETA: 0s - loss: 0.4435 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.4186 - acc: 0.7986
512/3938 [==>...........................] - ETA: 0s - loss: 0.4204 - acc: 0.7969
768/3938 [====>.........................] - ETA: 0s - loss: 0.4020 - acc: 0.8073
1024/3938 [======>.......................] - ETA: 0s - loss: 0.4020 - acc: 0.8086
1248/3938 [========>.....................] - ETA: 0s - loss: 0.3978 - acc: 0.8117
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3911 - acc: 0.8205
1760/3938 [============>.................] - ETA: 0s - loss: 0.3899 - acc: 0.8233
2048/3938 [==============>...............] - ETA: 0s - loss: 0.3985 - acc: 0.8164
2336/3938 [================>.............] - ETA: 0s - loss: 0.3987 - acc: 0.8181
2624/3938 [==================>...........] - ETA: 0s - loss: 0.4009 - acc: 0.8136
2912/3938 [=====================>........] - ETA: 0s - loss: 0.4049 - acc: 0.8101
3200/3938 [=======================>......] - ETA: 0s - loss: 0.4066 - acc: 0.8084
3488/3938 [=========================>....] - ETA: 0s - loss: 0.4050 - acc: 0.8079
3744/3938 [===========================>..] - ETA: 0s - loss: 0.4030 - acc: 0.8098
3938/3938 [==============================] - 0s - loss: 0.4054 - acc: 0.8080 - val_loss: 0.4214 - val_acc: 0.8051
Epoch 9/30
32/3938 [..............................] - ETA: 0s - loss: 0.3163 - acc: 0.8438
320/3938 [=>............................] - ETA: 0s - loss: 0.4023 - acc: 0.8219
608/3938 [===>..........................] - ETA: 0s - loss: 0.4063 - acc: 0.8026
864/3938 [=====>........................] - ETA: 0s - loss: 0.3968 - acc: 0.8171
1120/3938 [=======>......................] - ETA: 0s - loss: 0.3803 - acc: 0.8232
1344/3938 [=========>....................] - ETA: 0s - loss: 0.3926 - acc: 0.8095
1632/3938 [===========>..................] - ETA: 0s - loss: 0.3948 - acc: 0.8094
1920/3938 [=============>................] - ETA: 0s - loss: 0.4015 - acc: 0.8057
2176/3938 [===============>..............] - ETA: 0s - loss: 0.3966 - acc: 0.8107
2400/3938 [=================>............] - ETA: 0s - loss: 0.4015 - acc: 0.8079
2624/3938 [==================>...........] - ETA: 0s - loss: 0.4017 - acc: 0.8075
2912/3938 [=====================>........] - ETA: 0s - loss: 0.4039 - acc: 0.8077
3200/3938 [=======================>......] - ETA: 0s - loss: 0.4029 - acc: 0.8075
3488/3938 [=========================>....] - ETA: 0s - loss: 0.4038 - acc: 0.8068
3808/3938 [============================>.] - ETA: 0s - loss: 0.4061 - acc: 0.8049
3938/3938 [==============================] - 0s - loss: 0.4046 - acc: 0.8068 - val_loss: 0.4209 - val_acc: 0.8063
Epoch 10/30
32/3938 [..............................] - ETA: 0s - loss: 0.6744 - acc: 0.7188
288/3938 [=>............................] - ETA: 0s - loss: 0.4694 - acc: 0.7743
544/3938 [===>..........................] - ETA: 0s - loss: 0.4687 - acc: 0.7647
800/3938 [=====>........................] - ETA: 0s - loss: 0.4568 - acc: 0.7800
1088/3938 [=======>......................] - ETA: 0s - loss: 0.4477 - acc: 0.7858
1376/3938 [=========>....................] - ETA: 0s - loss: 0.4261 - acc: 0.8016
1632/3938 [===========>..................] - ETA: 0s - loss: 0.4210 - acc: 0.8015
1920/3938 [=============>................] - ETA: 0s - loss: 0.4148 - acc: 0.8052
2208/3938 [===============>..............] - ETA: 0s - loss: 0.4067 - acc: 0.8057
2496/3938 [==================>...........] - ETA: 0s - loss: 0.4137 - acc: 0.7997
2784/3938 [====================>.........] - ETA: 0s - loss: 0.4077 - acc: 0.7999
3072/3938 [======================>.......] - ETA: 0s - loss: 0.4058 - acc: 0.7998
3392/3938 [========================>.....] - ETA: 0s - loss: 0.4042 - acc: 0.8031
3712/3938 [===========================>..] - ETA: 0s - loss: 0.4054 - acc: 0.8033
3938/3938 [==============================] - 0s - loss: 0.4016 - acc: 0.8055 - val_loss: 0.4218 - val_acc: 0.8092
Epoch 11/30
32/3938 [..............................] - ETA: 0s - loss: 0.7047 - acc: 0.6250
320/3938 [=>............................] - ETA: 0s - loss: 0.4091 - acc: 0.7969
576/3938 [===>..........................] - ETA: 0s - loss: 0.4121 - acc: 0.7969
832/3938 [=====>........................] - ETA: 0s - loss: 0.4237 - acc: 0.7981
1088/3938 [=======>......................] - ETA: 0s - loss: 0.4005 - acc: 0.8153
1344/3938 [=========>....................] - ETA: 0s - loss: 0.3982 - acc: 0.8170
1632/3938 [===========>..................] - ETA: 0s - loss: 0.3996 - acc: 0.8162
1920/3938 [=============>................] - ETA: 0s - loss: 0.4004 - acc: 0.8104
2208/3938 [===============>..............] - ETA: 0s - loss: 0.4077 - acc: 0.8034
2496/3938 [==================>...........] - ETA: 0s - loss: 0.4019 - acc: 0.8077
2784/3938 [====================>.........] - ETA: 0s - loss: 0.4007 - acc: 0.8089
3072/3938 [======================>.......] - ETA: 0s - loss: 0.4040 - acc: 0.8076
3360/3938 [========================>.....] - ETA: 0s - loss: 0.4025 - acc: 0.8065
3648/3938 [==========================>...] - ETA: 0s - loss: 0.4029 - acc: 0.8076
3936/3938 [============================>.] - ETA: 0s - loss: 0.4042 - acc: 0.8056
3938/3938 [==============================] - 0s - loss: 0.4041 - acc: 0.8057 - val_loss: 0.4200 - val_acc: 0.8057
Epoch 12/30
32/3938 [..............................] - ETA: 0s - loss: 0.3940 - acc: 0.7500
256/3938 [>.............................] - ETA: 0s - loss: 0.3947 - acc: 0.7812
512/3938 [==>...........................] - ETA: 0s - loss: 0.4040 - acc: 0.7910
768/3938 [====>.........................] - ETA: 0s - loss: 0.4184 - acc: 0.7891
1024/3938 [======>.......................] - ETA: 0s - loss: 0.4110 - acc: 0.7979
1312/3938 [========>.....................] - ETA: 0s - loss: 0.4145 - acc: 0.7942
1600/3938 [===========>..................] - ETA: 0s - loss: 0.4124 - acc: 0.7956
1888/3938 [=============>................] - ETA: 0s - loss: 0.4063 - acc: 0.8008
2176/3938 [===============>..............] - ETA: 0s - loss: 0.4009 - acc: 0.8065
2432/3938 [=================>............] - ETA: 0s - loss: 0.3995 - acc: 0.8084
2688/3938 [===================>..........] - ETA: 0s - loss: 0.4031 - acc: 0.8058
2944/3938 [=====================>........] - ETA: 0s - loss: 0.4045 - acc: 0.8057
3232/3938 [=======================>......] - ETA: 0s - loss: 0.4033 - acc: 0.8063
3520/3938 [=========================>....] - ETA: 0s - loss: 0.4049 - acc: 0.8068
3840/3938 [============================>.] - ETA: 0s - loss: 0.4020 - acc: 0.8078
3938/3938 [==============================] - 0s - loss: 0.4014 - acc: 0.8085 - val_loss: 0.4234 - val_acc: 0.8051
Epoch 13/30
32/3938 [..............................] - ETA: 0s - loss: 0.3575 - acc: 0.8438
288/3938 [=>............................] - ETA: 0s - loss: 0.3721 - acc: 0.8333
576/3938 [===>..........................] - ETA: 0s - loss: 0.3844 - acc: 0.8403
832/3938 [=====>........................] - ETA: 0s - loss: 0.4049 - acc: 0.8173
1152/3938 [=======>......................] - ETA: 0s - loss: 0.4043 - acc: 0.8194
1408/3938 [=========>....................] - ETA: 0s - loss: 0.3969 - acc: 0.8232
1696/3938 [===========>..................] - ETA: 0s - loss: 0.3985 - acc: 0.8190
1984/3938 [==============>...............] - ETA: 0s - loss: 0.3936 - acc: 0.8211
2304/3938 [================>.............] - ETA: 0s - loss: 0.3974 - acc: 0.8173
2592/3938 [==================>...........] - ETA: 0s - loss: 0.4015 - acc: 0.8140
2880/3938 [====================>.........] - ETA: 0s - loss: 0.4007 - acc: 0.8153
3136/3938 [======================>.......] - ETA: 0s - loss: 0.3963 - acc: 0.8176
3392/3938 [========================>.....] - ETA: 0s - loss: 0.3945 - acc: 0.8175
3648/3938 [==========================>...] - ETA: 0s - loss: 0.3983 - acc: 0.8130
3936/3938 [============================>.] - ETA: 0s - loss: 0.4014 - acc: 0.8092
3938/3938 [==============================] - 0s - loss: 0.4015 - acc: 0.8090 - val_loss: 0.4223 - val_acc: 0.8086
Epoch 14/30
32/3938 [..............................] - ETA: 0s - loss: 0.3619 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.3496 - acc: 0.8438
544/3938 [===>..........................] - ETA: 0s - loss: 0.3827 - acc: 0.8309
800/3938 [=====>........................] - ETA: 0s - loss: 0.3905 - acc: 0.8325
1056/3938 [=======>......................] - ETA: 0s - loss: 0.4016 - acc: 0.8201
1312/3938 [========>.....................] - ETA: 0s - loss: 0.4047 - acc: 0.8125
1568/3938 [==========>...................] - ETA: 0s - loss: 0.4049 - acc: 0.8138
1888/3938 [=============>................] - ETA: 0s - loss: 0.4008 - acc: 0.8141
2112/3938 [===============>..............] - ETA: 0s - loss: 0.4004 - acc: 0.8125
2400/3938 [=================>............] - ETA: 0s - loss: 0.4036 - acc: 0.8104
2688/3938 [===================>..........] - ETA: 0s - loss: 0.4043 - acc: 0.8110
2976/3938 [=====================>........] - ETA: 0s - loss: 0.4035 - acc: 0.8112
3264/3938 [=======================>......] - ETA: 0s - loss: 0.4057 - acc: 0.8119
3520/3938 [=========================>....] - ETA: 0s - loss: 0.4041 - acc: 0.8119
3808/3938 [============================>.] - ETA: 0s - loss: 0.4017 - acc: 0.8128
3938/3938 [==============================] - 0s - loss: 0.4007 - acc: 0.8131 - val_loss: 0.4269 - val_acc: 0.8063
Epoch 15/30
32/3938 [..............................] - ETA: 0s - loss: 0.4550 - acc: 0.7500
256/3938 [>.............................] - ETA: 0s - loss: 0.3743 - acc: 0.8359
512/3938 [==>...........................] - ETA: 0s - loss: 0.3804 - acc: 0.8281
768/3938 [====>.........................] - ETA: 0s - loss: 0.3939 - acc: 0.8229
992/3938 [======>.......................] - ETA: 0s - loss: 0.4138 - acc: 0.8155
1280/3938 [========>.....................] - ETA: 0s - loss: 0.4106 - acc: 0.8172
1536/3938 [==========>...................] - ETA: 0s - loss: 0.4143 - acc: 0.8112
1792/3938 [============>.................] - ETA: 0s - loss: 0.4074 - acc: 0.8108
2048/3938 [==============>...............] - ETA: 0s - loss: 0.4051 - acc: 0.8125
2304/3938 [================>.............] - ETA: 0s - loss: 0.4035 - acc: 0.8142
2528/3938 [==================>...........] - ETA: 0s - loss: 0.4009 - acc: 0.8161
2816/3938 [====================>.........] - ETA: 0s - loss: 0.4024 - acc: 0.8157
3168/3938 [=======================>......] - ETA: 0s - loss: 0.3989 - acc: 0.8179
3456/3938 [=========================>....] - ETA: 0s - loss: 0.3965 - acc: 0.8203
3776/3938 [===========================>..] - ETA: 0s - loss: 0.3984 - acc: 0.8162
3938/3938 [==============================] - 1s - loss: 0.3983 - acc: 0.8154 - val_loss: 0.4273 - val_acc: 0.8081
Epoch 16/30
32/3938 [..............................] - ETA: 0s - loss: 0.4109 - acc: 0.7812
256/3938 [>.............................] - ETA: 0s - loss: 0.3737 - acc: 0.8281
512/3938 [==>...........................] - ETA: 0s - loss: 0.3952 - acc: 0.8086
800/3938 [=====>........................] - ETA: 0s - loss: 0.4294 - acc: 0.7775
1088/3938 [=======>......................] - ETA: 0s - loss: 0.4272 - acc: 0.7840
1344/3938 [=========>....................] - ETA: 0s - loss: 0.4226 - acc: 0.7857
1600/3938 [===========>..................] - ETA: 0s - loss: 0.4175 - acc: 0.7913
1888/3938 [=============>................] - ETA: 0s - loss: 0.4069 - acc: 0.7961
2176/3938 [===============>..............] - ETA: 0s - loss: 0.4027 - acc: 0.8015
2464/3938 [=================>............] - ETA: 0s - loss: 0.3990 - acc: 0.8060
2752/3938 [===================>..........] - ETA: 0s - loss: 0.3992 - acc: 0.8074
3008/3938 [=====================>........] - ETA: 0s - loss: 0.3973 - acc: 0.8108
3232/3938 [=======================>......] - ETA: 0s - loss: 0.3950 - acc: 0.8125
3488/3938 [=========================>....] - ETA: 0s - loss: 0.3944 - acc: 0.8125
3776/3938 [===========================>..] - ETA: 0s - loss: 0.3931 - acc: 0.8154
3938/3938 [==============================] - 0s - loss: 0.3938 - acc: 0.8172 - val_loss: 0.4290 - val_acc: 0.8063
Epoch 17/30
32/3938 [..............................] - ETA: 0s - loss: 0.5621 - acc: 0.7188
256/3938 [>.............................] - ETA: 0s - loss: 0.3686 - acc: 0.8438
480/3938 [==>...........................] - ETA: 0s - loss: 0.4148 - acc: 0.8063
736/3938 [====>.........................] - ETA: 0s - loss: 0.3931 - acc: 0.8152
960/3938 [======>.......................] - ETA: 0s - loss: 0.3946 - acc: 0.8198
1216/3938 [========>.....................] - ETA: 0s - loss: 0.3855 - acc: 0.8273
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3866 - acc: 0.8218
1824/3938 [============>.................] - ETA: 0s - loss: 0.3784 - acc: 0.8246
2112/3938 [===============>..............] - ETA: 0s - loss: 0.3762 - acc: 0.8272
2400/3938 [=================>............] - ETA: 0s - loss: 0.3848 - acc: 0.8229
2688/3938 [===================>..........] - ETA: 0s - loss: 0.3867 - acc: 0.8199
2976/3938 [=====================>........] - ETA: 0s - loss: 0.3955 - acc: 0.8159
3264/3938 [=======================>......] - ETA: 0s - loss: 0.3952 - acc: 0.8143
3520/3938 [=========================>....] - ETA: 0s - loss: 0.3940 - acc: 0.8151
3744/3938 [===========================>..] - ETA: 0s - loss: 0.3939 - acc: 0.8152
3938/3938 [==============================] - 0s - loss: 0.3963 - acc: 0.8144 - val_loss: 0.4244 - val_acc: 0.8039
Epoch 18/30
32/3938 [..............................] - ETA: 0s - loss: 0.3232 - acc: 0.7812
288/3938 [=>............................] - ETA: 0s - loss: 0.3309 - acc: 0.8403
512/3938 [==>...........................] - ETA: 0s - loss: 0.3459 - acc: 0.8398
768/3938 [====>.........................] - ETA: 0s - loss: 0.3700 - acc: 0.8411
1024/3938 [======>.......................] - ETA: 0s - loss: 0.3640 - acc: 0.8398
1312/3938 [========>.....................] - ETA: 0s - loss: 0.3688 - acc: 0.8377
1600/3938 [===========>..................] - ETA: 0s - loss: 0.3770 - acc: 0.8287
1888/3938 [=============>................] - ETA: 0s - loss: 0.3748 - acc: 0.8316
2176/3938 [===============>..............] - ETA: 0s - loss: 0.3816 - acc: 0.8277
2432/3938 [=================>............] - ETA: 0s - loss: 0.3849 - acc: 0.8224
2688/3938 [===================>..........] - ETA: 0s - loss: 0.3900 - acc: 0.8207
2912/3938 [=====================>........] - ETA: 0s - loss: 0.3925 - acc: 0.8194
3136/3938 [======================>.......] - ETA: 0s - loss: 0.3926 - acc: 0.8192
3360/3938 [========================>.....] - ETA: 0s - loss: 0.3928 - acc: 0.8205
3648/3938 [==========================>...] - ETA: 0s - loss: 0.3936 - acc: 0.8196
3938/3938 [==============================] - 0s - loss: 0.3962 - acc: 0.8174 - val_loss: 0.4252 - val_acc: 0.8051
Epoch 19/30
32/3938 [..............................] - ETA: 0s - loss: 0.5642 - acc: 0.7500
288/3938 [=>............................] - ETA: 0s - loss: 0.4101 - acc: 0.8229
544/3938 [===>..........................] - ETA: 0s - loss: 0.4069 - acc: 0.8217
768/3938 [====>.........................] - ETA: 0s - loss: 0.4171 - acc: 0.8138
1024/3938 [======>.......................] - ETA: 0s - loss: 0.4134 - acc: 0.8154
1344/3938 [=========>....................] - ETA: 0s - loss: 0.4068 - acc: 0.8207
1632/3938 [===========>..................] - ETA: 0s - loss: 0.4013 - acc: 0.8229
1984/3938 [==============>...............] - ETA: 0s - loss: 0.4020 - acc: 0.8185
2272/3938 [================>.............] - ETA: 0s - loss: 0.3961 - acc: 0.8209
2528/3938 [==================>...........] - ETA: 0s - loss: 0.3940 - acc: 0.8204
2784/3938 [====================>.........] - ETA: 0s - loss: 0.3923 - acc: 0.8236
3040/3938 [======================>.......] - ETA: 0s - loss: 0.3898 - acc: 0.8220
3264/3938 [=======================>......] - ETA: 0s - loss: 0.3892 - acc: 0.8238
3520/3938 [=========================>....] - ETA: 0s - loss: 0.3906 - acc: 0.8247
3808/3938 [============================>.] - ETA: 0s - loss: 0.3911 - acc: 0.8238
3938/3938 [==============================] - 0s - loss: 0.3911 - acc: 0.8222 - val_loss: 0.4267 - val_acc: 0.8051
Epoch 20/30
32/3938 [..............................] - ETA: 0s - loss: 0.5481 - acc: 0.7188
256/3938 [>.............................] - ETA: 0s - loss: 0.4456 - acc: 0.7930
512/3938 [==>...........................] - ETA: 0s - loss: 0.4256 - acc: 0.8125
768/3938 [====>.........................] - ETA: 0s - loss: 0.4137 - acc: 0.8216
992/3938 [======>.......................] - ETA: 0s - loss: 0.4066 - acc: 0.8226
1248/3938 [========>.....................] - ETA: 0s - loss: 0.3909 - acc: 0.8253
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3873 - acc: 0.8278
1760/3938 [============>.................] - ETA: 0s - loss: 0.3836 - acc: 0.8290
1984/3938 [==============>...............] - ETA: 0s - loss: 0.3834 - acc: 0.8286
2272/3938 [================>.............] - ETA: 0s - loss: 0.3847 - acc: 0.8297
2560/3938 [==================>...........] - ETA: 0s - loss: 0.3855 - acc: 0.8297
2848/3938 [====================>.........] - ETA: 0s - loss: 0.3835 - acc: 0.8287
3136/3938 [======================>.......] - ETA: 0s - loss: 0.3861 - acc: 0.8291
3424/3938 [=========================>....] - ETA: 0s - loss: 0.3857 - acc: 0.8283
3712/3938 [===========================>..] - ETA: 0s - loss: 0.3887 - acc: 0.8268
3936/3938 [============================>.] - ETA: 0s - loss: 0.3896 - acc: 0.8239
3938/3938 [==============================] - 1s - loss: 0.3896 - acc: 0.8240 - val_loss: 0.4280 - val_acc: 0.7962
Epoch 21/30
32/3938 [..............................] - ETA: 0s - loss: 0.3896 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.3607 - acc: 0.8472
544/3938 [===>..........................] - ETA: 0s - loss: 0.3754 - acc: 0.8474
800/3938 [=====>........................] - ETA: 0s - loss: 0.3831 - acc: 0.8375
1088/3938 [=======>......................] - ETA: 0s - loss: 0.3771 - acc: 0.8373
1344/3938 [=========>....................] - ETA: 0s - loss: 0.3869 - acc: 0.8296
1472/3938 [==========>...................] - ETA: 0s - loss: 0.3831 - acc: 0.8322
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3829 - acc: 0.8318
1568/3938 [==========>...................] - ETA: 0s - loss: 0.3823 - acc: 0.8329
1728/3938 [============>.................] - ETA: 0s - loss: 0.3842 - acc: 0.8287
1984/3938 [==============>...............] - ETA: 0s - loss: 0.3832 - acc: 0.8327
2240/3938 [================>.............] - ETA: 0s - loss: 0.3869 - acc: 0.8281
2496/3938 [==================>...........] - ETA: 0s - loss: 0.3883 - acc: 0.8241
2752/3938 [===================>..........] - ETA: 0s - loss: 0.3891 - acc: 0.8219
3008/3938 [=====================>........] - ETA: 0s - loss: 0.3934 - acc: 0.8201
3264/3938 [=======================>......] - ETA: 0s - loss: 0.3902 - acc: 0.8220
3488/3938 [=========================>....] - ETA: 0s - loss: 0.3877 - acc: 0.8234
3712/3938 [===========================>..] - ETA: 0s - loss: 0.3907 - acc: 0.8227
3938/3938 [==============================] - 1s - loss: 0.3900 - acc: 0.8240 - val_loss: 0.4285 - val_acc: 0.8021
Epoch 22/30
32/3938 [..............................] - ETA: 0s - loss: 0.3351 - acc: 0.8750
288/3938 [=>............................] - ETA: 0s - loss: 0.3456 - acc: 0.8333
544/3938 [===>..........................] - ETA: 0s - loss: 0.3733 - acc: 0.8199
800/3938 [=====>........................] - ETA: 0s - loss: 0.3800 - acc: 0.8175
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3780 - acc: 0.8172
1280/3938 [========>.....................] - ETA: 0s - loss: 0.3793 - acc: 0.8180
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3728 - acc: 0.8218
1696/3938 [===========>..................] - ETA: 0s - loss: 0.3795 - acc: 0.8184
1856/3938 [=============>................] - ETA: 0s - loss: 0.3798 - acc: 0.8173
2112/3938 [===============>..............] - ETA: 0s - loss: 0.3843 - acc: 0.8163
2368/3938 [=================>............] - ETA: 0s - loss: 0.3853 - acc: 0.8180
2592/3938 [==================>...........] - ETA: 0s - loss: 0.3930 - acc: 0.8129
2848/3938 [====================>.........] - ETA: 0s - loss: 0.3883 - acc: 0.8174
3104/3938 [======================>.......] - ETA: 0s - loss: 0.3933 - acc: 0.8160
3392/3938 [========================>.....] - ETA: 0s - loss: 0.3911 - acc: 0.8172
3616/3938 [==========================>...] - ETA: 0s - loss: 0.3925 - acc: 0.8169
3904/3938 [============================>.] - ETA: 0s - loss: 0.3911 - acc: 0.8171
3938/3938 [==============================] - 1s - loss: 0.3908 - acc: 0.8174 - val_loss: 0.4284 - val_acc: 0.8021
Epoch 23/30
32/3938 [..............................] - ETA: 0s - loss: 0.2805 - acc: 0.8750
320/3938 [=>............................] - ETA: 0s - loss: 0.3782 - acc: 0.8187
544/3938 [===>..........................] - ETA: 0s - loss: 0.3897 - acc: 0.8254
800/3938 [=====>........................] - ETA: 0s - loss: 0.3933 - acc: 0.8300
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3918 - acc: 0.8286
1312/3938 [========>.....................] - ETA: 0s - loss: 0.3936 - acc: 0.8239
1536/3938 [==========>...................] - ETA: 0s - loss: 0.3968 - acc: 0.8190
1760/3938 [============>.................] - ETA: 0s - loss: 0.3981 - acc: 0.8176
1984/3938 [==============>...............] - ETA: 0s - loss: 0.3950 - acc: 0.8201
2208/3938 [===============>..............] - ETA: 0s - loss: 0.3928 - acc: 0.8216
2464/3938 [=================>............] - ETA: 0s - loss: 0.3905 - acc: 0.8251
2656/3938 [===================>..........] - ETA: 0s - loss: 0.3922 - acc: 0.8234
2912/3938 [=====================>........] - ETA: 0s - loss: 0.3892 - acc: 0.8276
3104/3938 [======================>.......] - ETA: 0s - loss: 0.3875 - acc: 0.8276
3392/3938 [========================>.....] - ETA: 0s - loss: 0.3875 - acc: 0.8258
3680/3938 [===========================>..] - ETA: 0s - loss: 0.3848 - acc: 0.8288
3938/3938 [==============================] - 1s - loss: 0.3853 - acc: 0.8281 - val_loss: 0.4369 - val_acc: 0.8015
Epoch 24/30
32/3938 [..............................] - ETA: 0s - loss: 0.2858 - acc: 0.8750
288/3938 [=>............................] - ETA: 0s - loss: 0.3310 - acc: 0.8368
544/3938 [===>..........................] - ETA: 0s - loss: 0.3566 - acc: 0.8309
800/3938 [=====>........................] - ETA: 0s - loss: 0.3550 - acc: 0.8425
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3531 - acc: 0.8466
1312/3938 [========>.....................] - ETA: 0s - loss: 0.3600 - acc: 0.8453
1600/3938 [===========>..................] - ETA: 0s - loss: 0.3647 - acc: 0.8400
1888/3938 [=============>................] - ETA: 0s - loss: 0.3689 - acc: 0.8337
2016/3938 [==============>...............] - ETA: 0s - loss: 0.3749 - acc: 0.8309
2304/3938 [================>.............] - ETA: 0s - loss: 0.3764 - acc: 0.8268
2592/3938 [==================>...........] - ETA: 0s - loss: 0.3770 - acc: 0.8275
2848/3938 [====================>.........] - ETA: 0s - loss: 0.3778 - acc: 0.8269
3040/3938 [======================>.......] - ETA: 0s - loss: 0.3777 - acc: 0.8257
3232/3938 [=======================>......] - ETA: 0s - loss: 0.3794 - acc: 0.8243
3488/3938 [=========================>....] - ETA: 0s - loss: 0.3833 - acc: 0.8214
3776/3938 [===========================>..] - ETA: 0s - loss: 0.3821 - acc: 0.8242
3938/3938 [==============================] - 1s - loss: 0.3835 - acc: 0.8230 - val_loss: 0.4291 - val_acc: 0.7962
Epoch 25/30
32/3938 [..............................] - ETA: 0s - loss: 0.4123 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.4097 - acc: 0.7882
544/3938 [===>..........................] - ETA: 0s - loss: 0.3884 - acc: 0.8051
800/3938 [=====>........................] - ETA: 0s - loss: 0.3958 - acc: 0.8050
1024/3938 [======>.......................] - ETA: 0s - loss: 0.3875 - acc: 0.8145
1280/3938 [========>.....................] - ETA: 0s - loss: 0.3962 - acc: 0.8094
1536/3938 [==========>...................] - ETA: 0s - loss: 0.3842 - acc: 0.8210
1792/3938 [============>.................] - ETA: 0s - loss: 0.3825 - acc: 0.8203
2048/3938 [==============>...............] - ETA: 0s - loss: 0.3792 - acc: 0.8218
2272/3938 [================>.............] - ETA: 0s - loss: 0.3802 - acc: 0.8235
2496/3938 [==================>...........] - ETA: 0s - loss: 0.3806 - acc: 0.8233
2720/3938 [===================>..........] - ETA: 0s - loss: 0.3757 - acc: 0.8268
2944/3938 [=====================>........] - ETA: 0s - loss: 0.3779 - acc: 0.8254
3200/3938 [=======================>......] - ETA: 0s - loss: 0.3799 - acc: 0.8263
3520/3938 [=========================>....] - ETA: 0s - loss: 0.3865 - acc: 0.8239
3808/3938 [============================>.] - ETA: 0s - loss: 0.3854 - acc: 0.8230
3938/3938 [==============================] - 1s - loss: 0.3841 - acc: 0.8233 - val_loss: 0.4327 - val_acc: 0.8015
Epoch 26/30
32/3938 [..............................] - ETA: 0s - loss: 0.2913 - acc: 0.8750
320/3938 [=>............................] - ETA: 0s - loss: 0.3491 - acc: 0.8469
608/3938 [===>..........................] - ETA: 0s - loss: 0.3974 - acc: 0.8207
864/3938 [=====>........................] - ETA: 0s - loss: 0.3882 - acc: 0.8183
1120/3938 [=======>......................] - ETA: 0s - loss: 0.3922 - acc: 0.8232
1376/3938 [=========>....................] - ETA: 0s - loss: 0.3933 - acc: 0.8234
1664/3938 [===========>..................] - ETA: 0s - loss: 0.3899 - acc: 0.8245
1952/3938 [=============>................] - ETA: 0s - loss: 0.3801 - acc: 0.8315
2240/3938 [================>.............] - ETA: 0s - loss: 0.3795 - acc: 0.8335
2496/3938 [==================>...........] - ETA: 0s - loss: 0.3850 - acc: 0.8285
2784/3938 [====================>.........] - ETA: 0s - loss: 0.3850 - acc: 0.8276
3104/3938 [======================>.......] - ETA: 0s - loss: 0.3858 - acc: 0.8247
3392/3938 [========================>.....] - ETA: 0s - loss: 0.3860 - acc: 0.8237
3648/3938 [==========================>...] - ETA: 0s - loss: 0.3882 - acc: 0.8226
3872/3938 [============================>.] - ETA: 0s - loss: 0.3853 - acc: 0.8246
3938/3938 [==============================] - 1s - loss: 0.3831 - acc: 0.8261 - val_loss: 0.4338 - val_acc: 0.8009
Epoch 27/30
32/3938 [..............................] - ETA: 0s - loss: 0.2561 - acc: 0.9062
320/3938 [=>............................] - ETA: 0s - loss: 0.3501 - acc: 0.8344
576/3938 [===>..........................] - ETA: 0s - loss: 0.3710 - acc: 0.8316
800/3938 [=====>........................] - ETA: 0s - loss: 0.3850 - acc: 0.8325
1024/3938 [======>.......................] - ETA: 0s - loss: 0.3761 - acc: 0.8369
1280/3938 [========>.....................] - ETA: 0s - loss: 0.3841 - acc: 0.8297
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3849 - acc: 0.8291
1728/3938 [============>.................] - ETA: 0s - loss: 0.3856 - acc: 0.8281
1920/3938 [=============>................] - ETA: 0s - loss: 0.3846 - acc: 0.8276
2176/3938 [===============>..............] - ETA: 0s - loss: 0.3829 - acc: 0.8313
2432/3938 [=================>............] - ETA: 0s - loss: 0.3818 - acc: 0.8310
2656/3938 [===================>..........] - ETA: 0s - loss: 0.3808 - acc: 0.8309
2880/3938 [====================>.........] - ETA: 0s - loss: 0.3853 - acc: 0.8285
3136/3938 [======================>.......] - ETA: 0s - loss: 0.3854 - acc: 0.8265
3424/3938 [=========================>....] - ETA: 0s - loss: 0.3835 - acc: 0.8268
3744/3938 [===========================>..] - ETA: 0s - loss: 0.3859 - acc: 0.8240
3938/3938 [==============================] - 1s - loss: 0.3862 - acc: 0.8243 - val_loss: 0.4308 - val_acc: 0.7974
Epoch 28/30
32/3938 [..............................] - ETA: 0s - loss: 0.4191 - acc: 0.7188
288/3938 [=>............................] - ETA: 0s - loss: 0.3836 - acc: 0.8125
544/3938 [===>..........................] - ETA: 0s - loss: 0.3655 - acc: 0.8254
832/3938 [=====>........................] - ETA: 0s - loss: 0.3645 - acc: 0.8365
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3657 - acc: 0.8362
1344/3938 [=========>....................] - ETA: 0s - loss: 0.3657 - acc: 0.8326
1600/3938 [===========>..................] - ETA: 0s - loss: 0.3752 - acc: 0.8287
1888/3938 [=============>................] - ETA: 0s - loss: 0.3752 - acc: 0.8279
2144/3938 [===============>..............] - ETA: 0s - loss: 0.3743 - acc: 0.8293
2432/3938 [=================>............] - ETA: 0s - loss: 0.3742 - acc: 0.8298
2720/3938 [===================>..........] - ETA: 0s - loss: 0.3796 - acc: 0.8261
3008/3938 [=====================>........] - ETA: 0s - loss: 0.3790 - acc: 0.8275
3296/3938 [========================>.....] - ETA: 0s - loss: 0.3790 - acc: 0.8271
3584/3938 [==========================>...] - ETA: 0s - loss: 0.3757 - acc: 0.8287
3872/3938 [============================>.] - ETA: 0s - loss: 0.3830 - acc: 0.8239
3938/3938 [==============================] - 0s - loss: 0.3845 - acc: 0.8238 - val_loss: 0.4341 - val_acc: 0.7962
Epoch 29/30
32/3938 [..............................] - ETA: 0s - loss: 0.3200 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.3695 - acc: 0.8368
544/3938 [===>..........................] - ETA: 0s - loss: 0.3682 - acc: 0.8438
768/3938 [====>.........................] - ETA: 0s - loss: 0.3684 - acc: 0.8424
992/3938 [======>.......................] - ETA: 0s - loss: 0.3826 - acc: 0.8337
1248/3938 [========>.....................] - ETA: 0s - loss: 0.3826 - acc: 0.8325
1504/3938 [==========>...................] - ETA: 0s - loss: 0.3842 - acc: 0.8305
1792/3938 [============>.................] - ETA: 0s - loss: 0.3849 - acc: 0.8320
2016/3938 [==============>...............] - ETA: 0s - loss: 0.3847 - acc: 0.8284
2272/3938 [================>.............] - ETA: 0s - loss: 0.3833 - acc: 0.8257
2528/3938 [==================>...........] - ETA: 0s - loss: 0.3843 - acc: 0.8232
2784/3938 [====================>.........] - ETA: 0s - loss: 0.3816 - acc: 0.8240
3040/3938 [======================>.......] - ETA: 0s - loss: 0.3803 - acc: 0.8250
3296/3938 [========================>.....] - ETA: 0s - loss: 0.3797 - acc: 0.8252
3616/3938 [==========================>...] - ETA: 0s - loss: 0.3805 - acc: 0.8266
3904/3938 [============================>.] - ETA: 0s - loss: 0.3790 - acc: 0.8281
3938/3938 [==============================] - 1s - loss: 0.3785 - acc: 0.8291 - val_loss: 0.4419 - val_acc: 0.7974
Epoch 30/30
32/3938 [..............................] - ETA: 0s - loss: 0.4301 - acc: 0.8125
288/3938 [=>............................] - ETA: 0s - loss: 0.3288 - acc: 0.8472
544/3938 [===>..........................] - ETA: 0s - loss: 0.3952 - acc: 0.8199
800/3938 [=====>........................] - ETA: 0s - loss: 0.3982 - acc: 0.8225
1056/3938 [=======>......................] - ETA: 0s - loss: 0.3892 - acc: 0.8229
1344/3938 [=========>....................] - ETA: 0s - loss: 0.3894 - acc: 0.8155
1664/3938 [===========>..................] - ETA: 0s - loss: 0.3886 - acc: 0.8203
1952/3938 [=============>................] - ETA: 0s - loss: 0.3880 - acc: 0.8222
2240/3938 [================>.............] - ETA: 0s - loss: 0.3921 - acc: 0.8187
2496/3938 [==================>...........] - ETA: 0s - loss: 0.3864 - acc: 0.8237
2752/3938 [===================>..........] - ETA: 0s - loss: 0.3861 - acc: 0.8230
3008/3938 [=====================>........] - ETA: 0s - loss: 0.3833 - acc: 0.8241
3264/3938 [=======================>......] - ETA: 0s - loss: 0.3805 - acc: 0.8266
3552/3938 [==========================>...] - ETA: 0s - loss: 0.3806 - acc: 0.8260
3840/3938 [============================>.] - ETA: 0s - loss: 0.3800 - acc: 0.8258
3938/3938 [==============================] - 0s - loss: 0.3818 - acc: 0.8248 - val_loss: 0.4354 - val_acc: 0.8004
Making prediction
We have quite good validation accuracy but it’s time to evaluate model on test data.
predict_classes()Generates class values as a matrix of ones and zeros. Since we are dealing with binary classification, we’ll convert the output to a vector.predict_proba(). Generates the class probabilities as a numeric matrix indicating the probability of being a class. Again, we convert to a numeric vector because there is only one column output.
#predicted classes
yhat_keras_class_vec <- model_keras %>% predict_classes(as.matrix(x_test_tbl)) %>% as.vector()
yhat_keras_prob_vec <- model_keras %>% predict_proba(as.matrix(x_test_tbl)) %>% as.vector()#Inspect Performance With Yardstick
The yardstick package has a collection of handy functions for measuring performance of machine learning models. First, let’s get the data formatted for yardstick. We create a data frame with the truth (actual values as factors), estimate (predicted values as factors), and the class probability (probability of yes as numeric). We use the fct_recode() function from the forcats package to assist with recoding as Yes/No values.
estimates_keras_tbl <- tibble(
truth = as.factor(y_test_vec) %>% forcats::fct_recode(yes = "1", no = "0"),
estimate = as.factor(yhat_keras_class_vec) %>% forcats::fct_recode(yes = "1", no = "0"),
class_prob = yhat_keras_prob_vec
)
estimates_keras_tblNow that we have the data formatted, we can take advantage of the yardstick package. The only other thing we need to do is to set options(yardstick.event_first = FALSE)
options(yardstick.event_first = FALSE)Confusion table, accuracy and ROC
estimates_keras_tbl %>% roc_auc(truth, class_prob)
[1] 0.8631234**Precision and recall, f1 score*
tibble(
precision = estimates_keras_tbl %>% precision(truth, estimate),
recall = estimates_keras_tbl %>% recall(truth, estimate),
f1 = estimates_keras_tbl %>% f_meas(truth, estimate, beta = 1)
)#Explain the model with LIME
The lime package implements LIME methodology in R. One thing to note is that it’s not setup out-of-the-box to work with keras. The good news is with a few functions we can get everything working properly. We’ll need to make two custom functions:
model_type- used to telllimewhat type of a model we’re dealing with. It could be classification, regression, survival.predict_model- used to allowlimeto perform prediction that its algorithm can interpret.
The first thing we need to do is identify the class of our model object. We do this with the class() function.
class(model_keras)[1] "keras.models.Sequential" "keras.engine.training.Model" "keras.engine.topology.Container"
[4] "keras.engine.topology.Layer" "python.builtin.object" Next we create our model_type() function. It’s only input is x the keras model. The function simply returns “classification”, which tells LIME we are classifying.
model_type.keras.models.Sequential <- function(x, ...){
"classification"
}Then we create predict_model() function, which wraps keras::predict_proba(). The trick here is to realize that it’s inputs must be x - a model, newdata - a dataframe object (this is important), and type which is not used but can be use to switch - the output type. The output is also a little tricky because it must be in the format of probabilities by classification (this is important; shown next).
predict_model.keras.models.Sequential <- function(x, newdata, type, ...){
pred <- predict_proba(object = x, x = as.matrix(newdata))
data.frame(Yes = pred, No = 1 - pred)
}predict_model(x = model_keras, newdata = x_test_tbl, type = "raw") %>%
tibble::as.tibble()Let’s create explainer using the lime() function. We need to pass to the function a training data frame, model (model_keras) and set bin_continuous = FALSE. We could tell the algorithm to bin continuous variables, but this may not make sense for categorical numeric data that we didn’t change to factors.
explainer <- lime::lime(
x = x_train_tbl,
model = model_keras,
bin_continuous = FALSE
)Now we can run explain() function, which returns our explanation.
explanation <- lime::explain(
x_test_tbl[1:10, ], #only first 10 cases
explainer = explainer,
n_labels = 1,
n_features = 4,
kernel_width = 0.5
)We set n_labels = 1 because we care about explaining a single class. Setting n_features = 4 returns the top four features that are critical to each case. Finally, setting kernel_width = 0.5 allows us to increase the “model” value by shrinking the localized evaluation.
Feature importance visualization
The payoff for the work we put in using LIME is this feature importance plot. This allows us to visualize each of the first ten cases (observations) from the test data. The top four features for each case are shown. Note that they are not the same for each case. The green bars mean that the feature supports the model conclusion, and the red bars contradict.
options(scipen = 999)
plot_features(explanation) +
labs(title = "LIME Feature Importance Visualization",
subtitle = "Hold Out (Test) Set, First 10 Cases Shown")
Another excellent visualization can be performed using plot_explanations(), which produces a facetted heatmap of all case/label/feature combinations.
plot_explanations(explanation) +
labs(title = "LIME Feature Importance Heatmap",
subtitle = "Hold Out (Test) Set, First 10 Cases Shown")
Check Explanation with Correlation Analysis
One thing we need to be careful with the LIME visualization is that we are only doing a sample of the data, in our case the first 10 test observations. Therefore, we are gaining a very localized understanding of how the ANN works. However, we also want to know on from a global perspective what drives feature importance.
We can perform a correlation analysis on the training set as well as get understanding which features correlated globally to “Churn”. We’ll use corrr package, which performs tidy correlations with a function correlate()
The correlation visualization helps in distinguishing which features are relavant to Churn.
corrr_analysis %>%
ggplot(aes(x = Churn, y = forcats::fct_reorder(feature, desc(Churn)))) +
geom_point() +
# Positive Correlations - Contribute to churn
geom_segment(aes(xend = 0, yend = feature),
color = palette_light()[[2]],
data = corrr_analysis %>% filter(Churn > 0)) +
geom_point(color = palette_light()[[2]],
data = corrr_analysis %>% filter(Churn > 0)) +
# Negative Correlations - Prevent churn
geom_segment(aes(xend = 0, yend = feature),
color = palette_light()[[1]],
data = corrr_analysis %>% filter(Churn < 0)) +
geom_point(color = palette_light()[[1]],
data = corrr_analysis %>% filter(Churn < 0)) +
#vertical lines
geom_vline(xintercept = 0, color = palette_light()[[5]], size = 1, linetype = 2) +
geom_vline(xintercept = -0.25, color = palette_light()[[5]], size = 1, linetype = 2) +
geom_vline(xintercept = 0.25, color = palette_light()[[5]], size = 1, linetype = 2) +
theme_tq() +
labs(title = "Churn Correlation Analysis",
subtitle = "Positive Correlations (contribute to churn), Negative Correlations (prevent churn)",
y = "Feature Importance")
The correlation analysis helps us quickly disseminate which features that the LIME analysis may be excluding. We can see that the following features are highly correlated (magnitude > 0.25):
Increases Likelihood of Churn (Red): - Tenure = Bin 1 (<12 Months) - Internet Service = “Fiber Optic” - Payment Method = “Electronic Check”
Decreases Likelihood of Churn (Blue): - Contract = “Two Year” - Total Charges (Note that this may be a biproduct of additional services such as Online Security)
LS0tCnRpdGxlOiAiQ2h1cm4gcHJlZGljdGlvbnMgd2l0aCBrZXJhcywgcnNhbXBsZSBhbmQgcmVjaXBlcyIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKYGBge3J9CiMgTG9hZCBsaWJyYXJpZXMKbGlicmFyeShrZXJhcykKbGlicmFyeShsaW1lKQpsaWJyYXJ5KHRpZHlxdWFudCkKbGlicmFyeShyc2FtcGxlKQpsaWJyYXJ5KHJlY2lwZXMpCmxpYnJhcnkoeWFyZHN0aWNrKQpsaWJyYXJ5KGNvcnJyKQpgYGAKCiMgUHJlcHJvY2VzcyBEYXRhCgojIyBQcnVuZSB0aGUgZGF0YQoKTG9hZCBkYXRhIGFuZCBsb29rIGF0IHRoZSBzdHJ1Y3R1cmUKYGBge3J9CmNodXJuX2RhdGFfcmF3IDwtIHJlYWR4bDo6cmVhZF9leGNlbCgiL1VzZXJzL2FuZHJleW1hcmtpbi9Eb3dubG9hZHMvV0FfRm4tVXNlQ18tVGVsY28tQ3VzdG9tZXItQ2h1cm4ueGxzeCIpCmdsaW1wc2UoY2h1cm5fZGF0YV9yYXcpCmBgYAoKRGVzZWxlY3QgYGN1c3RvbWVySURgIGFuZCBkcm9wIE5BcwoKYGBge3J9CmNodXJuX2RhdGFfdGJsIDwtIGNodXJuX2RhdGFfcmF3ICU+JSAKICBzZWxlY3QoLWN1c3RvbWVySUQpICU+JSAKICBkcm9wX25hKCkgJT4lIAogIHNlbGVjdChDaHVybiwgZXZlcnl0aGluZygpKQpgYGAKCiMjIFNwbGl0IGludG8gdHJhaW4vdGVzdCBzcGxpdHMKCldlIGhhdmUgYSBuZXcgcGFja2FnZSwgYHJzYW1wbGVgLCB3aGljaCBpcyB2ZXJ5IHVzZWZ1bCBmb3Igc2FtcGxpbmcgbWV0aG9kcy4gSXQgaGFzIHRoZSBgaW5pdGlhbF9zcGxpdCgpYCBmdW5jdGlvbiBmb3Igc3BsaXR0aW5nIGRhdGEgc2V0cyBpbnRvIHRyYWluaW5nIGFuZCB0ZXN0aW5nIHNldHMuIFRoZSByZXR1cm4gaXMgYSBzcGVjaWFsIGByc3BsaXRgIG9iamVjdC4KCmBgYHtyfQpzZXQuc2VlZCgxMDApCnRyYWluX3Rlc3Rfc3BsaXQgPC0gaW5pdGlhbF9zcGxpdChjaHVybl9kYXRhX3RibCwgcHJvcCA9IDAuOCkKdHJhaW5fdGVzdF9zcGxpdApgYGAKCldlIGNhbiByZXRyaWV2ZSBvdXIgdHJhaW5pbmcgYW5kIHRlc3Rpbmcgc2V0cyB1c2luZyBgdHJhaW5pbmcoKWAgYW5kIGB0ZXN0aW5nKClgIGZ1bmN0aW9ucwoKYGBge3J9CnRyYWluX3RibCA8LSB0cmFpbmluZyh0cmFpbl90ZXN0X3NwbGl0KQp0ZXN0X3RibCA8LSB0cmFpbmluZyh0cmFpbl90ZXN0X3NwbGl0KQpgYGAKCiMjIEV4cGxvcmF0aW9uOiB3aGF0IG90aGVyIHRyYWluc2Zvcm1hdGlvbnMgYXJlIG5lZWRlZD8KCioqRElTQ1JFVElaRSBUSEUg4oCcVEVOVVJF4oCdIEZFQVRVUkUqKgoKCmBgYHtyfQpjaHVybl9kYXRhX3RibCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gdGVudXJlKSkgKyAKICBnZW9tX2hpc3RvZ3JhbSgpCgpjaHVybl9kYXRhX3RibCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gdGVudXJlKSkgKyAKICBnZW9tX2hpc3RvZ3JhbShiaW5zID0gNikKYGBgCgoqKlRSQU5TRk9STSBUSEUg4oCcVE9UQUxDSEFSR0VT4oCdIEZFQVRVUkUqKgoKYGBge3J9CmNodXJuX2RhdGFfdGJsICU+JSAKICBnZ3Bsb3QoYWVzKHggPSBUb3RhbENoYXJnZXMpKSArIAogIGdlb21faGlzdG9ncmFtKGJpbnMgPSAxMDApCgpjaHVybl9kYXRhX3RibCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gbG9nKFRvdGFsQ2hhcmdlcykpKSArIAogIGdlb21faGlzdG9ncmFtKGJpbnMgPSAxMDApCmBgYAoKKipQcm8gVGlwOiBBIHF1aWNrIHRlc3QgaXMgdG8gc2VlIGlmIHRoZSBsb2cgdHJhbnNmb3JtYXRpb24gaW5jcmVhc2VzIHRoZSBtYWduaXR1ZGUgb2YgdGhlIGNvcnJlbGF0aW9uKiogYmV0d2VlbiDigJxUb3RhbENoYXJnZXPigJ0gYW5kIOKAnENodXJu4oCdLiBXZeKAmWxsIHVzZSBhIGZldyBgZHBseXJgIG9wZXJhdGlvbnMgYWxvbmcgd2l0aCB0aGUgYGNvcnJyYCBwYWNrYWdlIHRvIHBlcmZvcm0gYSBxdWljayBjb3JyZWxhdGlvbi4KCiogYGNvcnJlbGF0ZSgpYDogcGVyZm9ybXMgdGlkeSBjb3JyZWxhdGlvbnMgb24gbnVtZXJpYyBkYXRhCiogYGZvY3VzKClgOiBTaW1pbGFyIHRvIGBzZWxlY3QoKWAuIFRha2VzIGNvbHVtbnMgYW5kIGZvY3VzZXMgb24gb25seSB0aGUgcm93cy9jb2x1bW5zIG9mIGltcG9ydGFuY2UuCiogYGZhc2hpb24oKWA6IE1ha2VzIHRoZSBmb3JtYXR0aW5nIGVhc2llciB0byByZWFkLgoKYGBge3J9CiMgRGV0ZXJtaW5lIGlmIGxvZyB0cmFuc2Zvcm1hdGlvbiBpbXByb3ZlcyBjb3JyZWxhdGlvbiBiZXR3ZWVuIFRvdGFsQ2hhcmdlcyBhbmQgQ2h1cm4KY2h1cm5fZGF0YV90YmwgJT4lIAogIHNlbGVjdChDaHVybiwgVG90YWxDaGFyZ2VzKSAlPiUgCiAgbXV0YXRlKENodXJuID0gQ2h1cm4gJT4lIGFzLmZhY3RvcigpICU+JSBhcy5udW1lcmljKCksIAogICAgICAgICBMb2dUb3RhbENoYXJnZXMgPSBsb2coVG90YWxDaGFyZ2VzKSkgJT4lIAogIGNvcnJlbGF0ZSgpICU+JSAKICBmb2N1cyhDaHVybikgJT4lIAogIGZhc2hpb24oKQoKYGBgCgoKKipPTkUtSE9UIEVOQ09ESU5HKioKCioqRkVBVFVSRSBTQ0FMSU5HKioKCiMjIFByZXByb2Nlc3Npbmcgd2l0aCBgcmVjaXBpZXNgCgojIyMgQ3JlYXRlIGEgcmVjaXBlCgpBIOKAnHJlY2lwZeKAnSBpcyBub3RoaW5nIG1vcmUgdGhhbiBhIHNlcmllcyBvZiBzdGVwcyB5b3Ugd291bGQgbGlrZSB0byBwZXJmb3JtIG9uIHRoZSB0cmFpbmluZywgdGVzdGluZyBhbmQvb3IgdmFsaWRhdGlvbiBzZXRzLgoKV2UgdXNlIHRoZSBgcmVjaXBlKClgIGZ1bmN0aW9uIHRvIGltcGxlbWVudCBvdXIgcHJlcHJvY2Vzc2luZyBzdGVwcy4gVGhlIGZ1bmN0aW9uIHRha2VzIHR3byBhcmd1bWVudHMgYG9iamVjdCA9IENodXJuIH4gLmAgYW5kIGRhdGEgYXJndW1lbnQgYGRhdGEgPSB0cmFpbl90YmxgLgoKQSByZWNpcGUgaXMgbm90IHZlcnkgdXNlZnVsIHVudGlsIHdlIGFkZCDigJxzdGVwc+KAnSwgd2hpY2ggYXJlIHVzZWQgdG8gdHJhbnNmb3JtIHRoZSBkYXRhLiBGb3Igb3VyIG1vZGVsIHdlJ2xsIHVzZToKCiogYHN0ZXBfZGlzY3JldGl6ZSgpYCB3aXRoIGBvcHRpb25zID0gbGlzdChjdXQgPSA2KWAgdG8gY3V0IGNvbnRpbnVvdXMgZGF0YSBpbnRvIHNpeCBiaW5zCiogYHN0ZXBfbG9nKClgIHRvIGxvZyB0cmFuc2Zvcm0gIlRvdGFsQ2hhcmdlcyIKKiBgc3RlcF9kdW1teSgpYCB0byBvbmUtaG90IGVuY29kZSBjYXRlZ29yaWNhbCBmZWF0dXJlcy4KKiBgc3RlcF9jZW50ZXIoKWAgdG8gbWVhbi1jZW50ZXIgZGF0YQoqIGBzdGVwX3NjYWxlKClgIHRvIHNjYWxlIHRoZSBkYXRhCgpUaGUgbGFzdCBzdGVwcyBpcyB0byBwcmVwYXJlIHJlY2lwZSB3aXRoIGBwcmVwKClgIGZ1bmN0aW9uLiBUaGlzIHN0ZXAgaXMgdXNlZCB0byDigJxlc3RpbWF0ZSB0aGUgcmVxdWlyZWQgcGFyYW1ldGVycyBmcm9tIGEgdHJhaW5pbmcgc2V0IHRoYXQgY2FuIGxhdGVyIGJlIGFwcGxpZWQgdG8gb3RoZXIgZGF0YSBzZXRz4oCdLiBUaGlzIGlzIGltcG9ydGFudCBmb3IgY2VudGVyaW5nIGFuZCBzY2FsaW5nIGFuZCBvdGhlciBmdW5jdGlvbnMgdGhhdCB1c2UgcGFyYW1ldGVycyBkZWZpbmVkIGZyb20gdGhlIHRyYWluaW5nIHNldC4KCmBgYHtyfQojY3JlYXRlIHJlY2lwZQpyZWNfb2JqIDwtIHJlY2lwZShDaHVybiB+IC4sIGRhdGEgPSB0cmFpbl90YmwpICU+JSAKICBzdGVwX2Rpc2NyZXRpemUodGVudXJlLCBvcHRpb25zID0gbGlzdChjdXRzID0gNikpICU+JSAKICBzdGVwX2xvZyhUb3RhbENoYXJnZXMpICU+JSAKICBzdGVwX2R1bW15KGFsbF9ub21pbmFsKCksIC1hbGxfb3V0Y29tZXMoKSkgJT4lIAogIHN0ZXBfY2VudGVyKGFsbF9wcmVkaWN0b3JzKCksIC1hbGxfb3V0Y29tZXMoKSkgJT4lIAogIHN0ZXBfc2NhbGUoYWxsX3ByZWRpY3RvcnMoKSwgLWFsbF9vdXRjb21lcygpKSAlPiUgCiAgcHJlcChkYXRhID0gdHJhaW5fdGJsKQoKc2F2ZVJEUyhyZWNfb2JqLCAicmVjaXBlX2Fubl9jaHVybi5SRFMiKQpgYGAKCmBgYHtyfQpyZWNfb2JqCmBgYAoKCiMjIyBCYWtpbmcgd2l0aCB5b3VyIHJlY2lwZQoKV2UgY2FuIGFwcGx5IHRoZSByZWNpcGUgdG8gYW55IGRhdGFzZXQgd2l0aCBgYmFrZSgpYCBmdW5jdGlvbiBhbmQgaXQgcHJvY2Vzc2VzIHRoZSBkYXRhIGZvbGxvd2luZyBvdXIgcmVjaXBlIHN0ZXBzLiAKCmBgYHtyfQojcHJlZGljdG9ycwp4X3RyYWluX3RibCA8LSBiYWtlKHJlY19vYmosIG5ld2RhdGEgPSB0cmFpbl90YmwpICU+JSBzZWxlY3QoLUNodXJuKQp4X3Rlc3RfdGJsIDwtIGJha2UocmVjX29iaiwgbmV3ZGF0YSA9IHRlc3RfdGJsKSAlPiUgc2VsZWN0KC1DaHVybikKCmdsaW1wc2UoeF90cmFpbl90YmwpCgpgYGAKCiMjIyBEb24ndCBmb3JnZXQgdGhlIHRhcmdldAoKYGBge3J9CnlfdHJhaW5fdmVjIDwtIGlmZWxzZShwdWxsKHRyYWluX3RibCwgQ2h1cm4pID09ICJZZXMiLCAxLCAwKQp5X3Rlc3RfdmVjIDwtIGlmZWxzZShwdWxsKHRlc3RfdGJsLCBDaHVybikgPT0gIlllcyIsIDEsIDApCmBgYAoKCiNNb2RlbCBDdXN0b21lciBDaHVybiBpbiBLZXJhcwoKYGBge3J9Cm1vZGVsX2tlcmFzIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUgCiAgbGF5ZXJfZGVuc2UoCiAgICB1bml0cyA9IDY0LCAKICAgIGtlcm5lbF9pbml0aWFsaXplciA9ICJ1bmlmb3JtIiwKICAgIGFjdGl2YXRpb24gPSAicmVsdSIsCiAgICBpbnB1dF9zaGFwZSA9IG5jb2woeF90cmFpbl90YmwpCiAgKSAlPiUgCiAgbGF5ZXJfZHJvcG91dChyYXRlID0gMC4zKSAlPiUgCiAgbGF5ZXJfZGVuc2UoCiAgICB1bml0cyA9IDY0LCAKICAgIGtlcm5lbF9pbml0aWFsaXplciA9ICJ1bmlmb3JtIiwKICAgIGFjdGl2YXRpb24gPSAicmVsdSIKICApICU+JSAKICBsYXllcl9kcm9wb3V0KHJhdGUgPSAwLjMpICU+JSAKICBsYXllcl9kZW5zZSgKICAgIHVuaXRzID0gMSwKICAgIGtlcm5lbF9pbml0aWFsaXplciA9ICJ1bmlmb3JtIiwKICAgIGFjdGl2YXRpb24gPSAic2lnbW9pZCIpCgptb2RlbF9rZXJhcyAlPiUgY29tcGlsZSgKICBsb3NzID0gImJpbmFyeV9jcm9zc2VudHJvcHkiLAogIG9wdGltaXplciA9ICJhZGFtIiwgCiAgbWV0cmljcyA9ICJhY2N1cmFjeSIKKQpgYGAKCgpgYGB7cn0KaGlzdG9yeSA8LSBtb2RlbF9rZXJhcyAlPiUgZml0KAogIGFzLm1hdHJpeCh4X3RyYWluX3RibCksIAogIHlfdHJhaW5fdmVjLCAKICBiYXRjaF9zaXplID0gMzIsCiAgZXBvY2hzID0gMzAsIAogIHZhbGlkYXRpb25fc3BsaXQgPSAwLjMKKQpgYGAKCmBgYHtyfQpoaXN0b3J5CnBsb3QoaGlzdG9yeSkKYGBgCgoqKk1ha2luZyBwcmVkaWN0aW9uKioKCldlIGhhdmUgcXVpdGUgZ29vZCB2YWxpZGF0aW9uIGFjY3VyYWN5IGJ1dCBpdCdzIHRpbWUgdG8gZXZhbHVhdGUgbW9kZWwgb24gYHRlc3RgIGRhdGEuCgoqIGBwcmVkaWN0X2NsYXNzZXMoKWAgR2VuZXJhdGVzIGNsYXNzIHZhbHVlcyBhcyBhIG1hdHJpeCBvZiBvbmVzIGFuZCB6ZXJvcy4gU2luY2Ugd2UgYXJlIGRlYWxpbmcgd2l0aCBiaW5hcnkgY2xhc3NpZmljYXRpb24sIHdl4oCZbGwgY29udmVydCB0aGUgb3V0cHV0IHRvIGEgdmVjdG9yLgoqIGBwcmVkaWN0X3Byb2JhKClgLiBHZW5lcmF0ZXMgdGhlIGNsYXNzIHByb2JhYmlsaXRpZXMgYXMgYSBudW1lcmljIG1hdHJpeCBpbmRpY2F0aW5nIHRoZSBwcm9iYWJpbGl0eSBvZiBiZWluZyBhIGNsYXNzLiBBZ2Fpbiwgd2UgY29udmVydCB0byBhIG51bWVyaWMgdmVjdG9yIGJlY2F1c2UgdGhlcmUgaXMgb25seSBvbmUgY29sdW1uIG91dHB1dC4KCmBgYHtyfQojcHJlZGljdGVkIGNsYXNzZXMKCnloYXRfa2VyYXNfY2xhc3NfdmVjIDwtIG1vZGVsX2tlcmFzICU+JSBwcmVkaWN0X2NsYXNzZXMoYXMubWF0cml4KHhfdGVzdF90YmwpKSAlPiUgYXMudmVjdG9yKCkKeWhhdF9rZXJhc19wcm9iX3ZlYyAgPC0gbW9kZWxfa2VyYXMgJT4lIHByZWRpY3RfcHJvYmEoYXMubWF0cml4KHhfdGVzdF90YmwpKSAlPiUgYXMudmVjdG9yKCkKYGBgCgojSW5zcGVjdCBQZXJmb3JtYW5jZSBXaXRoIFlhcmRzdGljawoKVGhlIGB5YXJkc3RpY2tgIHBhY2thZ2UgaGFzIGEgY29sbGVjdGlvbiBvZiBoYW5keSBmdW5jdGlvbnMgZm9yIG1lYXN1cmluZyBwZXJmb3JtYW5jZSBvZiBtYWNoaW5lIGxlYXJuaW5nIG1vZGVscy4KRmlyc3QsIGxldOKAmXMgZ2V0IHRoZSBkYXRhIGZvcm1hdHRlZCBmb3IgeWFyZHN0aWNrLiBXZSBjcmVhdGUgYSBkYXRhIGZyYW1lIHdpdGggdGhlIHRydXRoIChhY3R1YWwgdmFsdWVzIGFzIGZhY3RvcnMpLCBlc3RpbWF0ZSAocHJlZGljdGVkIHZhbHVlcyBhcyBmYWN0b3JzKSwgYW5kIHRoZSBjbGFzcyBwcm9iYWJpbGl0eSAocHJvYmFiaWxpdHkgb2YgeWVzIGFzIG51bWVyaWMpLiBXZSB1c2UgdGhlIGBmY3RfcmVjb2RlKClgIGZ1bmN0aW9uIGZyb20gdGhlIGBmb3JjYXRzYCBwYWNrYWdlIHRvIGFzc2lzdCB3aXRoIHJlY29kaW5nIGFzIFllcy9ObyB2YWx1ZXMuCgpgYGB7cn0KZXN0aW1hdGVzX2tlcmFzX3RibCA8LSB0aWJibGUoCiAgdHJ1dGggPSBhcy5mYWN0b3IoeV90ZXN0X3ZlYykgJT4lIGZvcmNhdHM6OmZjdF9yZWNvZGUoeWVzID0gIjEiLCBubyA9ICIwIiksCiAgZXN0aW1hdGUgPSBhcy5mYWN0b3IoeWhhdF9rZXJhc19jbGFzc192ZWMpICU+JSBmb3JjYXRzOjpmY3RfcmVjb2RlKHllcyA9ICIxIiwgbm8gPSAiMCIpLAogIGNsYXNzX3Byb2IgPSB5aGF0X2tlcmFzX3Byb2JfdmVjCikKCmVzdGltYXRlc19rZXJhc190YmwKYGBgCk5vdyB0aGF0IHdlIGhhdmUgdGhlIGRhdGEgZm9ybWF0dGVkLCB3ZSBjYW4gdGFrZSBhZHZhbnRhZ2Ugb2YgdGhlIGB5YXJkc3RpY2tgIHBhY2thZ2UuIFRoZSBvbmx5IG90aGVyIHRoaW5nIHdlIG5lZWQgdG8gZG8gaXMgdG8gc2V0IGBvcHRpb25zKHlhcmRzdGljay5ldmVudF9maXJzdCA9IEZBTFNFKWAKCmBgYHtyfQpvcHRpb25zKHlhcmRzdGljay5ldmVudF9maXJzdCA9IEZBTFNFKQpgYGAKCioqQ29uZnVzaW9uIHRhYmxlLCBhY2N1cmFjeSBhbmQgUk9DKioKCmBgYHtyfQplc3RpbWF0ZXNfa2VyYXNfdGJsICU+JSBjb25mX21hdCh0cnV0aCwgZXN0aW1hdGUpCgplc3RpbWF0ZXNfa2VyYXNfdGJsICU+JSBtZXRyaWNzKHRydXRoLCBlc3RpbWF0ZSkKCmVzdGltYXRlc19rZXJhc190YmwgJT4lIHJvY19hdWModHJ1dGgsIGNsYXNzX3Byb2IpCmBgYAoKKipQcmVjaXNpb24gYW5kIHJlY2FsbCwgZjEgc2NvcmUqCgpgYGB7cn0KdGliYmxlKAogIHByZWNpc2lvbiA9IGVzdGltYXRlc19rZXJhc190YmwgJT4lIHByZWNpc2lvbih0cnV0aCwgZXN0aW1hdGUpLAogIHJlY2FsbCA9IGVzdGltYXRlc19rZXJhc190YmwgJT4lIHJlY2FsbCh0cnV0aCwgZXN0aW1hdGUpLCAKICBmMSA9IGVzdGltYXRlc19rZXJhc190YmwgJT4lIGZfbWVhcyh0cnV0aCwgZXN0aW1hdGUsIGJldGEgPSAxKQopCmBgYAoKI0V4cGxhaW4gdGhlIG1vZGVsIHdpdGggTElNRQoKVGhlIGBsaW1lYCBwYWNrYWdlIGltcGxlbWVudHMgKipMSU1FKiogbWV0aG9kb2xvZ3kgaW4gUi4gT25lIHRoaW5nIHRvIG5vdGUgaXMgdGhhdCBpdOKAmXMgbm90IHNldHVwIG91dC1vZi10aGUtYm94IHRvIHdvcmsgd2l0aCBrZXJhcy4gVGhlIGdvb2QgbmV3cyBpcyB3aXRoIGEgZmV3IGZ1bmN0aW9ucyB3ZSBjYW4gZ2V0IGV2ZXJ5dGhpbmcgd29ya2luZyBwcm9wZXJseS4gV2XigJlsbCBuZWVkIHRvIG1ha2UgdHdvIGN1c3RvbSBmdW5jdGlvbnM6CgoqIGBtb2RlbF90eXBlYCAtIHVzZWQgdG8gdGVsbCBgbGltZWAgd2hhdCB0eXBlIG9mIGEgbW9kZWwgd2UncmUgZGVhbGluZyB3aXRoLiBJdCBjb3VsZCBiZSBjbGFzc2lmaWNhdGlvbiwgcmVncmVzc2lvbiwgc3Vydml2YWwuCiogYHByZWRpY3RfbW9kZWxgIC0gdXNlZCB0byBhbGxvdyBgbGltZWAgdG8gcGVyZm9ybSBwcmVkaWN0aW9uIHRoYXQgaXRzIGFsZ29yaXRobSBjYW4gaW50ZXJwcmV0LiAKClRoZSBmaXJzdCB0aGluZyB3ZSBuZWVkIHRvIGRvIGlzIGlkZW50aWZ5IHRoZSBjbGFzcyBvZiBvdXIgbW9kZWwgb2JqZWN0LiBXZSBkbyB0aGlzIHdpdGggdGhlIGBjbGFzcygpYCBmdW5jdGlvbi4KCmBgYHtyfQpjbGFzcyhtb2RlbF9rZXJhcykKYGBgCgpOZXh0IHdlIGNyZWF0ZSBvdXIgYG1vZGVsX3R5cGUoKWAgZnVuY3Rpb24uIEl04oCZcyBvbmx5IGlucHV0IGlzIGB4YCB0aGUga2VyYXMgbW9kZWwuIFRoZSBmdW5jdGlvbiBzaW1wbHkgcmV0dXJucyDigJxjbGFzc2lmaWNhdGlvbuKAnSwgd2hpY2ggdGVsbHMgTElNRSB3ZSBhcmUgY2xhc3NpZnlpbmcuCgpgYGB7cn0KbW9kZWxfdHlwZS5rZXJhcy5tb2RlbHMuU2VxdWVudGlhbCA8LSBmdW5jdGlvbih4LCAuLi4pewogICJjbGFzc2lmaWNhdGlvbiIKfQpgYGAKClRoZW4gd2UgY3JlYXRlIGBwcmVkaWN0X21vZGVsKClgIGZ1bmN0aW9uLCB3aGljaCB3cmFwcyBga2VyYXM6OnByZWRpY3RfcHJvYmEoKWAuIFRoZSB0cmljayBoZXJlIGlzIHRvIHJlYWxpemUgdGhhdCBpdOKAmXMgaW5wdXRzIG11c3QgYmUgYHhgICAtIGEgbW9kZWwsIGBuZXdkYXRhYCAgLSBhIGRhdGFmcmFtZSBvYmplY3QgKHRoaXMgaXMgaW1wb3J0YW50KSwgYW5kIGB0eXBlYCB3aGljaCBpcyBub3QgdXNlZCBidXQgY2FuIGJlIHVzZSB0byBzd2l0Y2ggLSB0aGUgb3V0cHV0IHR5cGUuIFRoZSBvdXRwdXQgaXMgYWxzbyBhIGxpdHRsZSB0cmlja3kgYmVjYXVzZSBpdCBtdXN0IGJlIGluIHRoZSBmb3JtYXQgb2YgcHJvYmFiaWxpdGllcyBieSBjbGFzc2lmaWNhdGlvbiAodGhpcyBpcyBpbXBvcnRhbnQ7IHNob3duIG5leHQpLgoKYGBge3J9CnByZWRpY3RfbW9kZWwua2VyYXMubW9kZWxzLlNlcXVlbnRpYWwgPC0gZnVuY3Rpb24oeCwgbmV3ZGF0YSwgdHlwZSwgLi4uKXsKICBwcmVkIDwtIHByZWRpY3RfcHJvYmEob2JqZWN0ID0geCwgeCA9IGFzLm1hdHJpeChuZXdkYXRhKSkKICBkYXRhLmZyYW1lKFllcyA9IHByZWQsIE5vID0gMSAtIHByZWQpCn0KYGBgCgpgYGB7cn0KcHJlZGljdF9tb2RlbCh4ID0gbW9kZWxfa2VyYXMsIG5ld2RhdGEgPSB4X3Rlc3RfdGJsLCB0eXBlID0gInJhdyIpICU+JSAKICB0aWJibGU6OmFzLnRpYmJsZSgpCmBgYAoKTGV0J3MgY3JlYXRlICoqZXhwbGFpbmVyKiogdXNpbmcgdGhlIGBsaW1lKClgIGZ1bmN0aW9uLiBXZSBuZWVkIHRvIHBhc3MgdG8gdGhlIGZ1bmN0aW9uIGEgdHJhaW5pbmcgZGF0YSBmcmFtZSwgbW9kZWwgKGBtb2RlbF9rZXJhc2ApIGFuZCBzZXQgYGJpbl9jb250aW51b3VzID0gRkFMU0VgLiBXZSBjb3VsZCB0ZWxsIHRoZSBhbGdvcml0aG0gdG8gYmluIGNvbnRpbnVvdXMgdmFyaWFibGVzLCBidXQgdGhpcyBtYXkgbm90IG1ha2Ugc2Vuc2UgZm9yIGNhdGVnb3JpY2FsIG51bWVyaWMgZGF0YSB0aGF0IHdlIGRpZG7igJl0IGNoYW5nZSB0byBmYWN0b3JzLiAKCmBgYHtyfQpleHBsYWluZXIgPC0gbGltZTo6bGltZSgKICB4ID0geF90cmFpbl90YmwsIAogIG1vZGVsID0gbW9kZWxfa2VyYXMsIAogIGJpbl9jb250aW51b3VzID0gRkFMU0UKKQpgYGAKCk5vdyB3ZSBjYW4gcnVuIGBleHBsYWluKClgIGZ1bmN0aW9uLCB3aGljaCByZXR1cm5zIG91ciBgZXhwbGFuYXRpb25gLiAKCmBgYHtyfQpleHBsYW5hdGlvbiA8LSBsaW1lOjpleHBsYWluKAogIHhfdGVzdF90YmxbMToxMCwgXSwgI29ubHkgZmlyc3QgMTAgY2FzZXMKICBleHBsYWluZXIgPSBleHBsYWluZXIsCiAgbl9sYWJlbHMgPSAxLCAKICBuX2ZlYXR1cmVzID0gNCwgCiAga2VybmVsX3dpZHRoID0gMC41CikKYGBgCgpXZSBzZXQgYG5fbGFiZWxzID0gMWAgYmVjYXVzZSB3ZSBjYXJlIGFib3V0IGV4cGxhaW5pbmcgYSBzaW5nbGUgY2xhc3MuIFNldHRpbmcgYG5fZmVhdHVyZXMgPSA0YCByZXR1cm5zIHRoZSB0b3AgZm91ciBmZWF0dXJlcyB0aGF0IGFyZSBjcml0aWNhbCB0byBlYWNoIGNhc2UuIEZpbmFsbHksIHNldHRpbmcgYGtlcm5lbF93aWR0aCA9IDAuNWAgYWxsb3dzIHVzIHRvIGluY3JlYXNlIHRoZSDigJxtb2RlbOKAnSB2YWx1ZSBieSBzaHJpbmtpbmcgdGhlIGxvY2FsaXplZCBldmFsdWF0aW9uLgoKKipGZWF0dXJlIGltcG9ydGFuY2UgdmlzdWFsaXphdGlvbioqCgpUaGUgcGF5b2ZmIGZvciB0aGUgd29yayB3ZSBwdXQgaW4gdXNpbmcgTElNRSBpcyB0aGlzICoqZmVhdHVyZSBpbXBvcnRhbmNlIHBsb3QqKi4gVGhpcyBhbGxvd3MgdXMgdG8gdmlzdWFsaXplIGVhY2ggb2YgdGhlIGZpcnN0IHRlbiBjYXNlcyAob2JzZXJ2YXRpb25zKSBmcm9tIHRoZSB0ZXN0IGRhdGEuIFRoZSB0b3AgZm91ciBmZWF0dXJlcyBmb3IgZWFjaCBjYXNlIGFyZSBzaG93bi4gTm90ZSB0aGF0IHRoZXkgYXJlIG5vdCB0aGUgc2FtZSBmb3IgZWFjaCBjYXNlLiBUaGUgZ3JlZW4gYmFycyBtZWFuIHRoYXQgdGhlIGZlYXR1cmUgc3VwcG9ydHMgdGhlIG1vZGVsIGNvbmNsdXNpb24sIGFuZCB0aGUgcmVkIGJhcnMgY29udHJhZGljdC4KCmBgYHtyfQpwbG90X2ZlYXR1cmVzKGV4cGxhbmF0aW9uKSArIAogIGxhYnModGl0bGUgPSAiTElNRSBGZWF0dXJlIEltcG9ydGFuY2UgVmlzdWFsaXphdGlvbiIsCiAgICAgICBzdWJ0aXRsZSA9ICJIb2xkIE91dCAoVGVzdCkgU2V0LCBGaXJzdCAxMCBDYXNlcyBTaG93biIpCmBgYAoKQW5vdGhlciBleGNlbGxlbnQgdmlzdWFsaXphdGlvbiBjYW4gYmUgcGVyZm9ybWVkIHVzaW5nIGBwbG90X2V4cGxhbmF0aW9ucygpYCwgd2hpY2ggcHJvZHVjZXMgYSBmYWNldHRlZCBoZWF0bWFwIG9mIGFsbCBjYXNlL2xhYmVsL2ZlYXR1cmUgY29tYmluYXRpb25zLgoKYGBge3J9CnBsb3RfZXhwbGFuYXRpb25zKGV4cGxhbmF0aW9uKSArIAogIGxhYnModGl0bGUgPSAiTElNRSBGZWF0dXJlIEltcG9ydGFuY2UgSGVhdG1hcCIsCiAgICAgICAgIHN1YnRpdGxlID0gIkhvbGQgT3V0IChUZXN0KSBTZXQsIEZpcnN0IDEwIENhc2VzIFNob3duIikKYGBgCgojIENoZWNrIEV4cGxhbmF0aW9uIHdpdGggQ29ycmVsYXRpb24gQW5hbHlzaXMKCk9uZSB0aGluZyB3ZSBuZWVkIHRvIGJlIGNhcmVmdWwgd2l0aCB0aGUgTElNRSB2aXN1YWxpemF0aW9uIGlzIHRoYXQgd2UgYXJlIG9ubHkgZG9pbmcgYSBzYW1wbGUgb2YgdGhlIGRhdGEsIGluIG91ciBjYXNlIHRoZSBmaXJzdCAxMCB0ZXN0IG9ic2VydmF0aW9ucy4gVGhlcmVmb3JlLCB3ZSBhcmUgZ2FpbmluZyBhIHZlcnkgbG9jYWxpemVkIHVuZGVyc3RhbmRpbmcgb2YgaG93IHRoZSBBTk4gd29ya3MuIEhvd2V2ZXIsIHdlIGFsc28gd2FudCB0byBrbm93IG9uIGZyb20gYSBnbG9iYWwgcGVyc3BlY3RpdmUgd2hhdCBkcml2ZXMgZmVhdHVyZSBpbXBvcnRhbmNlLgoKV2UgY2FuIHBlcmZvcm0gYSAqKmNvcnJlbGF0aW9uIGFuYWx5c2lzKiogb24gdGhlIHRyYWluaW5nIHNldCBhcyB3ZWxsIGFzIGdldCB1bmRlcnN0YW5kaW5nIHdoaWNoIGZlYXR1cmVzIGNvcnJlbGF0ZWQgZ2xvYmFsbHkgdG8gIkNodXJuIi4gV2UnbGwgdXNlIGBjb3JycmAgcGFja2FnZSwgd2hpY2ggcGVyZm9ybXMgdGlkeSBjb3JyZWxhdGlvbnMgd2l0aCBhIGZ1bmN0aW9uIGBjb3JyZWxhdGUoKWAKCmBgYHtyfQpjb3Jycl9hbmFseXNpcyA8LSB4X3RyYWluX3RibCAlPiUgCiAgbXV0YXRlKENodXJuID0geV90cmFpbl92ZWMpICU+JSAKICBjb3JyZWxhdGUoKSAlPiUgCiAgZm9jdXMoQ2h1cm4pICU+JSAKICByZW5hbWUoZmVhdHVyZSA9IHJvd25hbWUpICU+JSAKICBhcnJhbmdlKGRlc2MoYWJzKENodXJuKSkpICU+JSAKICBtdXRhdGUoZmVhdHVyZSA9IGFzLmZhY3RvcihmZWF0dXJlKSkKYGBgCgpUaGUgY29ycmVsYXRpb24gdmlzdWFsaXphdGlvbiBoZWxwcyBpbiBkaXN0aW5ndWlzaGluZyB3aGljaCBmZWF0dXJlcyBhcmUgcmVsYXZhbnQgdG8gQ2h1cm4uCgpgYGB7cn0KY29ycnJfYW5hbHlzaXMgJT4lIAogIGdncGxvdChhZXMoeCA9IENodXJuLCB5ID0gZm9yY2F0czo6ZmN0X3Jlb3JkZXIoZmVhdHVyZSwgZGVzYyhDaHVybikpKSkgKyAKICBnZW9tX3BvaW50KCkgKyAKICAjIFBvc2l0aXZlIENvcnJlbGF0aW9ucyAtIENvbnRyaWJ1dGUgdG8gY2h1cm4KICBnZW9tX3NlZ21lbnQoYWVzKHhlbmQgPSAwLCB5ZW5kID0gZmVhdHVyZSksIAogICAgICAgICAgICAgICBjb2xvciA9IHBhbGV0dGVfbGlnaHQoKVtbMl1dLAogICAgICAgICAgICAgICBkYXRhID0gY29ycnJfYW5hbHlzaXMgJT4lIGZpbHRlcihDaHVybiA+IDApKSArIAogIGdlb21fcG9pbnQoY29sb3IgPSBwYWxldHRlX2xpZ2h0KClbWzJdXSwgCiAgICAgICAgICAgICBkYXRhID0gY29ycnJfYW5hbHlzaXMgJT4lIGZpbHRlcihDaHVybiA+IDApKSArIAogICMgTmVnYXRpdmUgQ29ycmVsYXRpb25zIC0gUHJldmVudCBjaHVybgogIGdlb21fc2VnbWVudChhZXMoeGVuZCA9IDAsIHllbmQgPSBmZWF0dXJlKSwgCiAgICAgICAgICAgICAgIGNvbG9yID0gcGFsZXR0ZV9saWdodCgpW1sxXV0sIAogICAgICAgICAgICAgICBkYXRhID0gY29ycnJfYW5hbHlzaXMgJT4lIGZpbHRlcihDaHVybiA8IDApKSArCiAgZ2VvbV9wb2ludChjb2xvciA9IHBhbGV0dGVfbGlnaHQoKVtbMV1dLCAKICAgICAgICAgICAgIGRhdGEgPSBjb3Jycl9hbmFseXNpcyAlPiUgZmlsdGVyKENodXJuIDwgMCkpICsgCiAgI3ZlcnRpY2FsIGxpbmVzCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gMCwgY29sb3IgPSBwYWxldHRlX2xpZ2h0KClbWzVdXSwgc2l6ZSA9IDEsIGxpbmV0eXBlID0gMikgKyAKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAtMC4yNSwgY29sb3IgPSBwYWxldHRlX2xpZ2h0KClbWzVdXSwgc2l6ZSA9IDEsIGxpbmV0eXBlID0gMikgKyAKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAwLjI1LCBjb2xvciA9IHBhbGV0dGVfbGlnaHQoKVtbNV1dLCBzaXplID0gMSwgbGluZXR5cGUgPSAyKSArIAogIHRoZW1lX3RxKCkgKyAKICBsYWJzKHRpdGxlID0gIkNodXJuIENvcnJlbGF0aW9uIEFuYWx5c2lzIiwKICAgICAgIHN1YnRpdGxlID0gIlBvc2l0aXZlIENvcnJlbGF0aW9ucyAoY29udHJpYnV0ZSB0byBjaHVybiksIE5lZ2F0aXZlIENvcnJlbGF0aW9ucyAocHJldmVudCBjaHVybikiLAogICAgICAgeSA9ICJGZWF0dXJlIEltcG9ydGFuY2UiKQpgYGAKVGhlIGNvcnJlbGF0aW9uIGFuYWx5c2lzIGhlbHBzIHVzIHF1aWNrbHkgZGlzc2VtaW5hdGUgd2hpY2ggZmVhdHVyZXMgdGhhdCB0aGUgTElNRSBhbmFseXNpcyBtYXkgYmUgZXhjbHVkaW5nLiBXZSBjYW4gc2VlIHRoYXQgdGhlIGZvbGxvd2luZyBmZWF0dXJlcyBhcmUgaGlnaGx5IGNvcnJlbGF0ZWQgKG1hZ25pdHVkZSA+IDAuMjUpOgoKKipJbmNyZWFzZXMgTGlrZWxpaG9vZCBvZiBDaHVybiAoUmVkKSoqOiAtIFRlbnVyZSA9IEJpbiAxICg8MTIgTW9udGhzKSAtIEludGVybmV0IFNlcnZpY2UgPSDigJxGaWJlciBPcHRpY+KAnSAtIFBheW1lbnQgTWV0aG9kID0g4oCcRWxlY3Ryb25pYyBDaGVja+KAnQoKKipEZWNyZWFzZXMgTGlrZWxpaG9vZCBvZiBDaHVybiAoQmx1ZSkqKjogLSBDb250cmFjdCA9IOKAnFR3byBZZWFy4oCdIC0gVG90YWwgQ2hhcmdlcyAoTm90ZSB0aGF0IHRoaXMgbWF5IGJlIGEgYmlwcm9kdWN0IG9mIGFkZGl0aW9uYWwgc2VydmljZXMgc3VjaCBhcyBPbmxpbmUgU2VjdXJpdHkpCgoKCgo=