Chapter 5: Selecting models: a case study in churn prediction

In the final chapter of this course, you’ll learn how to use resamples() to compare multiple models and select (or ensemble) the best one(s).

5.1: Why reuse a trainControl?

Why reuse a trainControl?

Answer the question

50 XP

Possible Answers

  1. So you can use the same summaryFunction and tuning parameters for multiple models.

  2. So you don’t have to repeat code when fitting multiple models.

  3. So you can compare models on the exact same training and test data.

  4. All of the above.

5.2: Make custom train/test indices

As you saw in the video, for this chapter you will focus on a real-world dataset that brings together all of the concepts discussed in the previous chapters.

The churn dataset contains data on a variety of telecom customers and the modeling challenge is to predict which customers will cancel their service (or churn).

In this chapter, you will be exploring two different types of predictive models: glmnet and rf, so the first order of business is to create a reusable trainControl object you can use to reliably compare them.

Instructions

100 XP

  • churn_x and churn_y are loaded in your workspace.

  • Use createFolds() to create 5 CV folds on churn_y, your target variable for this exercise.

  • Pass them to trainControl() to create a reusable trainControl for comparing models.

Churn<-load(file="Churn.RData")
str(Churn) #Churn contains churn_x, churn_y in a list that can be called upon later
 chr [1:2] "churn_x" "churn_y"
str(churn_y)
 Factor w/ 2 levels "no","yes": 1 1 1 1 1 1 1 1 1 1 ...
str(churn_x)
'data.frame':   250 obs. of  70 variables:
 $ stateAK                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAL                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAR                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAZ                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCO                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateDC                      : int  0 0 0 0 0 0 0 0 0 1 ...
 $ stateDE                      : int  0 0 0 1 0 0 0 0 0 0 ...
 $ stateFL                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateGA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateHI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateIA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateID                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateIL                      : int  0 0 0 0 0 0 0 1 0 0 ...
 $ stateIN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateKS                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateKY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateLA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMD                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateME                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMO                      : int  0 1 0 0 0 0 0 0 0 0 ...
 $ stateMS                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNC                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateND                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNE                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNH                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNJ                      : int  0 0 0 0 0 0 0 0 1 0 ...
 $ stateNM                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNV                      : int  0 0 0 0 0 0 1 0 0 0 ...
 $ stateNY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOH                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOK                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOR                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ statePA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateRI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateSC                      : int  1 0 0 0 0 0 0 0 0 0 ...
 $ stateSD                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateTN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateTX                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateUT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateVA                      : int  0 0 0 0 0 1 0 0 0 0 ...
 $ stateVT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateWA                      : int  0 0 0 0 1 0 0 0 0 0 ...
 $ stateWI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateWV                      : int  0 0 1 0 0 0 0 0 0 0 ...
 $ stateWY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ account_length               : int  137 83 48 67 143 163 100 151 139 17 ...
 $ area_codearea_code_415       : int  0 1 1 1 0 1 1 0 1 0 ...
 $ area_codearea_code_510       : int  1 0 0 0 1 0 0 0 0 1 ...
 $ international_planyes        : int  0 0 0 0 0 0 0 0 0 0 ...
 $ voice_mail_planyes           : int  0 0 1 0 0 0 1 0 1 1 ...
 $ number_vmail_messages        : int  0 0 34 0 0 0 39 0 43 30 ...
 $ total_day_minutes            : num  110 197 198 164 133 ...
 $ total_day_calls              : int  112 117 70 79 107 100 74 106 85 101 ...
 $ total_day_charge             : num  18.7 33.4 33.7 28 22.7 ...
 $ total_eve_minutes            : num  224 272 274 110 224 ...
 $ total_eve_calls              : int  88 89 121 108 117 46 80 87 82 85 ...
 $ total_eve_charge             : num  19 23.12 23.26 9.38 19.03 ...
 $ total_night_minutes          : num  248 200 218 204 180 ...
 $ total_night_calls            : int  96 62 71 102 85 116 89 88 105 130 ...
 $ total_night_charge           : num  11.14 9 9.81 9.18 8.12 ...
 $ total_intl_minutes           : num  17.8 10.1 7.6 9.8 10.2 12.8 11.2 11.8 8.3 10.3 ...
 $ total_intl_calls             : int  2 11 4 2 13 3 4 5 5 2 ...
 $ total_intl_charge            : num  4.81 2.73 2.05 2.65 2.75 3.46 3.02 3.19 2.24 2.78 ...
 $ number_customer_service_calls: int  1 3 1 1 1 5 2 0 2 3 ...
# Create custom indices: myFolds
myFolds <- createFolds(churn_y, k = 5)
# Create reusable trainControl object: myControl
myControl <- trainControl(
  summaryFunction = twoClassSummary,
  classProbs = TRUE, # IMPORTANT!
  verboseIter = TRUE,
  savePredictions = TRUE,
  index = myFolds
)

5.3: Fit the baseline model

Now that you have a reusable trainControl object called myControl, you can start fitting different predictive models to your churn dataset and evaluate their predictive accuracy.

You’ll start with one of my favorite models, glmnet, which penalizes linear and logistic regression models on the size and number of coefficients to help prevent overfitting.

Instructions

100 XP

# Fit glmnet model: model_glmnet
model_glmnet <- train(
  x = churn_x, y = churn_y,
  metric = "ROC",
  method = "glmnet",
  trControl = myControl
)
+ Fold1: alpha=0.10, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold1: alpha=0.10, lambda=0.01821 
+ Fold1: alpha=0.55, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold1: alpha=0.55, lambda=0.01821 
+ Fold1: alpha=1.00, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold1: alpha=1.00, lambda=0.01821 
+ Fold2: alpha=0.10, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold2: alpha=0.10, lambda=0.01821 
+ Fold2: alpha=0.55, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold2: alpha=0.55, lambda=0.01821 
+ Fold2: alpha=1.00, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold2: alpha=1.00, lambda=0.01821 
+ Fold3: alpha=0.10, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold3: alpha=0.10, lambda=0.01821 
+ Fold3: alpha=0.55, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold3: alpha=0.55, lambda=0.01821 
+ Fold3: alpha=1.00, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold3: alpha=1.00, lambda=0.01821 
+ Fold4: alpha=0.10, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold4: alpha=0.10, lambda=0.01821 
+ Fold4: alpha=0.55, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold4: alpha=0.55, lambda=0.01821 
+ Fold4: alpha=1.00, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold4: alpha=1.00, lambda=0.01821 
+ Fold5: alpha=0.10, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold5: alpha=0.10, lambda=0.01821 
+ Fold5: alpha=0.55, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold5: alpha=0.55, lambda=0.01821 
+ Fold5: alpha=1.00, lambda=0.01821 
one multinomial or binomial class has fewer than 8  observations; dangerous ground
- Fold5: alpha=1.00, lambda=0.01821 
Aggregating results
Selecting tuning parameters
Fitting alpha = 0.55, lambda = 0.0182 on full training set

5.4: Random forest drawback

What’s the drawback of using a random forest model for churn prediction?

Answer the question

50 XP

Possible Answers

  1. Tree-based models are usually less accurate than linear models.

  2. You no longer have model coefficients to help interpret the model. [ans]

  3. Nobody else uses random forests to predict churn.

5.4: Random forest with custom trainControl

Another one of my favorite models is the random forest, which combines an ensemble of non-linear decision trees into a highly flexible (and usually quite accurate) model.

Rather than using the classic randomForest package, you’ll be using the ranger package, which is a re-implementation of randomForest that produces almost the exact same results, but is faster, more stable, and uses less memory. I highly recommend it as a starting point for random forest modeling in R.

Instructions

100 XP

  • churn_x and churn_y are loaded in your workspace.

  • Fit a random forest model to the churn dataset. Be sure to use myControl as the trainControl like you’ve done before and implement the “ranger” method.

# Fit random forest: model_rf
model_rf <- train(
  x = churn_x, y = churn_y,
  metric = "ROC",
  method = "ranger",
  trControl = myControl
)
+ Fold1: mtry= 2, min.node.size=1, splitrule=gini 
- Fold1: mtry= 2, min.node.size=1, splitrule=gini 
+ Fold1: mtry=36, min.node.size=1, splitrule=gini 
- Fold1: mtry=36, min.node.size=1, splitrule=gini 
+ Fold1: mtry=70, min.node.size=1, splitrule=gini 
- Fold1: mtry=70, min.node.size=1, splitrule=gini 
+ Fold1: mtry= 2, min.node.size=1, splitrule=extratrees 
- Fold1: mtry= 2, min.node.size=1, splitrule=extratrees 
+ Fold1: mtry=36, min.node.size=1, splitrule=extratrees 
- Fold1: mtry=36, min.node.size=1, splitrule=extratrees 
+ Fold1: mtry=70, min.node.size=1, splitrule=extratrees 
- Fold1: mtry=70, min.node.size=1, splitrule=extratrees 
+ Fold2: mtry= 2, min.node.size=1, splitrule=gini 
- Fold2: mtry= 2, min.node.size=1, splitrule=gini 
+ Fold2: mtry=36, min.node.size=1, splitrule=gini 
- Fold2: mtry=36, min.node.size=1, splitrule=gini 
+ Fold2: mtry=70, min.node.size=1, splitrule=gini 
- Fold2: mtry=70, min.node.size=1, splitrule=gini 
+ Fold2: mtry= 2, min.node.size=1, splitrule=extratrees 
- Fold2: mtry= 2, min.node.size=1, splitrule=extratrees 
+ Fold2: mtry=36, min.node.size=1, splitrule=extratrees 
- Fold2: mtry=36, min.node.size=1, splitrule=extratrees 
+ Fold2: mtry=70, min.node.size=1, splitrule=extratrees 
- Fold2: mtry=70, min.node.size=1, splitrule=extratrees 
+ Fold3: mtry= 2, min.node.size=1, splitrule=gini 
- Fold3: mtry= 2, min.node.size=1, splitrule=gini 
+ Fold3: mtry=36, min.node.size=1, splitrule=gini 
- Fold3: mtry=36, min.node.size=1, splitrule=gini 
+ Fold3: mtry=70, min.node.size=1, splitrule=gini 
- Fold3: mtry=70, min.node.size=1, splitrule=gini 
+ Fold3: mtry= 2, min.node.size=1, splitrule=extratrees 
- Fold3: mtry= 2, min.node.size=1, splitrule=extratrees 
+ Fold3: mtry=36, min.node.size=1, splitrule=extratrees 
- Fold3: mtry=36, min.node.size=1, splitrule=extratrees 
+ Fold3: mtry=70, min.node.size=1, splitrule=extratrees 
- Fold3: mtry=70, min.node.size=1, splitrule=extratrees 
+ Fold4: mtry= 2, min.node.size=1, splitrule=gini 
- Fold4: mtry= 2, min.node.size=1, splitrule=gini 
+ Fold4: mtry=36, min.node.size=1, splitrule=gini 
- Fold4: mtry=36, min.node.size=1, splitrule=gini 
+ Fold4: mtry=70, min.node.size=1, splitrule=gini 
- Fold4: mtry=70, min.node.size=1, splitrule=gini 
+ Fold4: mtry= 2, min.node.size=1, splitrule=extratrees 
- Fold4: mtry= 2, min.node.size=1, splitrule=extratrees 
+ Fold4: mtry=36, min.node.size=1, splitrule=extratrees 
- Fold4: mtry=36, min.node.size=1, splitrule=extratrees 
+ Fold4: mtry=70, min.node.size=1, splitrule=extratrees 
- Fold4: mtry=70, min.node.size=1, splitrule=extratrees 
+ Fold5: mtry= 2, min.node.size=1, splitrule=gini 
- Fold5: mtry= 2, min.node.size=1, splitrule=gini 
+ Fold5: mtry=36, min.node.size=1, splitrule=gini 
- Fold5: mtry=36, min.node.size=1, splitrule=gini 
+ Fold5: mtry=70, min.node.size=1, splitrule=gini 
- Fold5: mtry=70, min.node.size=1, splitrule=gini 
+ Fold5: mtry= 2, min.node.size=1, splitrule=extratrees 
- Fold5: mtry= 2, min.node.size=1, splitrule=extratrees 
+ Fold5: mtry=36, min.node.size=1, splitrule=extratrees 
- Fold5: mtry=36, min.node.size=1, splitrule=extratrees 
+ Fold5: mtry=70, min.node.size=1, splitrule=extratrees 
- Fold5: mtry=70, min.node.size=1, splitrule=extratrees 
Aggregating results
Selecting tuning parameters
Fitting mtry = 70, splitrule = extratrees, min.node.size = 1 on full training set

5.5: Create a resamples object

Now that you have fit two models to the churn dataset, it’s time to compare their out-of-sample predictions and choose which one is the best model for your dataset.

You can compare models in caret using the resamples() function, provided they have the same training data and use the same trainControl object with preset cross-validation folds. resamples() takes as input a list of models and can be used to compare dozens of models at once (though in this case you are only comparing two models).

Instructions

100 XP

  • model_glmnet and model_rf are loaded in your workspace.

  • Create a list() containing the glmnet model as item1 and the ranger model as item2.

  • Pass this list to the resamples() function and save the resulting object as resamples.

  • Summarize the results by calling summary() on resamples.

# Create model_list
model_list <- list(item1 = model_glmnet, item2 = model_rf)
# Pass model_list to resamples(): resamples
resamples<-resamples(model_list)
# Summarize the results
summary(resamples)

Call:
summary.resamples(object = resamples)

Models: item1, item2 
Number of resamples: 5 

ROC 
           Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
item1 0.5250286 0.5711760 0.6129531 0.6179179 0.6756044 0.7048276    0
item2 0.6497569 0.6539429 0.6876923 0.7002315 0.7328691 0.7768966    0

Sens 
           Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
item1 0.9142857 0.9200000 0.9252874 0.9381215 0.9540230 0.9770115    0
item2 0.9022989 0.9257143 0.9712644 0.9529918 0.9714286 0.9942529    0

Spec 
            Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
item1 0.08000000 0.1153846 0.1153846 0.1172308 0.1153846 0.1600000    0
item2 0.03846154 0.0400000 0.0800000 0.1163077 0.1153846 0.3076923    0

5.6: Create a box-and-whisker plot

caret provides a variety of methods to use for comparing models. All of these methods are based on the resamples() function. My favorite is the box-and-whisker plot, which allows you to compare the distribution of predictive accuracy (in this case AUC) for the two models.

In general, you want the model with the higher median AUC, as well as a smaller range between min and max AUC.

You can make this plot using the bwplot() function, which makes a box and whisker plot of the model’s out of sample scores. Box and whisker plots show the median of each distribution as a line and the interquartile range of each distribution as a box around the median line. You can pass the metric = “ROC” argument to the bwplot() function to show a plot of the model’s out-of-sample ROC scores and choose the model with the highest median ROC.

If you do not specify a metric to plot, bwplot() will automatically plot 3 of them.

Instructions

100 XP

  • Pass the resamples object to the bwplot() function to make a box-and-whisker plot. Look at the resulting plot and note which model has the higher median ROC statistic. Be sure to specify which metric you want to plot.
# Create bwplot
bwplot(resamples, metric = "ROC")

5.7: Create a scatterplot

Another useful plot for comparing models is the scatterplot, also known as the xy-plot. This plot shows you how similar the two models’ performances are on different folds.

It’s particularly useful for identifying if one model is consistently better than the other across all folds, or if there are situations when the inferior model produces better predictions on a particular subset of the data.

Instructions

100 XP

Pass the resamples object to the xyplot() function. Look at the resulting plot and note how similar the two models’ predictions are (or are not) on the different folds. Be sure to specify which metric you want to plot.

# Create xyplot
xyplot(resamples,metric="ROC")

5.8: Ensembling models

That concludes the course! As a teaser for a future course on making ensembles of caret models, I’ll show you how to fit a stacked ensemble of models using the caretEnsemble package.

caretEnsemble provides the caretList() function for creating multiple caret models at once on the same dataset, using the same resampling folds. You can also create your own lists of caret models.

In this exercise, I’ve made a caretList for you, containing the glmnet and ranger models you fit on the churn dataset. Use the caretStack() function to make a stack of caret models, with the two sub-models (glmnet and ranger) feeding into another (hopefully more accurate!) caret model.

Instructions

100 XP

  • Call the caretStack() function with two arguments, model_list and method = “glm”, to ensemble the two models using a logistic regression. Store the result as stack.

  • Summarize the resulting model with the summary() function.

str(churn_x)
'data.frame':   250 obs. of  70 variables:
 $ stateAK                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAL                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAR                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateAZ                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCO                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateCT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateDC                      : int  0 0 0 0 0 0 0 0 0 1 ...
 $ stateDE                      : int  0 0 0 1 0 0 0 0 0 0 ...
 $ stateFL                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateGA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateHI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateIA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateID                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateIL                      : int  0 0 0 0 0 0 0 1 0 0 ...
 $ stateIN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateKS                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateKY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateLA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMD                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateME                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMO                      : int  0 1 0 0 0 0 0 0 0 0 ...
 $ stateMS                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateMT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNC                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateND                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNE                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNH                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNJ                      : int  0 0 0 0 0 0 0 0 1 0 ...
 $ stateNM                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateNV                      : int  0 0 0 0 0 0 1 0 0 0 ...
 $ stateNY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOH                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOK                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateOR                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ statePA                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateRI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateSC                      : int  1 0 0 0 0 0 0 0 0 0 ...
 $ stateSD                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateTN                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateTX                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateUT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateVA                      : int  0 0 0 0 0 1 0 0 0 0 ...
 $ stateVT                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateWA                      : int  0 0 0 0 1 0 0 0 0 0 ...
 $ stateWI                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ stateWV                      : int  0 0 1 0 0 0 0 0 0 0 ...
 $ stateWY                      : int  0 0 0 0 0 0 0 0 0 0 ...
 $ account_length               : int  137 83 48 67 143 163 100 151 139 17 ...
 $ area_codearea_code_415       : int  0 1 1 1 0 1 1 0 1 0 ...
 $ area_codearea_code_510       : int  1 0 0 0 1 0 0 0 0 1 ...
 $ international_planyes        : int  0 0 0 0 0 0 0 0 0 0 ...
 $ voice_mail_planyes           : int  0 0 1 0 0 0 1 0 1 1 ...
 $ number_vmail_messages        : int  0 0 34 0 0 0 39 0 43 30 ...
 $ total_day_minutes            : num  110 197 198 164 133 ...
 $ total_day_calls              : int  112 117 70 79 107 100 74 106 85 101 ...
 $ total_day_charge             : num  18.7 33.4 33.7 28 22.7 ...
 $ total_eve_minutes            : num  224 272 274 110 224 ...
 $ total_eve_calls              : int  88 89 121 108 117 46 80 87 82 85 ...
 $ total_eve_charge             : num  19 23.12 23.26 9.38 19.03 ...
 $ total_night_minutes          : num  248 200 218 204 180 ...
 $ total_night_calls            : int  96 62 71 102 85 116 89 88 105 130 ...
 $ total_night_charge           : num  11.14 9 9.81 9.18 8.12 ...
 $ total_intl_minutes           : num  17.8 10.1 7.6 9.8 10.2 12.8 11.2 11.8 8.3 10.3 ...
 $ total_intl_calls             : int  2 11 4 2 13 3 4 5 5 2 ...
 $ total_intl_charge            : num  4.81 2.73 2.05 2.65 2.75 3.46 3.02 3.19 2.24 2.78 ...
 $ number_customer_service_calls: int  1 3 1 1 1 5 2 0 2 3 ...
# install.packages("caretEnsemble")
# library(caretEnsemble)
# model_list <- caretList(
#    ~ ., churn_x,
#   trControl = myControl,
#   methodList = c("glm", "rpart", "rf", "gbm", "glmnet")
#   )
# # Create ensemble model: stack
# stack <- caretStack(model_list,method="glm")
# 
# # Look at summary
# summary(stack)
LS0tDQp0aXRsZTogIkRhdGFjYW1wIFIgLSBNYWNoaW5lIExlYXJuaW5nIFRvb2xib3ggOiBDaGFwdGVyIDUiDQphdXRob3I6ICJDaGVuIFdlaXFpYW5nIg0KZGF0ZTogIk5vdmVtYmVyIDI5LCAyMDE4Ig0Kb3V0cHV0OiBodG1sX25vdGVib29rDQotLS0NCg0KIyBDaGFwdGVyIDU6IFNlbGVjdGluZyBtb2RlbHM6IGEgY2FzZSBzdHVkeSBpbiBjaHVybiBwcmVkaWN0aW9uDQoNCkluIHRoZSBmaW5hbCBjaGFwdGVyIG9mIHRoaXMgY291cnNlLCB5b3UnbGwgbGVhcm4gaG93IHRvIHVzZSByZXNhbXBsZXMoKSB0byBjb21wYXJlIG11bHRpcGxlIG1vZGVscyBhbmQgc2VsZWN0IChvciBlbnNlbWJsZSkgdGhlIGJlc3Qgb25lKHMpLg0KDQoNCmBgYHtyLCBlY2hvPVRSVUUscmVzdWx0cz0naGlkZScsZmlnLmtlZXA9J2FsbCd9DQpsaWJyYXJ5KGRwbHlyKQ0KbGlicmFyeShnZ3Bsb3QyKQ0KI2luc3RhbGwucGFja2FnZXMoIm1sYmVuY2giKQ0KI2xpYnJhcnkobWxiZW5jaCkgIyBzb25hciBkYXRhDQppbnN0YWxsLnBhY2thZ2VzKCJjYXJldCIpDQpsaWJyYXJ5KGNhcmV0KQ0KbGlicmFyeShjYVRvb2xzKQ0KYGBgDQoNCiMjIDUuMTogV2h5IHJldXNlIGEgdHJhaW5Db250cm9sPw0KDQpXaHkgcmV1c2UgYSB0cmFpbkNvbnRyb2w/DQoNCkFuc3dlciB0aGUgcXVlc3Rpb24NCg0KNTAgWFANCg0KUG9zc2libGUgQW5zd2Vycw0KDQoxLiBTbyB5b3UgY2FuIHVzZSB0aGUgc2FtZSBzdW1tYXJ5RnVuY3Rpb24gYW5kIHR1bmluZyBwYXJhbWV0ZXJzIGZvciBtdWx0aXBsZSBtb2RlbHMuDQoNCjIuIFNvIHlvdSBkb24ndCBoYXZlIHRvIHJlcGVhdCBjb2RlIHdoZW4gZml0dGluZyBtdWx0aXBsZSBtb2RlbHMuDQoNCjMuIFNvIHlvdSBjYW4gY29tcGFyZSBtb2RlbHMgb24gdGhlIGV4YWN0IHNhbWUgdHJhaW5pbmcgYW5kIHRlc3QgZGF0YS4NCg0KNC4gQWxsIG9mIHRoZSBhYm92ZS4NCg0KIyMgNS4yOiBNYWtlIGN1c3RvbSB0cmFpbi90ZXN0IGluZGljZXMNCg0KQXMgeW91IHNhdyBpbiB0aGUgdmlkZW8sIGZvciB0aGlzIGNoYXB0ZXIgeW91IHdpbGwgZm9jdXMgb24gYSByZWFsLXdvcmxkIGRhdGFzZXQgdGhhdCBicmluZ3MgdG9nZXRoZXIgYWxsIG9mIHRoZSBjb25jZXB0cyBkaXNjdXNzZWQgaW4gdGhlIHByZXZpb3VzIGNoYXB0ZXJzLg0KDQpUaGUgY2h1cm4gZGF0YXNldCBjb250YWlucyBkYXRhIG9uIGEgdmFyaWV0eSBvZiB0ZWxlY29tIGN1c3RvbWVycyBhbmQgdGhlIG1vZGVsaW5nIGNoYWxsZW5nZSBpcyB0byBwcmVkaWN0IHdoaWNoIGN1c3RvbWVycyB3aWxsIGNhbmNlbCB0aGVpciBzZXJ2aWNlIChvciBjaHVybikuDQoNCkluIHRoaXMgY2hhcHRlciwgeW91IHdpbGwgYmUgZXhwbG9yaW5nIHR3byBkaWZmZXJlbnQgdHlwZXMgb2YgcHJlZGljdGl2ZSBtb2RlbHM6IGdsbW5ldCBhbmQgcmYsIHNvIHRoZSBmaXJzdCBvcmRlciBvZiBidXNpbmVzcyBpcyB0byBjcmVhdGUgYSByZXVzYWJsZSB0cmFpbkNvbnRyb2wgb2JqZWN0IHlvdSBjYW4gdXNlIHRvIHJlbGlhYmx5IGNvbXBhcmUgdGhlbS4NCg0KSW5zdHJ1Y3Rpb25zDQoNCjEwMCBYUA0KDQotIGNodXJuX3ggYW5kIGNodXJuX3kgYXJlIGxvYWRlZCBpbiB5b3VyIHdvcmtzcGFjZS4NCg0KLSBVc2UgY3JlYXRlRm9sZHMoKSB0byBjcmVhdGUgNSBDViBmb2xkcyBvbiBjaHVybl95LCB5b3VyIHRhcmdldCB2YXJpYWJsZSBmb3IgdGhpcyBleGVyY2lzZS4NCg0KLSBQYXNzIHRoZW0gdG8gdHJhaW5Db250cm9sKCkgdG8gY3JlYXRlIGEgcmV1c2FibGUgdHJhaW5Db250cm9sIGZvciBjb21wYXJpbmcgbW9kZWxzLg0KDQpgYGB7cn0NCkNodXJuPC1sb2FkKGZpbGU9IkNodXJuLlJEYXRhIikNCnN0cihDaHVybikgI0NodXJuIGNvbnRhaW5zIGNodXJuX3gsIGNodXJuX3kgaW4gYSBsaXN0IHRoYXQgY2FuIGJlIGNhbGxlZCB1cG9uIGxhdGVyDQpzdHIoY2h1cm5feSkNCnN0cihjaHVybl94KQ0KDQpgYGANCg0KYGBge3J9DQojIENyZWF0ZSBjdXN0b20gaW5kaWNlczogbXlGb2xkcw0KbXlGb2xkcyA8LSBjcmVhdGVGb2xkcyhjaHVybl95LCBrID0gNSkNCg0KIyBDcmVhdGUgcmV1c2FibGUgdHJhaW5Db250cm9sIG9iamVjdDogbXlDb250cm9sDQpteUNvbnRyb2wgPC0gdHJhaW5Db250cm9sKA0KICBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnksDQogIGNsYXNzUHJvYnMgPSBUUlVFLCAjIElNUE9SVEFOVCENCiAgdmVyYm9zZUl0ZXIgPSBUUlVFLA0KICBzYXZlUHJlZGljdGlvbnMgPSBUUlVFLA0KICBpbmRleCA9IG15Rm9sZHMNCikNCmBgYA0KDQojIDUuMzogRml0IHRoZSBiYXNlbGluZSBtb2RlbA0KDQpOb3cgdGhhdCB5b3UgaGF2ZSBhIHJldXNhYmxlIHRyYWluQ29udHJvbCBvYmplY3QgY2FsbGVkIG15Q29udHJvbCwgeW91IGNhbiBzdGFydCBmaXR0aW5nIGRpZmZlcmVudCBwcmVkaWN0aXZlIG1vZGVscyB0byB5b3VyIGNodXJuIGRhdGFzZXQgYW5kIGV2YWx1YXRlIHRoZWlyIHByZWRpY3RpdmUgYWNjdXJhY3kuDQoNCllvdSdsbCBzdGFydCB3aXRoIG9uZSBvZiBteSBmYXZvcml0ZSBtb2RlbHMsIGdsbW5ldCwgd2hpY2ggcGVuYWxpemVzIGxpbmVhciBhbmQgbG9naXN0aWMgcmVncmVzc2lvbiBtb2RlbHMgb24gdGhlIHNpemUgYW5kIG51bWJlciBvZiBjb2VmZmljaWVudHMgdG8gaGVscCBwcmV2ZW50IG92ZXJmaXR0aW5nLg0KDQpJbnN0cnVjdGlvbnMNCg0KMTAwIFhQDQoNCi0gRml0IGEgZ2xtbmV0IG1vZGVsIHRvIHRoZSBjaHVybiBkYXRhc2V0IGNhbGxlZCBtb2RlbF9nbG1uZXQuIE1ha2Ugc3VyZSB0byB1c2UgbXlDb250cm9sLCB3aGljaCB5b3UgY3JlYXRlZCBpbiB0aGUgZmlyc3QgZXhlcmNpc2UgYW5kIGlzIGF2YWlsYWJsZSBpbiB5b3VyIHdvcmtzcGFjZSwgYXMgdGhlIHRyYWluQ29udHJvbCBvYmplY3QuDQoNCmBgYHtyfQ0KIyBGaXQgZ2xtbmV0IG1vZGVsOiBtb2RlbF9nbG1uZXQNCm1vZGVsX2dsbW5ldCA8LSB0cmFpbigNCiAgeCA9IGNodXJuX3gsIHkgPSBjaHVybl95LA0KICBtZXRyaWMgPSAiUk9DIiwNCiAgbWV0aG9kID0gImdsbW5ldCIsDQogIHRyQ29udHJvbCA9IG15Q29udHJvbA0KKQ0KYGBgDQoNCiMjIDUuNDogUmFuZG9tIGZvcmVzdCBkcmF3YmFjaw0KDQpXaGF0J3MgdGhlIGRyYXdiYWNrIG9mIHVzaW5nIGEgcmFuZG9tIGZvcmVzdCBtb2RlbCBmb3IgY2h1cm4gcHJlZGljdGlvbj8NCg0KQW5zd2VyIHRoZSBxdWVzdGlvbg0KDQo1MCBYUA0KDQpQb3NzaWJsZSBBbnN3ZXJzDQoNCjEuIFRyZWUtYmFzZWQgbW9kZWxzIGFyZSB1c3VhbGx5IGxlc3MgYWNjdXJhdGUgdGhhbiBsaW5lYXIgbW9kZWxzLg0KDQoyLiBZb3Ugbm8gbG9uZ2VyIGhhdmUgbW9kZWwgY29lZmZpY2llbnRzIHRvIGhlbHAgaW50ZXJwcmV0IHRoZSBtb2RlbC4gW2Fuc10NCg0KMy4gTm9ib2R5IGVsc2UgdXNlcyByYW5kb20gZm9yZXN0cyB0byBwcmVkaWN0IGNodXJuLg0KDQoNCiMjIDUuNDogUmFuZG9tIGZvcmVzdCB3aXRoIGN1c3RvbSB0cmFpbkNvbnRyb2wNCg0KQW5vdGhlciBvbmUgb2YgbXkgZmF2b3JpdGUgbW9kZWxzIGlzIHRoZSByYW5kb20gZm9yZXN0LCB3aGljaCBjb21iaW5lcyBhbiBlbnNlbWJsZSBvZiBub24tbGluZWFyIGRlY2lzaW9uIHRyZWVzIGludG8gYSBoaWdobHkgZmxleGlibGUgKGFuZCB1c3VhbGx5IHF1aXRlIGFjY3VyYXRlKSBtb2RlbC4NCg0KUmF0aGVyIHRoYW4gdXNpbmcgdGhlIGNsYXNzaWMgcmFuZG9tRm9yZXN0IHBhY2thZ2UsIHlvdSdsbCBiZSB1c2luZyB0aGUgcmFuZ2VyIHBhY2thZ2UsIHdoaWNoIGlzIGEgcmUtaW1wbGVtZW50YXRpb24gb2YgcmFuZG9tRm9yZXN0IHRoYXQgcHJvZHVjZXMgYWxtb3N0IHRoZSBleGFjdCBzYW1lIHJlc3VsdHMsIGJ1dCBpcyBmYXN0ZXIsIG1vcmUgc3RhYmxlLCBhbmQgdXNlcyBsZXNzIG1lbW9yeS4gSSBoaWdobHkgcmVjb21tZW5kIGl0IGFzIGEgc3RhcnRpbmcgcG9pbnQgZm9yIHJhbmRvbSBmb3Jlc3QgbW9kZWxpbmcgaW4gUi4NCg0KSW5zdHJ1Y3Rpb25zDQoNCjEwMCBYUA0KDQotIGNodXJuX3ggYW5kIGNodXJuX3kgYXJlIGxvYWRlZCBpbiB5b3VyIHdvcmtzcGFjZS4NCg0KLSBGaXQgYSByYW5kb20gZm9yZXN0IG1vZGVsIHRvIHRoZSBjaHVybiBkYXRhc2V0LiBCZSBzdXJlIHRvIHVzZSBteUNvbnRyb2wgYXMgdGhlIHRyYWluQ29udHJvbCBsaWtlIHlvdSd2ZSBkb25lIGJlZm9yZSBhbmQgaW1wbGVtZW50IHRoZSAicmFuZ2VyIiBtZXRob2QuDQoNCmBgYHtyfQ0KIyBGaXQgcmFuZG9tIGZvcmVzdDogbW9kZWxfcmYNCm1vZGVsX3JmIDwtIHRyYWluKA0KICB4ID0gY2h1cm5feCwgeSA9IGNodXJuX3ksDQogIG1ldHJpYyA9ICJST0MiLA0KICBtZXRob2QgPSAicmFuZ2VyIiwNCiAgdHJDb250cm9sID0gbXlDb250cm9sDQopDQpgYGANCg0KIyMgNS41OiBDcmVhdGUgYSByZXNhbXBsZXMgb2JqZWN0DQoNCk5vdyB0aGF0IHlvdSBoYXZlIGZpdCB0d28gbW9kZWxzIHRvIHRoZSBjaHVybiBkYXRhc2V0LCBpdCdzIHRpbWUgdG8gY29tcGFyZSB0aGVpciBvdXQtb2Ytc2FtcGxlIHByZWRpY3Rpb25zIGFuZCBjaG9vc2Ugd2hpY2ggb25lIGlzIHRoZSBiZXN0IG1vZGVsIGZvciB5b3VyIGRhdGFzZXQuDQoNCllvdSBjYW4gY29tcGFyZSBtb2RlbHMgaW4gY2FyZXQgdXNpbmcgdGhlIHJlc2FtcGxlcygpIGZ1bmN0aW9uLCBwcm92aWRlZCB0aGV5IGhhdmUgdGhlIHNhbWUgdHJhaW5pbmcgZGF0YSBhbmQgdXNlIHRoZSBzYW1lIHRyYWluQ29udHJvbCBvYmplY3Qgd2l0aCBwcmVzZXQgY3Jvc3MtdmFsaWRhdGlvbiBmb2xkcy4gcmVzYW1wbGVzKCkgdGFrZXMgYXMgaW5wdXQgYSBsaXN0IG9mIG1vZGVscyBhbmQgY2FuIGJlIHVzZWQgdG8gY29tcGFyZSBkb3plbnMgb2YgbW9kZWxzIGF0IG9uY2UgKHRob3VnaCBpbiB0aGlzIGNhc2UgeW91IGFyZSBvbmx5IGNvbXBhcmluZyB0d28gbW9kZWxzKS4NCg0KSW5zdHJ1Y3Rpb25zDQoNCjEwMCBYUA0KDQotIG1vZGVsX2dsbW5ldCBhbmQgbW9kZWxfcmYgYXJlIGxvYWRlZCBpbiB5b3VyIHdvcmtzcGFjZS4NCg0KLSBDcmVhdGUgYSBsaXN0KCkgY29udGFpbmluZyB0aGUgZ2xtbmV0IG1vZGVsIGFzIGl0ZW0xIGFuZCB0aGUgcmFuZ2VyIG1vZGVsIGFzIGl0ZW0yLg0KDQotIFBhc3MgdGhpcyBsaXN0IHRvIHRoZSByZXNhbXBsZXMoKSBmdW5jdGlvbiBhbmQgc2F2ZSB0aGUgcmVzdWx0aW5nIG9iamVjdCBhcyByZXNhbXBsZXMuDQoNCi0gU3VtbWFyaXplIHRoZSByZXN1bHRzIGJ5IGNhbGxpbmcgc3VtbWFyeSgpIG9uIHJlc2FtcGxlcy4NCmBgYHtyfQ0KIyBDcmVhdGUgbW9kZWxfbGlzdA0KbW9kZWxfbGlzdCA8LSBsaXN0KGl0ZW0xID0gbW9kZWxfZ2xtbmV0LCBpdGVtMiA9IG1vZGVsX3JmKQ0KDQojIFBhc3MgbW9kZWxfbGlzdCB0byByZXNhbXBsZXMoKTogcmVzYW1wbGVzDQpyZXNhbXBsZXM8LXJlc2FtcGxlcyhtb2RlbF9saXN0KQ0KDQojIFN1bW1hcml6ZSB0aGUgcmVzdWx0cw0Kc3VtbWFyeShyZXNhbXBsZXMpDQpgYGANCg0KIyMgNS42OiBDcmVhdGUgYSBib3gtYW5kLXdoaXNrZXIgcGxvdA0KDQpjYXJldCBwcm92aWRlcyBhIHZhcmlldHkgb2YgbWV0aG9kcyB0byB1c2UgZm9yIGNvbXBhcmluZyBtb2RlbHMuIEFsbCBvZiB0aGVzZSBtZXRob2RzIGFyZSBiYXNlZCBvbiB0aGUgcmVzYW1wbGVzKCkgZnVuY3Rpb24uIE15IGZhdm9yaXRlIGlzIHRoZSBib3gtYW5kLXdoaXNrZXIgcGxvdCwgd2hpY2ggYWxsb3dzIHlvdSB0byBjb21wYXJlIHRoZSBkaXN0cmlidXRpb24gb2YgcHJlZGljdGl2ZSBhY2N1cmFjeSAoaW4gdGhpcyBjYXNlIEFVQykgZm9yIHRoZSB0d28gbW9kZWxzLg0KDQpJbiBnZW5lcmFsLCB5b3Ugd2FudCB0aGUgbW9kZWwgd2l0aCB0aGUgaGlnaGVyIG1lZGlhbiBBVUMsIGFzIHdlbGwgYXMgYSBzbWFsbGVyIHJhbmdlIGJldHdlZW4gbWluIGFuZCBtYXggQVVDLg0KDQpZb3UgY2FuIG1ha2UgdGhpcyBwbG90IHVzaW5nIHRoZSBid3Bsb3QoKSBmdW5jdGlvbiwgd2hpY2ggbWFrZXMgYSBib3ggYW5kIHdoaXNrZXIgcGxvdCBvZiB0aGUgbW9kZWwncyBvdXQgb2Ygc2FtcGxlIHNjb3Jlcy4gQm94IGFuZCB3aGlza2VyIHBsb3RzIHNob3cgdGhlIG1lZGlhbiBvZiBlYWNoIGRpc3RyaWJ1dGlvbiBhcyBhIGxpbmUgYW5kIHRoZSBpbnRlcnF1YXJ0aWxlIHJhbmdlIG9mIGVhY2ggZGlzdHJpYnV0aW9uIGFzIGEgYm94IGFyb3VuZCB0aGUgbWVkaWFuIGxpbmUuIFlvdSBjYW4gcGFzcyB0aGUgbWV0cmljID0gIlJPQyIgYXJndW1lbnQgdG8gdGhlIGJ3cGxvdCgpIGZ1bmN0aW9uIHRvIHNob3cgYSBwbG90IG9mIHRoZSBtb2RlbCdzIG91dC1vZi1zYW1wbGUgUk9DIHNjb3JlcyBhbmQgY2hvb3NlIHRoZSBtb2RlbCB3aXRoIHRoZSBoaWdoZXN0IG1lZGlhbiBST0MuDQoNCklmIHlvdSBkbyBub3Qgc3BlY2lmeSBhIG1ldHJpYyB0byBwbG90LCBid3Bsb3QoKSB3aWxsIGF1dG9tYXRpY2FsbHkgcGxvdCAzIG9mIHRoZW0uDQoNCkluc3RydWN0aW9ucw0KDQoxMDAgWFANCg0KLSBQYXNzIHRoZSByZXNhbXBsZXMgb2JqZWN0IHRvIHRoZSBid3Bsb3QoKSBmdW5jdGlvbiB0byBtYWtlIGEgYm94LWFuZC13aGlza2VyIHBsb3QuIExvb2sgYXQgdGhlIHJlc3VsdGluZyBwbG90IGFuZCBub3RlIHdoaWNoIG1vZGVsIGhhcyB0aGUgaGlnaGVyIG1lZGlhbiBST0Mgc3RhdGlzdGljLiBCZSBzdXJlIHRvIHNwZWNpZnkgd2hpY2ggbWV0cmljIHlvdSB3YW50IHRvIHBsb3QuDQoNCmBgYHtyfQ0KIyBDcmVhdGUgYndwbG90DQpid3Bsb3QocmVzYW1wbGVzLCBtZXRyaWMgPSAiUk9DIikNCmBgYA0KDQojIyA1Ljc6IENyZWF0ZSBhIHNjYXR0ZXJwbG90DQoNCkFub3RoZXIgdXNlZnVsIHBsb3QgZm9yIGNvbXBhcmluZyBtb2RlbHMgaXMgdGhlIHNjYXR0ZXJwbG90LCBhbHNvIGtub3duIGFzIHRoZSB4eS1wbG90LiBUaGlzIHBsb3Qgc2hvd3MgeW91IGhvdyBzaW1pbGFyIHRoZSB0d28gbW9kZWxzJyBwZXJmb3JtYW5jZXMgYXJlIG9uIGRpZmZlcmVudCBmb2xkcy4NCg0KSXQncyBwYXJ0aWN1bGFybHkgdXNlZnVsIGZvciBpZGVudGlmeWluZyBpZiBvbmUgbW9kZWwgaXMgY29uc2lzdGVudGx5IGJldHRlciB0aGFuIHRoZSBvdGhlciBhY3Jvc3MgYWxsIGZvbGRzLCBvciBpZiB0aGVyZSBhcmUgc2l0dWF0aW9ucyB3aGVuIHRoZSBpbmZlcmlvciBtb2RlbCBwcm9kdWNlcyBiZXR0ZXIgcHJlZGljdGlvbnMgb24gYSBwYXJ0aWN1bGFyIHN1YnNldCBvZiB0aGUgZGF0YS4NCg0KSW5zdHJ1Y3Rpb25zDQoNCjEwMCBYUA0KDQpQYXNzIHRoZSByZXNhbXBsZXMgb2JqZWN0IHRvIHRoZSB4eXBsb3QoKSBmdW5jdGlvbi4gTG9vayBhdCB0aGUgcmVzdWx0aW5nIHBsb3QgYW5kIG5vdGUgaG93IHNpbWlsYXIgdGhlIHR3byBtb2RlbHMnIHByZWRpY3Rpb25zIGFyZSAob3IgYXJlIG5vdCkgb24gdGhlIGRpZmZlcmVudCBmb2xkcy4gQmUgc3VyZSB0byBzcGVjaWZ5IHdoaWNoIG1ldHJpYyB5b3Ugd2FudCB0byBwbG90Lg0KYGBge3J9DQojIENyZWF0ZSB4eXBsb3QNCnh5cGxvdChyZXNhbXBsZXMsbWV0cmljPSJST0MiKQ0KYGBgDQoNCiMjIDUuODogRW5zZW1ibGluZyBtb2RlbHMNCg0KVGhhdCBjb25jbHVkZXMgdGhlIGNvdXJzZSEgQXMgYSB0ZWFzZXIgZm9yIGEgZnV0dXJlIGNvdXJzZSBvbiBtYWtpbmcgZW5zZW1ibGVzIG9mIGNhcmV0IG1vZGVscywgSSdsbCBzaG93IHlvdSBob3cgdG8gZml0IGEgc3RhY2tlZCBlbnNlbWJsZSBvZiBtb2RlbHMgdXNpbmcgdGhlIGNhcmV0RW5zZW1ibGUgcGFja2FnZS4NCg0KY2FyZXRFbnNlbWJsZSBwcm92aWRlcyB0aGUgY2FyZXRMaXN0KCkgZnVuY3Rpb24gZm9yIGNyZWF0aW5nIG11bHRpcGxlIGNhcmV0IG1vZGVscyBhdCBvbmNlIG9uIHRoZSBzYW1lIGRhdGFzZXQsIHVzaW5nIHRoZSBzYW1lIHJlc2FtcGxpbmcgZm9sZHMuIFlvdSBjYW4gYWxzbyBjcmVhdGUgeW91ciBvd24gbGlzdHMgb2YgY2FyZXQgbW9kZWxzLg0KDQpJbiB0aGlzIGV4ZXJjaXNlLCBJJ3ZlIG1hZGUgYSBjYXJldExpc3QgZm9yIHlvdSwgY29udGFpbmluZyB0aGUgZ2xtbmV0IGFuZCByYW5nZXIgbW9kZWxzIHlvdSBmaXQgb24gdGhlIGNodXJuIGRhdGFzZXQuIFVzZSB0aGUgY2FyZXRTdGFjaygpIGZ1bmN0aW9uIHRvIG1ha2UgYSBzdGFjayBvZiBjYXJldCBtb2RlbHMsIHdpdGggdGhlIHR3byBzdWItbW9kZWxzIChnbG1uZXQgYW5kIHJhbmdlcikgZmVlZGluZyBpbnRvIGFub3RoZXIgKGhvcGVmdWxseSBtb3JlIGFjY3VyYXRlISkgY2FyZXQgbW9kZWwuDQoNCkluc3RydWN0aW9ucw0KDQoxMDAgWFANCg0KLSBDYWxsIHRoZSBjYXJldFN0YWNrKCkgZnVuY3Rpb24gd2l0aCB0d28gYXJndW1lbnRzLCBtb2RlbF9saXN0IGFuZCBtZXRob2QgPSAiZ2xtIiwgdG8gZW5zZW1ibGUgdGhlIHR3byBtb2RlbHMgdXNpbmcgYSBsb2dpc3RpYyByZWdyZXNzaW9uLiBTdG9yZSB0aGUgcmVzdWx0IGFzIHN0YWNrLg0KDQotIFN1bW1hcml6ZSB0aGUgcmVzdWx0aW5nIG1vZGVsIHdpdGggdGhlIHN1bW1hcnkoKSBmdW5jdGlvbi4NCmBgYHtyfQ0Kc3RyKGNodXJuX3gpDQpgYGANCg0KYGBge3IsIGVjaG89VFJVRSxyZXN1bHRzPSdoaWRlJyxmaWcua2VlcD0nYWxsJ30NCiMgaW5zdGFsbC5wYWNrYWdlcygiY2FyZXRFbnNlbWJsZSIpDQojIGxpYnJhcnkoY2FyZXRFbnNlbWJsZSkNCiMgbW9kZWxfbGlzdCA8LSBjYXJldExpc3QoDQojICAgIH4gLiwgY2h1cm5feCwNCiMgICB0ckNvbnRyb2wgPSBteUNvbnRyb2wsDQojICAgbWV0aG9kTGlzdCA9IGMoImdsbSIsICJycGFydCIsICJyZiIsICJnYm0iLCAiZ2xtbmV0IikNCiMgICApDQojICMgQ3JlYXRlIGVuc2VtYmxlIG1vZGVsOiBzdGFjaw0KIyBzdGFjayA8LSBjYXJldFN0YWNrKG1vZGVsX2xpc3QsbWV0aG9kPSJnbG0iKQ0KIyANCiMgIyBMb29rIGF0IHN1bW1hcnkNCiMgc3VtbWFyeShzdGFjaykNCmBgYA0KDQoNCg==