El paquete caret

El paquete caret es un framework que unifica diferentes metodos para crear modelos predictions. El manual se encuentra disponible en github page.

set.seed(1)
library(caret)
library(readr)
library(tidyr)

Titanic Data

raw_data <- read.csv("train.csv", stringsAsFactors=FALSE)

Una mirada al dataset

raw_data

Modelo de Machine Learning

Nuestro archivo train.csv es el unico archivo que contiene observaciones, por lo que lo vamos a utilizar para entrenar y evaluar nuestro modelo de machine learning. Primero separamos el archivo train.csv en dos utilizando la funcion createDataPartition. A la función se le indica como parametro la clase (Survive en este caso) y el por centaje en el que vamos a separar el dataset. En ete caso vamos a separarlo en un 80/20. Es decir 80% del dataset sera para entrenar y un 20% para testear.

Ajustando el modelo caret

Para ajustar los parametros de los modelo se utilizan diferentes tecnicas que utilizaran train_data. Normalmente se utilizan tecnicas de resampling que lo que hacen es partir de diferente manera los datos para luego entrenar y validar los resultados. Todo este proceso caret lo hace automaticamente a traves de trainControl. En este caso le indicamos que vamos a aplicar un metodo de validacion Cruzada (CrossValidation) de 2x4. Lo que significa que a data_train lo vamos a separar en 4 partes y vamos a usar 3 partes para entrenar un modelo y la restante para validarlo.

train_control <- trainControl(method="cv", number=5)

Creacion de un Modelo

El campo Survived tiene los valores [0,1], los cuales pueden incorrectamente considerarse como numeros cuando en realidad se trata de categorias. Para esto resulta necesario convertirlos a factores.

Regresion lineal

La funcion train entrena un modelo. El primer parametro Survived~. es lo que se conoce como sintaxis de formula en R. Y basicamente se indica a la funcion que vamos a utilizar Survived como la clase a predecir y el . indica que utilicemos TODOS los atributos(columnas) durante el entrenamiento.

    glm_model1 <- train(Survived ~ ., data = train_data,  method = "glm", family = "binomial",trainControl=train_control)
Error in na.fail.default(list(Survived = c(0L, 1L, 1L, 1L, 0L, 0L, 0L,  : 
  missing values in object

El modelo deberia dar un error, ya que existen ciertos registros que no tiene valores. (NA). Esto podemos verifcarlo:

sapply(raw_data, anyNA)
PassengerId    Survived      Pclass        Name         Sex         Age       SibSp       Parch      Ticket        Fare       Cabin 
      FALSE       FALSE       FALSE       FALSE       FALSE        TRUE       FALSE       FALSE       FALSE       FALSE       FALSE 
   Embarked 
      FALSE

Se verifica que los campos Age, Cabin y Embarked contienen datos incompletos. Podemos intentar solucionar esto, pero lo veremos mas adelante. Por ahora simplemente eliminemoslos.

train_data_removed<-train_data %>% select(-Age,-Cabin,-Embarked)
 glm_model1 <- train(Survived ~ ., data = train_data_removed,  method = "glm", family = "binomial",trainControl=train_control)
You are trying to do regression and your outcome only has two possible values Are you trying to do classification? If so, use a 2 level factor as your outcome column.model fit failed for Resample01: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample02: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample03: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample04: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample05: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample06: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample07: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample08: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample09: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample10: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample11: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample12: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample13: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample14: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample15: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample16: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample17: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample18: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample19: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample20: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample21: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample22: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample23: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample24: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
model fit failed for Resample25: parameter=none Error in glm.control(trainControl = list(method = "cv", number = 5, repeats = NA,  : 
  unused argument (trainControl = list("cv", 5, NA, "grid", 0.75, NULL, 1, TRUE, 0, FALSE, TRUE, "final", FALSE, FALSE, function (data, lev = NULL, model = NULL) 
{
    if (is.character(data$obs)) data$obs <- factor(data$obs, levels = lev)
    postResample(data[, "pred"], data[, "obs"])
}, "best", list(0.95, 3, 5, 19, 10, 0.9), NULL, NULL, NULL, NULL, 0, c(FALSE, FALSE), NA, list(5, 0.05, "gls", TRUE), FALSE, TRUE))
There were missing values in resampled performance measures.
Something is wrong; all the RMSE metric values are missing:
      RMSE        Rsquared        MAE     
 Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :1     NA's   :1     NA's   :1    
Error: Stopping

La función train() vuelve a dar un error ya que existen atributos(columnas) no numericas como entrada y claramente resulta imposible realizar una regresion. Ejemplo de esto son el campo Name y Ticket. Esto puede solucionarse tambien, pero por ahora vamos a simplemente eliminarlo.

train_data_removed<-train_data %>% select(-Name,-Ticket,-Age,-Cabin,-Embarked,-Sex)
glm_model1 <- train(formula, data = train_data_removed,  method = "glm", family=binomial, trControl=train_control)
 
 
PredTrain = predict(glm_model1, newdata=train, type="raw")
table(train$Survived, PredTrain > 0.5)

La funcion confusionMatrix() del paquete caret permite calcular la matriz de confusion junto a otras métricas

confusionMatrix(as.factor(train$Survived), as.factor(ifelse(PredTrain > 0.5,1,0)))
LS0tCnRpdGxlOiAiTWFjaGluZSBMZWFybmluZyB3aXRoIHRoZSBDYXJldCBwYWNrYWdlIGFuZCB0aGUgVGl0YW5pYyIKYXV0aG9yOiAiSGFycG8gTUF4eCIKZGF0ZTogIkRlY2VtYmVyIDEwLCAyMDE3IgpvdXRwdXQ6IAogIGh0bWxfbm90ZWJvb2sKICAgIAotLS0KYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSkKYGBgCgojIEVsIHBhcXVldGUgY2FyZXQKRWwgcGFxdWV0ZSBjYXJldCBlcyB1biBmcmFtZXdvcmsgcXVlIHVuaWZpY2EgZGlmZXJlbnRlcyBtZXRvZG9zIHBhcmEgY3JlYXIgbW9kZWxvcyBwcmVkaWN0aW9ucy4KRWwgbWFudWFsIHNlIGVuY3VlbnRyYSBkaXNwb25pYmxlIGVuICBbZ2l0aHViIHBhZ2VdKGh0dHA6Ly90b3BlcG8uZ2l0aHViLmlvL2NhcmV0L2luZGV4Lmh0bWwpLgoKCmBgYHtyIG1lc3NhZ2UgPSBGQUxTRSwgd2FybmluZyA9IEZBTFNFfQpzZXQuc2VlZCgxKQpsaWJyYXJ5KGNhcmV0KQpsaWJyYXJ5KHJlYWRyKQpsaWJyYXJ5KHRpZHlyKQpgYGAKCiMgVGl0YW5pYyBEYXRhCmBgYHtyfQpyYXdfZGF0YSA8LSByZWFkLmNzdigidHJhaW4uY3N2Iiwgc3RyaW5nc0FzRmFjdG9ycz1GQUxTRSkKYGBgCgojIyBVbmEgbWlyYWRhIGFsIGRhdGFzZXQKCmBgYHtyfQpyYXdfZGF0YQpgYGAKCiMjIE1vZGVsbyBkZSBNYWNoaW5lIExlYXJuaW5nCk51ZXN0cm8gYXJjaGl2byB0cmFpbi5jc3YgZXMgZWwgdW5pY28gYXJjaGl2byBxdWUgY29udGllbmUgb2JzZXJ2YWNpb25lcywgcG9yIGxvIHF1ZSBsbyB2YW1vcyBhIHV0aWxpemFyIHBhcmEgZW50cmVuYXIgeSBldmFsdWFyIG51ZXN0cm8gbW9kZWxvIGRlIG1hY2hpbmUgbGVhcm5pbmcuClByaW1lcm8gc2VwYXJhbW9zIGVsIGFyY2hpdm8gYHRyYWluLmNzdmAgZW4gZG9zIHV0aWxpemFuZG8gbGEgZnVuY2lvbiBjcmVhdGVEYXRhUGFydGl0aW9uLiBBIGxhIGZ1bmNpw7NuIHNlIGxlIGluZGljYSBjb21vIHBhcmFtZXRybyBsYSBjbGFzZSAoU3Vydml2ZSBlbiBlc3RlIGNhc28pIHkgZWwgcG9yIGNlbnRhamUgZW4gZWwgcXVlIHZhbW9zIGEgc2VwYXJhciBlbCBkYXRhc2V0LiBFbiBldGUgY2FzbyB2YW1vcyBhIHNlcGFyYXJsbyBlbiB1biA4MC8yMC4gRXMgZGVjaXIgODAlIGRlbCBkYXRhc2V0IHNlcmEgcGFyYSBlbnRyZW5hciB5IHVuIDIwJSBwYXJhIHRlc3RlYXIuCmBgYHtyfQp0cmFpbl9kYXRhX2luZCA8LSBjcmVhdGVEYXRhUGFydGl0aW9uKHJhd19kYXRhJFN1cnZpdmVkLCBwID0gMC43LCBsaXN0ID0gRkFMU0UpCnRyYWluX2RhdGEgPC0gcmF3X2RhdGFbdHJhaW5fZGF0YV9pbmQsXQp0ZXN0X2RhdGEgPC0gcmF3X2RhdGFbLXRyYWluX2RhdGFfaW5kLF0KYGBgCgojIyNBanVzdGFuZG8gZWwgbW9kZWxvIGNhcmV0ClBhcmEgYWp1c3RhciBsb3MgcGFyYW1ldHJvcyBkZSBsb3MgbW9kZWxvIHNlIHV0aWxpemFuIGRpZmVyZW50ZXMgdGVjbmljYXMgcXVlIHV0aWxpemFyYW4gYHRyYWluX2RhdGFgLiBOb3JtYWxtZW50ZSBzZSB1dGlsaXphbiB0ZWNuaWNhcyBkZSByZXNhbXBsaW5nIHF1ZSBsbyBxdWUgaGFjZW4gZXMgcGFydGlyIGRlIGRpZmVyZW50ZSBtYW5lcmEgbG9zIGRhdG9zIHBhcmEgbHVlZ28gZW50cmVuYXIgeSB2YWxpZGFyIGxvcyByZXN1bHRhZG9zLiBUb2RvIGVzdGUgcHJvY2VzbyBjYXJldCBsbyBoYWNlIGF1dG9tYXRpY2FtZW50ZSBhIHRyYXZlcyBkZSBgdHJhaW5Db250cm9sYC4gRW4gZXN0ZSBjYXNvIGxlIGluZGljYW1vcyBxdWUgdmFtb3MgYSBhcGxpY2FyIHVuIG1ldG9kbyBkZSB2YWxpZGFjaW9uIENydXphZGEgKENyb3NzVmFsaWRhdGlvbikgZGUgMng0LiBMbyBxdWUgc2lnbmlmaWNhIHF1ZSBhIGBkYXRhX3RyYWluYCBsbyB2YW1vcyBhIHNlcGFyYXIgZW4gNCBwYXJ0ZXMgeSB2YW1vcyBhIHVzYXIgMyBwYXJ0ZXMgcGFyYSBlbnRyZW5hciB1biBtb2RlbG8geSBsYSByZXN0YW50ZSBwYXJhIHZhbGlkYXJsby4gIApgYGB7cn0KdHJhaW5fY29udHJvbCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlcj01KQoKYGBgCgoKIyMjIENyZWFjaW9uIGRlICB1biBNb2RlbG8KRWwgY2FtcG8gU3Vydml2ZWQgdGllbmUgbG9zIHZhbG9yZXMgWzAsMV0sIGxvcyBjdWFsZXMgcHVlZGVuIGluY29ycmVjdGFtZW50ZSBjb25zaWRlcmFyc2UgY29tbyBudW1lcm9zIGN1YW5kbyBlbiByZWFsaWRhZCBzZSB0cmF0YSBkZSBjYXRlZ29yaWFzLiBQYXJhIGVzdG8gcmVzdWx0YSBuZWNlc2FyaW8gY29udmVydGlybG9zIGEgZmFjdG9yZXMuCmBgYHtyIGluY2x1ZGUgPSBGQUxTRX0KdHJhaW5fZGF0YSRTdXJ2aXZlZCA8LSBhcy5mYWN0b3IodHJhaW5fZGF0YSRTdXJ2aXZlZCkKdGVzdF9kYXRhJFN1cnZpdmVkIDwtIGFzLmZhY3Rvcih0ZXN0X2RhdGEkU3Vydml2ZWQpCgoKYGBgCgojIyMjIFJlZ3Jlc2lvbiBsaW5lYWwKTGEgZnVuY2lvbiB0cmFpbiBlbnRyZW5hIHVuIG1vZGVsby4gRWwgcHJpbWVyIHBhcmFtZXRybyBgU3Vydml2ZWR+LmAgZXMgbG8gcXVlIHNlIGNvbm9jZSBjb21vIHNpbnRheGlzIGRlIGZvcm11bGEgZW4gUi4gWSBiYXNpY2FtZW50ZSBzZSBpbmRpY2EgYSBsYSBmdW5jaW9uIHF1ZSB2YW1vcyBhIHV0aWxpemFyIFN1cnZpdmVkIGNvbW8gbGEgY2xhc2UgYSBwcmVkZWNpciB5IGVsIGAuYCBpbmRpY2EgcXVlIHV0aWxpY2Vtb3MgKlRPRE9TKiBsb3MgYXRyaWJ1dG9zKGNvbHVtbmFzKSBkdXJhbnRlIGVsIGVudHJlbmFtaWVudG8uCmBgYHtyfQogICAgZ2xtX21vZGVsMSA8LSB0cmFpbihTdXJ2aXZlZCB+IC4sIGRhdGEgPSB0cmFpbl9kYXRhLCAgbWV0aG9kID0gImdsbSIsIGZhbWlseSA9ICJiaW5vbWlhbCIsdHJhaW5Db250cm9sPXRyYWluX2NvbnRyb2wpCmBgYAoKRWwgbW9kZWxvIGRlYmVyaWEgZGFyIHVuIGVycm9yLCB5YSBxdWUgZXhpc3RlbiBjaWVydG9zIHJlZ2lzdHJvcyBxdWUgbm8gdGllbmUgdmFsb3Jlcy4gKE5BKS4gRXN0byBwb2RlbW9zIHZlcmlmY2FybG86CgpgYGB7cn0Kc2FwcGx5KHJhd19kYXRhLCBhbnlOQSkKYGBgCgpTZSB2ZXJpZmljYSBxdWUgbG9zIGNhbXBvcyBgQWdlYCwgYENhYmluYCB5IGBFbWJhcmtlZGAgY29udGllbmVuIGRhdG9zIGluY29tcGxldG9zLiBQb2RlbW9zIGludGVudGFyIHNvbHVjaW9uYXIgZXN0bywgcGVybyBsbyB2ZXJlbW9zIG1hcyBhZGVsYW50ZS4gUG9yIGFob3JhIHNpbXBsZW1lbnRlIGVsaW1pbmVtb3Nsb3MuCmBgYHtyfQp0cmFpbl9kYXRhX3JlbW92ZWQ8LXRyYWluX2RhdGEgJT4lIHNlbGVjdCgtQWdlLC1DYWJpbiwtRW1iYXJrZWQpCiBnbG1fbW9kZWwxIDwtIHRyYWluKFN1cnZpdmVkIH4gLiwgZGF0YSA9IHRyYWluX2RhdGFfcmVtb3ZlZCwgIG1ldGhvZCA9ICJnbG0iLCBmYW1pbHkgPSAiYmlub21pYWwiLHRyYWluQ29udHJvbD10cmFpbl9jb250cm9sKQpgYGAKCkxhIGZ1bmNpw7NuIGB0cmFpbigpYCB2dWVsdmUgYSBkYXIgdW4gZXJyb3IgeWEgcXVlIGV4aXN0ZW4gYXRyaWJ1dG9zKGNvbHVtbmFzKSBubyBudW1lcmljYXMgY29tbyBlbnRyYWRhIHkgY2xhcmFtZW50ZSByZXN1bHRhIGltcG9zaWJsZSByZWFsaXphciB1bmEgcmVncmVzaW9uLiBFamVtcGxvIGRlIGVzdG8gc29uIGVsIGNhbXBvIGBOYW1lYCB5IGBUaWNrZXRgLiBFc3RvIHB1ZWRlIHNvbHVjaW9uYXJzZSB0YW1iaWVuLCBwZXJvIHBvciBhaG9yYSB2YW1vcyBhIHNpbXBsZW1lbnRlIGVsaW1pbmFybG8uCgpgYGB7cn0KdHJhaW5fZGF0YV9yZW1vdmVkPC10cmFpbl9kYXRhICU+JSBzZWxlY3QoLU5hbWUsLVRpY2tldCwtQWdlLC1DYWJpbiwtRW1iYXJrZWQsLVNleCkKZ2xtX21vZGVsMSA8LSB0cmFpbihmb3JtdWxhLCBkYXRhID0gdHJhaW5fZGF0YV9yZW1vdmVkLCAgbWV0aG9kID0gImdsbSIsIGZhbWlseT1iaW5vbWlhbCwgdHJDb250cm9sPXRyYWluX2NvbnRyb2wpCiAKIApQcmVkVHJhaW4gPSBwcmVkaWN0KGdsbV9tb2RlbDEsIG5ld2RhdGE9dHJhaW4sIHR5cGU9InJhdyIpCnRhYmxlKHRyYWluJFN1cnZpdmVkLCBQcmVkVHJhaW4gPiAwLjUpCmBgYAoKTGEgZnVuY2lvbiBgY29uZnVzaW9uTWF0cml4KClgIGRlbCBwYXF1ZXRlIGBjYXJldGAgcGVybWl0ZSBjYWxjdWxhciBsYSBtYXRyaXogZGUgY29uZnVzaW9uIGp1bnRvIGEgb3RyYXMgbcOpdHJpY2FzIAoKYGBge3J9CmNvbmZ1c2lvbk1hdHJpeChhcy5mYWN0b3IodHJhaW4kU3Vydml2ZWQpLCBhcy5mYWN0b3IoaWZlbHNlKFByZWRUcmFpbiA+IDAuNSwxLDApKSkKYGBgCg==