Teoría

El paquete caret(Classification and regression training) es una herramienta poderosa para la implementación de modelos de Machine Learning.

Instalar paquetes y llamar librerías

# install.packages("caret")# Algoritmos de aprendizaje automatico
library(caret)
## Cargando paquete requerido: ggplot2
## Cargando paquete requerido: lattice
# install.packages("datasets")# Para usar la base de datos "iris"
library(datasets)
# install.packages("ggplot2")# Graficos con mejor diseño
library(ggplot2)
# install.packages("lattice")# Crear graficos
library(lattice)
# install.packages("DataExplorer")# Analisis Descriptivo
library(DataExplorer)
#install.packages("kernlab")
library(kernlab)
## 
## Adjuntando el paquete: 'kernlab'
## The following object is masked from 'package:ggplot2':
## 
##     alpha
#install.packages("rpart")
library(rpart)
#install.packages("randomForest")
library(randomForest)
## randomForest 4.7-1.2
## Type rfNews() to see new features/changes/bug fixes.
## 
## Adjuntando el paquete: 'randomForest'
## The following object is masked from 'package:ggplot2':
## 
##     margin

Importar la base de datos

df <- data.frame(iris)

Análisis descriptivo

# create_report(df)
plot_missing(df)

plot_histogram(df)

plot_correlation(df)

** NOTA: La variable que queremos predecir debe tener formato de FACTOR.**

Partir los datos 80-20

set.seed(123)
renglones_entrenamiento <- createDataPartition(df$Species, p=0.8, list=FALSE)
entrenamiento <- iris[renglones_entrenamiento, ]
prueba <- iris[-renglones_entrenamiento, ]

Distintos tipos de métodos para modelar

Los métodos más utilizados para modelar aprendizaje automático son:

  • SVM: Support Vector Machine o Maquina de Vectores de Soporte. Hay varios subtipos: Lineal (svmLinear), Radial (svmRadial), Polinómico (svmPoly), etc.
  • Árbol de Decisión: rpart
  • Redes Neuronales: nnet
  • Random Forest o Bosques Aleatorios: rf

La Validación cruzada (cross validation, cv) es una técnica para evaluar el rendimiento de un modelo, dividiendo los datos en múltiples subconjuntos, permitiendo medir su capacidad de generalización y evitar sobreajuste (overfitting).

La matriz de confusión permite analizar que tan bien funciona un modelo y que tipos de errores comete. Lo que hace es comparar las predicciones del con los valores reales de la variable objetivo.

Si la precision es es muy alta en entrenamiento (95-100%), pero baja en prueba (60-70%) es una señal de sobreajuste.

Modelo 1. SVM Linear

modelo1 <- train(Species ~ ., data = entrenamiento,
                 method = "svmLinear", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 tuneGrid = data.frame(C=1) #Cambiar hiperparametros
                 )

resultado_Entrenamiento1 <- predict(modelo1,entrenamiento)
resultado_prueba1 <- predict(modelo1, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre1 <- confusionMatrix(resultado_Entrenamiento1, entrenamiento$Species)
mcre1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp1 <- confusionMatrix(resultado_prueba1, prueba$Species)
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 2. SVM Radial

modelo2 <- train(Species ~ ., data = entrenamiento,
                 method = "svmRadial", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 tuneGrid = data.frame(sigma=1 ,C=1) #Cambiar hiperparametros
                 )

resultado_Entrenamiento2 <- predict(modelo2,entrenamiento)
resultado_prueba2 <- predict(modelo2, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre2 <- confusionMatrix(resultado_Entrenamiento1, entrenamiento$Species)
mcre2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp2 <- confusionMatrix(resultado_prueba2, prueba$Species)
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 3. SVM Polinómico

modelo3 <- train(Species ~ ., data = entrenamiento,
                 method = "svmPoly", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 tuneGrid = data.frame(degree =1, scale = 1, C=1) #Cambiar hiperparametros
                 )

resultado_Entrenamiento3 <- predict(modelo3,entrenamiento)
resultado_prueba3 <- predict(modelo3, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre3 <- confusionMatrix(resultado_Entrenamiento3, entrenamiento$Species)
mcre3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp3 <- confusionMatrix(resultado_prueba3, prueba$Species)
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 4. Árbol de decisión

modelo4 <- train(Species ~ ., data = entrenamiento,
                 method = "rpart", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 tuneLength = 10 #Cambiar hiperparametros
                 )

resultado_Entrenamiento4 <- predict(modelo4,entrenamiento)
resultado_prueba4 <- predict(modelo4, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre4 <- confusionMatrix(resultado_Entrenamiento4, entrenamiento$Species)
mcre4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
mcrp4 <- confusionMatrix(resultado_prueba4, prueba$Species)
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 5. Redes Neuronales

modelo5 <- train(Species ~ ., data = entrenamiento,
                 method = "nnet", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 trace = FALSE         #Cambiar hiperparametros
                 )

resultado_Entrenamiento5 <- predict(modelo5,entrenamiento)
resultado_prueba5 <- predict(modelo5, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre5 <- confusionMatrix(resultado_Entrenamiento5, entrenamiento$Species)
mcre5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
mcrp5 <- confusionMatrix(resultado_prueba5, prueba$Species)
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

Modelo 6. Bosques Aleatorios

modelo6 <- train(Species ~ ., data = entrenamiento,
                 method = "rf", # Cambiar 
                 preProcess=c("scale","center"),
                 trControl = trainControl(method="cv", number = 10),
                 tuneGrid = expand.grid(mtry = c(2,4,6)) #Cambiar hiperparametros
                 )
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
resultado_Entrenamiento6 <- predict(modelo6,entrenamiento)
resultado_prueba6 <- predict(modelo5, prueba)

# Matriz de Confusión del Resultado del Entrenamiento 
mcre6 <- confusionMatrix(resultado_Entrenamiento6, entrenamiento$Species)
mcre6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
mcrp6 <- confusionMatrix(resultado_prueba6, prueba$Species)
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

Resumen de Resultados

resultados <- data.frame(
  "SVM Lineal" = c(mcre1$overall["Accuracy"], mcrp1$overall["Accuracy"]),
  "SVM Radial"  = c(mcre2$overall["Accuracy"], mcrp2$overall["Accuracy"]),
  "SVM Polinómico"  = c(mcre3$overall["Accuracy"], mcrp3$overall["Accuracy"]),
  "Árbol de Decisión"  = c(mcre4$overall["Accuracy"], mcrp4$overall["Accuracy"]),
  "Redes Neuronales"  = c(mcre5$overall["Accuracy"], mcrp5$overall["Accuracy"]),
  "Bosques Aleatorios"  = c(mcre6$overall["Accuracy"], mcrp6$overall["Accuracy"])
)
rownames(resultados) <- c("Precisión de Entrenamiento", "Precisión de Prueba")
resultados
##                            SVM.Lineal SVM.Radial SVM.Polinómico
## Precisión de Entrenamiento  0.9916667  0.9916667      0.9916667
## Precisión de Prueba         0.9666667  0.9333333      0.9666667
##                            Árbol.de.Decisión Redes.Neuronales
## Precisión de Entrenamiento         0.9666667        0.9666667
## Precisión de Prueba                0.9333333        0.9666667
##                            Bosques.Aleatorios
## Precisión de Entrenamiento          1.0000000
## Precisión de Prueba                 0.9666667
LS0tDQp0aXRsZTogIkNBUkVUIg0KYXV0aG9yOiAiQWRyacOhbiBHYXJ6YSAtIEEwMTI4NTQxMiINCmRhdGU6ICIyMDI1LTAyLTIwIg0Kb3V0cHV0OiANCiAgaHRtbF9kb2N1bWVudDoNCiAgICB0b2M6IFRSVUUNCiAgICB0b2NfZmxvYXQ6IFRSVUUNCiAgICBjb2RlX2Rvd25sb2FkOiBUUlVFDQogICAgdGhlbWU6ICJ1bml0ZWQiDQogICAgaGlnaGxpZ2h0IDogImVzcHJlc3NvIg0KLS0tDQoNCiFbXShDOlxcVXNlcnNcXEJvb3RDYW1wXFxEb3dubG9hZHNcXGRlc2NhcmdhICgyKS5qcGVnKQ0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpwdXJwbGU7Ij5UZW9yw61hPC9zcGFuPg0KRWwgcGFxdWV0ZSBjYXJldCgqQ2xhc3NpZmljYXRpb24gYW5kIHJlZ3Jlc3Npb24gdHJhaW5pbmcqKSBlcyB1bmEgaGVycmFtaWVudGEgcG9kZXJvc2EgcGFyYSBsYSBpbXBsZW1lbnRhY2nDs24gZGUgbW9kZWxvcyBkZSAqKk1hY2hpbmUgTGVhcm5pbmcqKi4NCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6cHVycGxlOyI+SW5zdGFsYXIgcGFxdWV0ZXMgeSBsbGFtYXIgbGlicmVyw61hczwvc3Bhbj4NCmBgYHtyfQ0KIyBpbnN0YWxsLnBhY2thZ2VzKCJjYXJldCIpIyBBbGdvcml0bW9zIGRlIGFwcmVuZGl6YWplIGF1dG9tYXRpY28NCmxpYnJhcnkoY2FyZXQpDQojIGluc3RhbGwucGFja2FnZXMoImRhdGFzZXRzIikjIFBhcmEgdXNhciBsYSBiYXNlIGRlIGRhdG9zICJpcmlzIg0KbGlicmFyeShkYXRhc2V0cykNCiMgaW5zdGFsbC5wYWNrYWdlcygiZ2dwbG90MiIpIyBHcmFmaWNvcyBjb24gbWVqb3IgZGlzZcOxbw0KbGlicmFyeShnZ3Bsb3QyKQ0KIyBpbnN0YWxsLnBhY2thZ2VzKCJsYXR0aWNlIikjIENyZWFyIGdyYWZpY29zDQpsaWJyYXJ5KGxhdHRpY2UpDQojIGluc3RhbGwucGFja2FnZXMoIkRhdGFFeHBsb3JlciIpIyBBbmFsaXNpcyBEZXNjcmlwdGl2bw0KbGlicmFyeShEYXRhRXhwbG9yZXIpDQojaW5zdGFsbC5wYWNrYWdlcygia2VybmxhYiIpDQpsaWJyYXJ5KGtlcm5sYWIpDQojaW5zdGFsbC5wYWNrYWdlcygicnBhcnQiKQ0KbGlicmFyeShycGFydCkNCiNpbnN0YWxsLnBhY2thZ2VzKCJyYW5kb21Gb3Jlc3QiKQ0KbGlicmFyeShyYW5kb21Gb3Jlc3QpDQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6cHVycGxlOyI+SW1wb3J0YXIgbGEgYmFzZSBkZSBkYXRvczwvc3Bhbj4NCmBgYHtyfQ0KZGYgPC0gZGF0YS5mcmFtZShpcmlzKQ0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnB1cnBsZTsiPkFuw6FsaXNpcyBkZXNjcmlwdGl2bzwvc3Bhbj4NCmBgYHtyfQ0KIyBjcmVhdGVfcmVwb3J0KGRmKQ0KcGxvdF9taXNzaW5nKGRmKQ0KcGxvdF9oaXN0b2dyYW0oZGYpDQpwbG90X2NvcnJlbGF0aW9uKGRmKQ0KYGBgDQoqKiBOT1RBOiBMYSB2YXJpYWJsZSBxdWUgcXVlcmVtb3MgcHJlZGVjaXIgZGViZSB0ZW5lciBmb3JtYXRvIGRlIEZBQ1RPUi4qKg0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpwdXJwbGU7Ij5QYXJ0aXIgbG9zIGRhdG9zIDgwLTIwPC9zcGFuPg0KYGBge3J9DQpzZXQuc2VlZCgxMjMpDQpyZW5nbG9uZXNfZW50cmVuYW1pZW50byA8LSBjcmVhdGVEYXRhUGFydGl0aW9uKGRmJFNwZWNpZXMsIHA9MC44LCBsaXN0PUZBTFNFKQ0KZW50cmVuYW1pZW50byA8LSBpcmlzW3Jlbmdsb25lc19lbnRyZW5hbWllbnRvLCBdDQpwcnVlYmEgPC0gaXJpc1stcmVuZ2xvbmVzX2VudHJlbmFtaWVudG8sIF0NCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpwdXJwbGU7Ij5EaXN0aW50b3MgdGlwb3MgZGUgbcOpdG9kb3MgcGFyYSBtb2RlbGFyPC9zcGFuPg0KTG9zIG3DqXRvZG9zIG3DoXMgdXRpbGl6YWRvcyBwYXJhIG1vZGVsYXIgYXByZW5kaXphamUgYXV0b23DoXRpY28gc29uOiAgDQoNCiogKipTVk0qKjogKlN1cHBvcnQgVmVjdG9yIE1hY2hpbmUqIG8gTWFxdWluYSBkZSBWZWN0b3JlcyBkZSBTb3BvcnRlLiBIYXkgdmFyaW9zIHN1YnRpcG9zOiBMaW5lYWwgKHN2bUxpbmVhciksIFJhZGlhbCAoc3ZtUmFkaWFsKSwgUG9saW7Ds21pY28gKHN2bVBvbHkpLCBldGMuIA0KKiAqKsOBcmJvbCBkZSBEZWNpc2nDs24qKjogcnBhcnQNCiogKipSZWRlcyBOZXVyb25hbGVzKio6IG5uZXQNCiogKipSYW5kb20gRm9yZXN0KiogbyBCb3NxdWVzIEFsZWF0b3Jpb3M6IHJmDQoNCkxhICoqVmFsaWRhY2nDs24gY3J1emFkYSoqIChjcm9zcyB2YWxpZGF0aW9uLCBjdikgZXMgdW5hIHTDqWNuaWNhIHBhcmEgZXZhbHVhciBlbCByZW5kaW1pZW50byBkZSB1biBtb2RlbG8sIGRpdmlkaWVuZG8gbG9zIGRhdG9zIGVuIG3Dumx0aXBsZXMgc3ViY29uanVudG9zLCBwZXJtaXRpZW5kbyBtZWRpciBzdSBjYXBhY2lkYWQgZGUgZ2VuZXJhbGl6YWNpw7NuIHkgZXZpdGFyIHNvYnJlYWp1c3RlICgqb3ZlcmZpdHRpbmcqKS4NCg0KTGEgKiptYXRyaXogZGUgY29uZnVzacOzbioqIHBlcm1pdGUgYW5hbGl6YXIgcXVlIHRhbiBiaWVuIGZ1bmNpb25hIHVuIG1vZGVsbyB5IHF1ZSB0aXBvcyBkZSBlcnJvcmVzIGNvbWV0ZS4gTG8gcXVlIGhhY2UgZXMgY29tcGFyYXIgbGFzIHByZWRpY2Npb25lcyBkZWwgY29uIGxvcyB2YWxvcmVzIHJlYWxlcyBkZSBsYSB2YXJpYWJsZSBvYmpldGl2by4gDQoNClNpIGxhIHByZWNpc2lvbiBlcyBlcyBtdXkgYWx0YSBlbiBlbnRyZW5hbWllbnRvICg5NS0xMDAlKSwNCnBlcm8gYmFqYSBlbiBwcnVlYmEgKDYwLTcwJSkgZXMgdW5hIHNlw7FhbCBkZSBzb2JyZWFqdXN0ZS4NCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6cHVycGxlOyI+TW9kZWxvIDEuIFNWTSBMaW5lYXI8L3NwYW4+DQpgYGB7cn0NCm1vZGVsbzEgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGEgPSBlbnRyZW5hbWllbnRvLA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAic3ZtTGluZWFyIiwgIyBDYW1iaWFyIA0KICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwiY2VudGVyIiksDQogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyID0gMTApLA0KICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoQz0xKSAjQ2FtYmlhciBoaXBlcnBhcmFtZXRyb3MNCiAgICAgICAgICAgICAgICAgKQ0KDQpyZXN1bHRhZG9fRW50cmVuYW1pZW50bzEgPC0gcHJlZGljdChtb2RlbG8xLGVudHJlbmFtaWVudG8pDQpyZXN1bHRhZG9fcHJ1ZWJhMSA8LSBwcmVkaWN0KG1vZGVsbzEsIHBydWViYSkNCg0KIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvIA0KbWNyZTEgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19FbnRyZW5hbWllbnRvMSwgZW50cmVuYW1pZW50byRTcGVjaWVzKQ0KbWNyZTENCg0KbWNycDEgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmExLCBwcnVlYmEkU3BlY2llcykNCm1jcnAxDQoNCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpwdXJwbGU7Ij5Nb2RlbG8gMi4gU1ZNIFJhZGlhbDwvc3Bhbj4NCmBgYHtyfQ0KbW9kZWxvMiA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YSA9IGVudHJlbmFtaWVudG8sDQogICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJzdm1SYWRpYWwiLCAjIENhbWJpYXIgDQogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCJjZW50ZXIiKSwNCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXIgPSAxMCksDQogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZShzaWdtYT0xICxDPTEpICNDYW1iaWFyIGhpcGVycGFyYW1ldHJvcw0KICAgICAgICAgICAgICAgICApDQoNCnJlc3VsdGFkb19FbnRyZW5hbWllbnRvMiA8LSBwcmVkaWN0KG1vZGVsbzIsZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmEyIDwtIHByZWRpY3QobW9kZWxvMiwgcHJ1ZWJhKQ0KDQojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGVsIEVudHJlbmFtaWVudG8gDQptY3JlMiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX0VudHJlbmFtaWVudG8xLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlMg0KDQptY3JwMiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTIsIHBydWViYSRTcGVjaWVzKQ0KbWNycDINCg0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnB1cnBsZTsiPk1vZGVsbyAzLiBTVk0gUG9saW7Ds21pY288L3NwYW4+DQpgYGB7cn0NCm1vZGVsbzMgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGEgPSBlbnRyZW5hbWllbnRvLA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAic3ZtUG9seSIsICMgQ2FtYmlhciANCiAgICAgICAgICAgICAgICAgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsImNlbnRlciIpLA0KICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlciA9IDEwKSwNCiAgICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKGRlZ3JlZSA9MSwgc2NhbGUgPSAxLCBDPTEpICNDYW1iaWFyIGhpcGVycGFyYW1ldHJvcw0KICAgICAgICAgICAgICAgICApDQoNCnJlc3VsdGFkb19FbnRyZW5hbWllbnRvMyA8LSBwcmVkaWN0KG1vZGVsbzMsZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmEzIDwtIHByZWRpY3QobW9kZWxvMywgcHJ1ZWJhKQ0KDQojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGVsIEVudHJlbmFtaWVudG8gDQptY3JlMyA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX0VudHJlbmFtaWVudG8zLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlMw0KDQptY3JwMyA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTMsIHBydWViYSRTcGVjaWVzKQ0KbWNycDMNCg0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnB1cnBsZTsiPk1vZGVsbyA0LiDDgXJib2wgZGUgZGVjaXNpw7NuPC9zcGFuPg0KYGBge3J9DQptb2RlbG80IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhID0gZW50cmVuYW1pZW50bywNCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInJwYXJ0IiwgIyBDYW1iaWFyIA0KICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwiY2VudGVyIiksDQogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyID0gMTApLA0KICAgICAgICAgICAgICAgICB0dW5lTGVuZ3RoID0gMTAgI0NhbWJpYXIgaGlwZXJwYXJhbWV0cm9zDQogICAgICAgICAgICAgICAgICkNCg0KcmVzdWx0YWRvX0VudHJlbmFtaWVudG80IDwtIHByZWRpY3QobW9kZWxvNCxlbnRyZW5hbWllbnRvKQ0KcmVzdWx0YWRvX3BydWViYTQgPC0gcHJlZGljdChtb2RlbG80LCBwcnVlYmEpDQoNCiMgTWF0cml6IGRlIENvbmZ1c2nDs24gZGVsIFJlc3VsdGFkbyBkZWwgRW50cmVuYW1pZW50byANCm1jcmU0IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fRW50cmVuYW1pZW50bzQsIGVudHJlbmFtaWVudG8kU3BlY2llcykNCm1jcmU0DQoNCm1jcnA0IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhNCwgcHJ1ZWJhJFNwZWNpZXMpDQptY3JwNA0KDQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6cHVycGxlOyI+TW9kZWxvIDUuIFJlZGVzIE5ldXJvbmFsZXM8L3NwYW4+DQpgYGB7cn0NCm1vZGVsbzUgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGEgPSBlbnRyZW5hbWllbnRvLA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAibm5ldCIsICMgQ2FtYmlhciANCiAgICAgICAgICAgICAgICAgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsImNlbnRlciIpLA0KICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlciA9IDEwKSwNCiAgICAgICAgICAgICAgICAgdHJhY2UgPSBGQUxTRSAgICAgICAgICNDYW1iaWFyIGhpcGVycGFyYW1ldHJvcw0KICAgICAgICAgICAgICAgICApDQoNCnJlc3VsdGFkb19FbnRyZW5hbWllbnRvNSA8LSBwcmVkaWN0KG1vZGVsbzUsZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmE1IDwtIHByZWRpY3QobW9kZWxvNSwgcHJ1ZWJhKQ0KDQojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGVsIEVudHJlbmFtaWVudG8gDQptY3JlNSA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX0VudHJlbmFtaWVudG81LCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlNQ0KDQptY3JwNSA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTUsIHBydWViYSRTcGVjaWVzKQ0KbWNycDUNCg0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnB1cnBsZTsiPk1vZGVsbyA2LiBCb3NxdWVzIEFsZWF0b3Jpb3M8L3NwYW4+DQpgYGB7cn0NCm1vZGVsbzYgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGEgPSBlbnRyZW5hbWllbnRvLA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmYiLCAjIENhbWJpYXIgDQogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCJjZW50ZXIiKSwNCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXIgPSAxMCksDQogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZXhwYW5kLmdyaWQobXRyeSA9IGMoMiw0LDYpKSAjQ2FtYmlhciBoaXBlcnBhcmFtZXRyb3MNCiAgICAgICAgICAgICAgICAgKQ0KDQpyZXN1bHRhZG9fRW50cmVuYW1pZW50bzYgPC0gcHJlZGljdChtb2RlbG82LGVudHJlbmFtaWVudG8pDQpyZXN1bHRhZG9fcHJ1ZWJhNiA8LSBwcmVkaWN0KG1vZGVsbzUsIHBydWViYSkNCg0KIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvIA0KbWNyZTYgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19FbnRyZW5hbWllbnRvNiwgZW50cmVuYW1pZW50byRTcGVjaWVzKQ0KbWNyZTYNCg0KbWNycDYgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmE2LCBwcnVlYmEkU3BlY2llcykNCm1jcnA2DQoNCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpwdXJwbGU7Ij5SZXN1bWVuIGRlIFJlc3VsdGFkb3M8L3NwYW4+DQpgYGB7cn0NCnJlc3VsdGFkb3MgPC0gZGF0YS5mcmFtZSgNCiAgIlNWTSBMaW5lYWwiID0gYyhtY3JlMSRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwMSRvdmVyYWxsWyJBY2N1cmFjeSJdKSwNCiAgIlNWTSBSYWRpYWwiICA9IGMobWNyZTIkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDIkb3ZlcmFsbFsiQWNjdXJhY3kiXSksDQogICJTVk0gUG9saW7Ds21pY28iICA9IGMobWNyZTMkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDMkb3ZlcmFsbFsiQWNjdXJhY3kiXSksDQogICLDgXJib2wgZGUgRGVjaXNpw7NuIiAgPSBjKG1jcmU0JG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA0JG92ZXJhbGxbIkFjY3VyYWN5Il0pLA0KICAiUmVkZXMgTmV1cm9uYWxlcyIgID0gYyhtY3JlNSRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNSRvdmVyYWxsWyJBY2N1cmFjeSJdKSwNCiAgIkJvc3F1ZXMgQWxlYXRvcmlvcyIgID0gYyhtY3JlNiRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNiRvdmVyYWxsWyJBY2N1cmFjeSJdKQ0KKQ0Kcm93bmFtZXMocmVzdWx0YWRvcykgPC0gYygiUHJlY2lzacOzbiBkZSBFbnRyZW5hbWllbnRvIiwgIlByZWNpc2nDs24gZGUgUHJ1ZWJhIikNCnJlc3VsdGFkb3MNCmBgYA0KDQoNCg==