Teoría

El paquete caret(Classification and Regression Training) es una herramienta poderosa para la implementación de modelos de Machine Learning.

Instalar paquetes y llamar librerias

#install.packages("caret") # Algoritmos de aprendizaje automático
library(caret)
#install.packages("datasets") # Para usar la base de datos "Iris"
library(datasets)
#install.packages("ggplot2") # Gráficas con mejor diseño
library(ggplot2)
#install.packages("lattice") # Crear gráficos
library(lattice)
#install.packages("DataExplorer") # Análisis Descriptivo
library(DataExplorer)
#installed.packages("kernlab")
library(kernlab)

Importar la base de datos

df <- data.frame(iris)

Análisis Descriptivo

# create_report(df)

plot_missing(df)

plot_histogram(df)

plot_correlation(df)

** NOTA: La variable que queremos predecir debe tener formato de FACTOR.**

Partir los datos 80-20

set.seed(123)
renglones_entrenamiento <- createDataPartition(df$Species, p = 0.8, list = FALSE)
entrenamiento <- iris[renglones_entrenamiento, ]
prueba <- iris[-renglones_entrenamiento, ]

Distintos tipos de Métodos para Modelar

Los métodos más utilizados para modelar aprendizaje automático son:

  • SVM: “Support Vector Machine” o Màquina de Vectores de Soporte. Hay varios subtipos: Lineal (svmLinaer), Radia (svmRadial), Polinómico (svmPoly), etc.
  • Árbol de Decisión: rpart
  • Redes Neuronales: nnet
  • Random Forest ó Bosques Aleatorios: rf

La validación cruzada (cross validation, VC) es una técnica para evaluar el rendimiento de un modelo, dividiendo los datos en múltiples subconjuntos, permitiendo medir su capacidad de generalización y evitar sobreajuste (overfitting).

La matriz de confusión (*Confusion Matrix) permite analizar que tan bieen funciona un modelo y qué tipos de errores comete. Lo que hace es comprarar las predicciones del modelo conlos valores reales de la variable objetivo.

Si la preción es muy alta en entrenamiento (95-100%), pero baja en prueba (60-70%), es una señal de sobreajuste (overfitting).

Modelo 1. SVM Lineal

modelo1 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmLinear", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento1 <- predict(modelo1, entrenamiento)
resultado_prueba1 <- predict(modelo1, prueba)

# Matriz de Confusión del Entrenamiento
mcre1 <- confusionMatrix(resultado_entrenamiento1,
                         entrenamiento$Species)
mcre1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
# Matriz de Confusión del Resultado de la Prueba
mcrp1 <- confusionMatrix(resultado_prueba1, prueba$Species)
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 2. SVM Radial

modelo2 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmRadial", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(sigma = 1, C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento2 <- predict(modelo2, entrenamiento)
resultado_prueba2 <- predict(modelo2, prueba)

# Matriz de Confusión del Entrenamiento
mcre2 <- confusionMatrix(resultado_entrenamiento2,
                         entrenamiento$Species)
mcre2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
# Matriz de Confusión del Resultado de la Prueba
mcrp2 <- confusionMatrix(resultado_prueba2, prueba$Species)
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 3. SVM Polinómico

modelo3 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmPoly", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(degree = 1, scale= 1, C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento3 <- predict(modelo3, entrenamiento)
resultado_prueba3 <- predict(modelo3, prueba)

# Matriz de Confusión del Entrenamiento
mcre3 <- confusionMatrix(resultado_entrenamiento3,
                         entrenamiento$Species)
mcre3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
# Matriz de Confusión del Resultado de la Prueba
mcrp3 <- confusionMatrix(resultado_prueba3, prueba$Species)
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 4. Árbol de Decisiones

modelo4 <- train(Species ~ ., data=entrenamiento, 
                 method = "rpart", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneLength = 10 # Cambiar hiperparámetros
                 )

resultado_entrenamiento4 <- predict(modelo4, entrenamiento)
resultado_prueba4 <- predict(modelo4, prueba)

# Matriz de Confusión del Entrenamiento
mcre4 <- confusionMatrix(resultado_entrenamiento4,
                         entrenamiento$Species)
mcre4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
# Matriz de Confusión del Resultado de la Prueba
mcrp4 <- confusionMatrix(resultado_prueba4, prueba$Species)
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 5. Redes Neuronales

modelo5 <- train(Species ~ ., data=entrenamiento, 
                 method = "nnet", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                         # Cambiar hiperparámetros
                 trace = FALSE # Pata ocultar resultados
                 )

resultado_entrenamiento5 <- predict(modelo5, entrenamiento)
resultado_prueba5 <- predict(modelo5, prueba)

# Matriz de Confusión del Entrenamiento
mcre5 <- confusionMatrix(resultado_entrenamiento5,
                         entrenamiento$Species)
mcre5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
# Matriz de Confusión del Resultado de la Prueba
mcrp5 <- confusionMatrix(resultado_prueba5, prueba$Species)
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

Modelo 6. Bosques Aleatorios

modelo6 <- train(Species ~ ., data=entrenamiento, 
                 method = "rf", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = expand.grid(mtry = c(2, 4, 6)) # Cambiar hiperparámetros
                 )
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
resultado_entrenamiento6 <- predict(modelo6, entrenamiento)
resultado_prueba6 <- predict(modelo6, prueba)

# Matriz de Confusión del Entrenamiento
mcre6 <- confusionMatrix(resultado_entrenamiento6,
                         entrenamiento$Species)
mcre6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
# Matriz de Confusión del Resultado de la Prueba
mcrp6 <- confusionMatrix(resultado_prueba6, prueba$Species)
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 6. Resumen de Resultados

resultados <- data.frame (
  "SVM Lineal" = c(mcre1$overall["Accuracy"], mcrp1$overall["Accuracy"]),
  "SVM Radial" = c(mcre2$overall["Accuracy"], mcrp2$overall["Accuracy"]),
  "SVM Polinómico" = c(mcre3$overall["Accuracy"], mcrp3$overall["Accuracy"]),
  "Árbol de Decisión" = c(mcre4$overall["Accuracy"], mcrp4$overall["Accuracy"]),
  "Redes Neuronales" = c(mcre5$overall["Accuracy"], mcrp5$overall["Accuracy"]),
  "Bosques Aleatorio"= c(mcre6$overall["Accuracy"], mcrp6$overall["Accuracy"])
)
rownames (resultados) <- c("Precisión de Entrenamiento", "Precisión de las Pruebas")
resultados
##                            SVM.Lineal SVM.Radial SVM.Polinómico
## Precisión de Entrenamiento  0.9916667  0.9916667      0.9916667
## Precisión de las Pruebas    0.9666667  0.9333333      0.9666667
##                            Árbol.de.Decisión Redes.Neuronales Bosques.Aleatorio
## Precisión de Entrenamiento         0.9666667        0.9666667         1.0000000
## Precisión de las Pruebas           0.9333333        0.9666667         0.9333333
LS0tCnRpdGxlOiAiQ0FSRVQiCmF1dGhvcjogIkEwMTczNDI5OV9GcmFuY2lzY28gU2FuZG92YWwgSGlkYWxnbyIKZGF0ZTogIjIwMjUtMDItMjAiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiBUUlVFCiAgICB0b2NfZmxvYXQ6IFRSVUUKICAgIGNvZGVfZG93bmxvYWQ6IFRSVUUKICAgIHRoZW1lOiAidW5pdGVkIgogICAgaGlnaGxpZ2h0OiAiZXNwcmVzc28iCi0tLQoKIVtdKC9Vc2Vycy9yb25pZS9EZXNrdG9wL0NsYXNlcyBUZWMvQ29uY2VudHJhY2lvzIFuL2lyaXMucG5nKQoKIyMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU6Ij5UZW9yw61hPC9zcGFuPgpFbCBwYXF1ZXRlIGNhcmV0KCpDbGFzc2lmaWNhdGlvbiBhbmQgUmVncmVzc2lvbiBUcmFpbmluZyopIGVzIHVuYSBoZXJyYW1pZW50YSBwb2Rlcm9zYSBwYXJhIGxhIGltcGxlbWVudGFjacOzbiBkZSBtb2RlbG9zIGRlICoqTWFjaGluZSBMZWFybmluZyoqLgoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZToiPkluc3RhbGFyIHBhcXVldGVzIHkgbGxhbWFyIGxpYnJlcmlhczwvc3Bhbj4KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KCiNpbnN0YWxsLnBhY2thZ2VzKCJjYXJldCIpICMgQWxnb3JpdG1vcyBkZSBhcHJlbmRpemFqZSBhdXRvbcOhdGljbwpsaWJyYXJ5KGNhcmV0KQojaW5zdGFsbC5wYWNrYWdlcygiZGF0YXNldHMiKSAjIFBhcmEgdXNhciBsYSBiYXNlIGRlIGRhdG9zICJJcmlzIgpsaWJyYXJ5KGRhdGFzZXRzKQojaW5zdGFsbC5wYWNrYWdlcygiZ2dwbG90MiIpICMgR3LDoWZpY2FzIGNvbiBtZWpvciBkaXNlw7FvCmxpYnJhcnkoZ2dwbG90MikKI2luc3RhbGwucGFja2FnZXMoImxhdHRpY2UiKSAjIENyZWFyIGdyw6FmaWNvcwpsaWJyYXJ5KGxhdHRpY2UpCiNpbnN0YWxsLnBhY2thZ2VzKCJEYXRhRXhwbG9yZXIiKSAjIEFuw6FsaXNpcyBEZXNjcmlwdGl2bwpsaWJyYXJ5KERhdGFFeHBsb3JlcikKI2luc3RhbGxlZC5wYWNrYWdlcygia2VybmxhYiIpCmxpYnJhcnkoa2VybmxhYikKCmBgYAoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZToiPkltcG9ydGFyIGxhIGJhc2UgZGUgZGF0b3M8L3NwYW4+CmBgYHtyfQoKZGYgPC0gZGF0YS5mcmFtZShpcmlzKQoKYGBgCgojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOiI+QW7DoWxpc2lzIERlc2NyaXB0aXZvPC9zcGFuPgpgYGB7cn0KCiMgY3JlYXRlX3JlcG9ydChkZikKCnBsb3RfbWlzc2luZyhkZikKcGxvdF9oaXN0b2dyYW0oZGYpCnBsb3RfY29ycmVsYXRpb24oZGYpCgpgYGAKCioqIE5PVEE6IExhIHZhcmlhYmxlIHF1ZSBxdWVyZW1vcyBwcmVkZWNpciBkZWJlIHRlbmVyIGZvcm1hdG8gZGUgRkFDVE9SLioqCgojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOiI+UGFydGlyIGxvcyBkYXRvcyA4MC0yMDwvc3Bhbj4KYGBge3J9CgpzZXQuc2VlZCgxMjMpCnJlbmdsb25lc19lbnRyZW5hbWllbnRvIDwtIGNyZWF0ZURhdGFQYXJ0aXRpb24oZGYkU3BlY2llcywgcCA9IDAuOCwgbGlzdCA9IEZBTFNFKQplbnRyZW5hbWllbnRvIDwtIGlyaXNbcmVuZ2xvbmVzX2VudHJlbmFtaWVudG8sIF0KcHJ1ZWJhIDwtIGlyaXNbLXJlbmdsb25lc19lbnRyZW5hbWllbnRvLCBdCgpgYGAKCiMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU6Ij5EaXN0aW50b3MgdGlwb3MgZGUgTcOpdG9kb3MgcGFyYSBNb2RlbGFyPC9zcGFuPgpMb3MgbcOpdG9kb3MgbcOhcyB1dGlsaXphZG9zIHBhcmEgbW9kZWxhciBhcHJlbmRpemFqZSBhdXRvbcOhdGljbyBzb246CgoqICoqU1ZNKio6ICoiU3VwcG9ydCBWZWN0b3IgTWFjaGluZSIqIG8gTcOgcXVpbmEgZGUgVmVjdG9yZXMgZGUgU29wb3J0ZS4gSGF5IHZhcmlvcyBzdWJ0aXBvczogTGluZWFsIChzdm1MaW5hZXIpLCBSYWRpYSAoc3ZtUmFkaWFsKSwgUG9saW7Ds21pY28gKHN2bVBvbHkpLCBldGMuCiogKirDgXJib2wgZGUgRGVjaXNpw7NuKio6IHJwYXJ0CiogKipSZWRlcyBOZXVyb25hbGVzKio6IG5uZXQKKiAqKlJhbmRvbSBGb3Jlc3QqKiDDsyBCb3NxdWVzIEFsZWF0b3Jpb3M6IHJmCgojIExhICoqdmFsaWRhY2nDs24gY3J1emFkYSoqICgqY3Jvc3MgdmFsaWRhdGlvbiosIFZDKSBlcyB1bmEgdMOpY25pY2EgcGFyYSBldmFsdWFyIGVsIHJlbmRpbWllbnRvIGRlIHVuIG1vZGVsbywgZGl2aWRpZW5kbyBsb3MgZGF0b3MgZW4gbcO6bHRpcGxlcyBzdWJjb25qdW50b3MsIHBlcm1pdGllbmRvIG1lZGlyIHN1IGNhcGFjaWRhZCBkZSBnZW5lcmFsaXphY2nDs24geSBldml0YXIgc29icmVhanVzdGUgKCpvdmVyZml0dGluZyopLgoKTGEgKiptYXRyaXogZGUgY29uZnVzacOzbioqICgqQ29uZnVzaW9uIE1hdHJpeCkgcGVybWl0ZSBhbmFsaXphciBxdWUgdGFuIGJpZWVuIGZ1bmNpb25hIHVuIG1vZGVsbyB5IHF1w6kgdGlwb3MgZGUgZXJyb3JlcyBjb21ldGUuIExvIHF1ZSBoYWNlIGVzIGNvbXByYXJhciBsYXMgcHJlZGljY2lvbmVzIGRlbCBtb2RlbG8gY29ubG9zIHZhbG9yZXMgcmVhbGVzIGRlIGxhIHZhcmlhYmxlIG9iamV0aXZvLgoKU2kgbGEgcHJlY2nDs24gZXMgbXV5IGFsdGEgZW4gZW50cmVuYW1pZW50byAoOTUtMTAwJSksIHBlcm8gYmFqYSBlbiBwcnVlYmEgKDYwLTcwJSksIGVzIHVuYSBzZcOxYWwgZGUgKnNvYnJlYWp1c3RlKiAob3ZlcmZpdHRpbmcpLgoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZToiPk1vZGVsbyAxLiBTVk0gTGluZWFsPC9zcGFuPgpgYGB7cn0KCm1vZGVsbzEgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGE9ZW50cmVuYW1pZW50bywgCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bUxpbmVhciIsICMgQ2FtYmlhcgogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXI9MTApLAogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZShDPTEpICMgQ2FtYmlhciBoaXBlcnBhcsOhbWV0cm9zCiAgICAgICAgICAgICAgICAgKQoKcmVzdWx0YWRvX2VudHJlbmFtaWVudG8xIDwtIHByZWRpY3QobW9kZWxvMSwgZW50cmVuYW1pZW50bykKcmVzdWx0YWRvX3BydWViYTEgPC0gcHJlZGljdChtb2RlbG8xLCBwcnVlYmEpCgojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBFbnRyZW5hbWllbnRvCm1jcmUxIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fZW50cmVuYW1pZW50bzEsCiAgICAgICAgICAgICAgICAgICAgICAgICBlbnRyZW5hbWllbnRvJFNwZWNpZXMpCm1jcmUxCgojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGUgbGEgUHJ1ZWJhCm1jcnAxIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhMSwgcHJ1ZWJhJFNwZWNpZXMpCm1jcnAxCgpgYGAKCiMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU6Ij5Nb2RlbG8gMi4gU1ZNIFJhZGlhbDwvc3Bhbj4KYGBge3J9Cgptb2RlbG8yIDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIAogICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJzdm1SYWRpYWwiLCAjIENhbWJpYXIKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyPTEwKSwKICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoc2lnbWEgPSAxLCBDPTEpICMgQ2FtYmlhciBoaXBlcnBhcsOhbWV0cm9zCiAgICAgICAgICAgICAgICAgKQoKcmVzdWx0YWRvX2VudHJlbmFtaWVudG8yIDwtIHByZWRpY3QobW9kZWxvMiwgZW50cmVuYW1pZW50bykKcmVzdWx0YWRvX3BydWViYTIgPC0gcHJlZGljdChtb2RlbG8yLCBwcnVlYmEpCgojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBFbnRyZW5hbWllbnRvCm1jcmUyIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fZW50cmVuYW1pZW50bzIsCiAgICAgICAgICAgICAgICAgICAgICAgICBlbnRyZW5hbWllbnRvJFNwZWNpZXMpCm1jcmUyCgojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGUgbGEgUHJ1ZWJhCm1jcnAyIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhMiwgcHJ1ZWJhJFNwZWNpZXMpCm1jcnAyCgpgYGAKCiMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU6Ij5Nb2RlbG8gMy4gU1ZNIFBvbGluw7NtaWNvPC9zcGFuPgpgYGB7cn0KCm1vZGVsbzMgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGE9ZW50cmVuYW1pZW50bywgCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bVBvbHkiLCAjIENhbWJpYXIKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyPTEwKSwKICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoZGVncmVlID0gMSwgc2NhbGU9IDEsIEM9MSkgIyBDYW1iaWFyIGhpcGVycGFyw6FtZXRyb3MKICAgICAgICAgICAgICAgICApCgpyZXN1bHRhZG9fZW50cmVuYW1pZW50bzMgPC0gcHJlZGljdChtb2RlbG8zLCBlbnRyZW5hbWllbnRvKQpyZXN1bHRhZG9fcHJ1ZWJhMyA8LSBwcmVkaWN0KG1vZGVsbzMsIHBydWViYSkKCiMgTWF0cml6IGRlIENvbmZ1c2nDs24gZGVsIEVudHJlbmFtaWVudG8KbWNyZTMgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvMywKICAgICAgICAgICAgICAgICAgICAgICAgIGVudHJlbmFtaWVudG8kU3BlY2llcykKbWNyZTMKCiMgTWF0cml6IGRlIENvbmZ1c2nDs24gZGVsIFJlc3VsdGFkbyBkZSBsYSBQcnVlYmEKbWNycDMgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmEzLCBwcnVlYmEkU3BlY2llcykKbWNycDMKCmBgYAoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZToiPk1vZGVsbyA0LiDDgXJib2wgZGUgRGVjaXNpb25lczwvc3Bhbj4KYGBge3J9Cgptb2RlbG80IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIAogICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJycGFydCIsICMgQ2FtYmlhcgogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXI9MTApLAogICAgICAgICAgICAgICAgIHR1bmVMZW5ndGggPSAxMCAjIENhbWJpYXIgaGlwZXJwYXLDoW1ldHJvcwogICAgICAgICAgICAgICAgICkKCnJlc3VsdGFkb19lbnRyZW5hbWllbnRvNCA8LSBwcmVkaWN0KG1vZGVsbzQsIGVudHJlbmFtaWVudG8pCnJlc3VsdGFkb19wcnVlYmE0IDwtIHByZWRpY3QobW9kZWxvNCwgcHJ1ZWJhKQoKIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgRW50cmVuYW1pZW50bwptY3JlNCA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG80LAogICAgICAgICAgICAgICAgICAgICAgICAgZW50cmVuYW1pZW50byRTcGVjaWVzKQptY3JlNAoKIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQptY3JwNCA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTQsIHBydWViYSRTcGVjaWVzKQptY3JwNAoKYGBgCgojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOiI+TW9kZWxvIDUuIFJlZGVzIE5ldXJvbmFsZXM8L3NwYW4+CmBgYHtyfQoKbW9kZWxvNSA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YT1lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAibm5ldCIsICMgQ2FtYmlhcgogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXI9MTApLAogICAgICAgICAgICAgICAgICAgICAgICAgIyBDYW1iaWFyIGhpcGVycGFyw6FtZXRyb3MKICAgICAgICAgICAgICAgICB0cmFjZSA9IEZBTFNFICMgUGF0YSBvY3VsdGFyIHJlc3VsdGFkb3MKICAgICAgICAgICAgICAgICApCgpyZXN1bHRhZG9fZW50cmVuYW1pZW50bzUgPC0gcHJlZGljdChtb2RlbG81LCBlbnRyZW5hbWllbnRvKQpyZXN1bHRhZG9fcHJ1ZWJhNSA8LSBwcmVkaWN0KG1vZGVsbzUsIHBydWViYSkKCiMgTWF0cml6IGRlIENvbmZ1c2nDs24gZGVsIEVudHJlbmFtaWVudG8KbWNyZTUgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvNSwKICAgICAgICAgICAgICAgICAgICAgICAgIGVudHJlbmFtaWVudG8kU3BlY2llcykKbWNyZTUKCiMgTWF0cml6IGRlIENvbmZ1c2nDs24gZGVsIFJlc3VsdGFkbyBkZSBsYSBQcnVlYmEKbWNycDUgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmE1LCBwcnVlYmEkU3BlY2llcykKbWNycDUKCmBgYAoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZToiPk1vZGVsbyA2LiBCb3NxdWVzIEFsZWF0b3Jpb3M8L3NwYW4+CmBgYHtyfQoKbW9kZWxvNiA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YT1lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmYiLCAjIENhbWJpYXIKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyPTEwKSwKICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGV4cGFuZC5ncmlkKG10cnkgPSBjKDIsIDQsIDYpKSAjIENhbWJpYXIgaGlwZXJwYXLDoW1ldHJvcwogICAgICAgICAgICAgICAgICkKCnJlc3VsdGFkb19lbnRyZW5hbWllbnRvNiA8LSBwcmVkaWN0KG1vZGVsbzYsIGVudHJlbmFtaWVudG8pCnJlc3VsdGFkb19wcnVlYmE2IDwtIHByZWRpY3QobW9kZWxvNiwgcHJ1ZWJhKQoKIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgRW50cmVuYW1pZW50bwptY3JlNiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG82LAogICAgICAgICAgICAgICAgICAgICAgICAgZW50cmVuYW1pZW50byRTcGVjaWVzKQptY3JlNgoKIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQptY3JwNiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTYsIHBydWViYSRTcGVjaWVzKQptY3JwNgoKYGBgCgojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOiI+TW9kZWxvIDYuIFJlc3VtZW4gZGUgUmVzdWx0YWRvczwvc3Bhbj4KYGBge3J9CgpyZXN1bHRhZG9zIDwtIGRhdGEuZnJhbWUgKAogICJTVk0gTGluZWFsIiA9IGMobWNyZTEkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDEkb3ZlcmFsbFsiQWNjdXJhY3kiXSksCiAgIlNWTSBSYWRpYWwiID0gYyhtY3JlMiRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwMiRvdmVyYWxsWyJBY2N1cmFjeSJdKSwKICAiU1ZNIFBvbGluw7NtaWNvIiA9IGMobWNyZTMkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDMkb3ZlcmFsbFsiQWNjdXJhY3kiXSksCiAgIsOBcmJvbCBkZSBEZWNpc2nDs24iID0gYyhtY3JlNCRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNCRvdmVyYWxsWyJBY2N1cmFjeSJdKSwKICAiUmVkZXMgTmV1cm9uYWxlcyIgPSBjKG1jcmU1JG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA1JG92ZXJhbGxbIkFjY3VyYWN5Il0pLAogICJCb3NxdWVzIEFsZWF0b3JpbyI9IGMobWNyZTYkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDYkb3ZlcmFsbFsiQWNjdXJhY3kiXSkKKQpyb3duYW1lcyAocmVzdWx0YWRvcykgPC0gYygiUHJlY2lzacOzbiBkZSBFbnRyZW5hbWllbnRvIiwgIlByZWNpc2nDs24gZGUgbGFzIFBydWViYXMiKQpyZXN1bHRhZG9zCgpgYGAKCg==