Teoria

El paquete caret(Classification And REgression Training) es una herramienta poderosa para la implementacion de modelos de Machine Learning.

Instalar paquetes y librerias

#install.packages("caret") #Algoritmos de aprendizajes automatico
library(caret)
#install.packages("datasets") #Para usar la base de datos de "IRIS"
library(datasets)
#install.packages("ggplot2") #Graficas con mejor diseño
library(ggplot2)
#install.packages("lattice") # Crear graficos
library(lattice)
#install.packages("DataExplorer") #Analisis Descriptivo
library(DataExplorer)
#install.packages("kernlab") #Metodo de Aprendizaje Automatico
library(kernlab)

Importar base de datos

df <- data.frame(iris)

Analisis Descriptivo

# create_report(df)
plot_missing(df)

plot_histogram(df)

plot_correlation(df)

** NOTA: La variable que queremos predecir debe tener formato de FACTOR. **

Partir los datos 80-20

set.seed(123)
renglones_entrenamiento <- createDataPartition(df$Species, p=0.8, list=FALSE)
entrenamiento <- iris[renglones_entrenamiento, ]
prueba <- iris[-renglones_entrenamiento, ]

Distintos tipos de Metodos para modelar

Los metodos mas utilizados para modelar aprendizaje automatico son:

  • SVM: Support Vector Machine o Maquina de Vectores de Soporte. Hay varios subtipos: Lineal (svmLinear); Radial (svmRadial), Polinomico(svmPoly), etc.
  • Arbol de Decision: rpart
  • Redes Neuronales: nnet
  • Random Forest o Bosques Aleatorios: rf

La validacion cruzada (Cross validation, CV) es una tecnica para evaluar el rendimiento de un modelo, dividiendo los datos en mutliples subconjuntos, permitiendo medir su capacidad de generalizacion y evitar sobreajuste (overfitting).

La mmatriz de confusion(Confusion matrix) permite analizar que tan bien funciona un modelo y que tipos de errores comete. Lo que haces es comparar las predicciones del modelo con los valres reales de la variable objetivo

Si la precision es muy alta en entrenamiento (95-100%), pero baja en prueba (60-70%), es una señal de sobreajuste

Modelo 1. SVM Lineal

modelo1 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmLinear", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento1 <- predict(modelo1, entrenamiento)
resultado_prueba1 <- predict(modelo1, prueba)

#Matriz de Confusion del Resultado del Entrenamiento
mcre1 <- confusionMatrix(resultado_entrenamiento1, entrenamiento$Species)
mcre1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
#Matriz de Confusion del Resultado de la Prueba
mcrp1 <- confusionMatrix(resultado_prueba1, prueba$Species)
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 2. SVM Radial

modelo2 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmRadial", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(sigma=1, C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento2 <- predict(modelo2, entrenamiento)
resultado_prueba2 <- predict(modelo2, prueba)

#Matriz de Confusion del Resultado del Entrenamiento
mcre2 <- confusionMatrix(resultado_entrenamiento2, entrenamiento$Species)
mcre2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
#Matriz de Confusion del Resultado de la Prueba
mcrp2 <- confusionMatrix(resultado_prueba2, prueba$Species)
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 3. SVM Polinomico

modelo3 <- train(Species ~ ., data=entrenamiento, 
                 method = "svmPoly", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneGrid = data.frame(degree=1, scale=1, C=1) # Cambiar hiperparámetros
                 )

resultado_entrenamiento3 <- predict(modelo3, entrenamiento)
resultado_prueba3 <- predict(modelo3, prueba)

#Matriz de Confusion del Resultado del Entrenamiento
mcre3 <- confusionMatrix(resultado_entrenamiento3, entrenamiento$Species)
mcre3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
#Matriz de Confusion del Resultado de la Prueba
mcrp3 <- confusionMatrix(resultado_prueba3, prueba$Species)
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 4. Arbol de decisiones

modelo4 <- train(Species ~ ., data=entrenamiento, 
                 method = "rpart", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                 tuneLength = 10 # Cambiar hiperparámetros
                 )

resultado_entrenamiento4 <- predict(modelo4, entrenamiento)
resultado_prueba4 <- predict(modelo4, prueba)

#Matriz de Confusion del Resultado del Entrenamiento
mcre4 <- confusionMatrix(resultado_entrenamiento4, entrenamiento$Species)
mcre4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
#Matriz de Confusion del Resultado de la Prueba
mcrp4 <- confusionMatrix(resultado_prueba4, prueba$Species)
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 5. Redes neuronales

modelo5 <- train(Species ~ ., data=entrenamiento, 
                 method = "nnet", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                  # Cambiar hiperparámetros
                 trace=FALSE # Para ocultar resultados
                 )

resultado_entrenamiento5 <- predict(modelo5, entrenamiento)
resultado_prueba5 <- predict(modelo5, prueba)

#Matriz de Confusion del Resultado del Entrenamiento
mcre5 <- confusionMatrix(resultado_entrenamiento5, entrenamiento$Species)
mcre5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
#Matriz de Confusion del Resultado de la Prueba
mcrp5 <- confusionMatrix(resultado_prueba5, prueba$Species)
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

Modelo 6. Random Forest

modelo6 <- train(Species ~ ., data=entrenamiento, 
                 method = "rf", # Cambiar
                 preProcess=c("scale", "center"),
                 trControl = trainControl(method="cv", number=10),
                  tuneGrid = expand.grid(mtry= c(2,4,6))# Cambiar hiperparámetros
                 )
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
resultado_entrenamiento6 <- predict(modelo6, entrenamiento)
resultado_prueba6 <- predict(modelo6, prueba)

# Matriz de Confusión del Entrenamiento
mcre6 <- confusionMatrix(resultado_entrenamiento6,
                         entrenamiento$Species)
mcre6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
# Matriz de Confusión del Resultado de la Prueba
mcrp6 <- confusionMatrix(resultado_prueba6, prueba$Species)
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Resumen de Resultados

resultados <- data.frame(
  "SVM Lineal" = c(mcre1$overall["Accuracy"], mcrp1$overall["Accuracy"]), 
  "SVM Radial" = c(mcre2$overall["Accuracy"], mcrp2$overall["Accuracy"]), 
  "SVM Polinomico" = c(mcre3$overall["Accuracy"], mcrp3$overall["Accuracy"]), 
  "Arbol de Decision" = c(mcre4$overall["Accuracy"], mcrp4$overall["Accuracy"]), 
  "Redes Neuronales"  = c(mcre5$overall["Accuracy"], mcrp5$overall["Accuracy"]), 
  "Bosques Aleatorios" = c(mcre6$overall["Accuracy"], mcrp6$overall["Accuracy"])
)
rownames(resultados) <- c("Precision de Entrenamiento", "Precision de Prueba")
resultados
##                            SVM.Lineal SVM.Radial SVM.Polinomico
## Precision de Entrenamiento  0.9916667  0.9916667      0.9916667
## Precision de Prueba         0.9666667  0.9333333      0.9666667
##                            Arbol.de.Decision Redes.Neuronales
## Precision de Entrenamiento         0.9666667        0.9666667
## Precision de Prueba                0.9333333        0.9666667
##                            Bosques.Aleatorios
## Precision de Entrenamiento          1.0000000
## Precision de Prueba                 0.9333333
LS0tDQp0aXRsZTogIkNBUkVUIg0KYXV0aG9yOiAiRGFuaWVsIFrDoXJhdGUgLSAgQTAxMjg1NTYxIg0KZGF0ZTogIjIwMjUtMDItMjEiDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgdG9jOiBUUlVFDQogICAgdG9jX2Zsb2F0OiBUUlVFDQogICAgY29kZV9kb3dubG9hZDogVFJVRQ0KICAgIHRoZW1lOiAidW5pdGVkIg0KICAgIGhpZ2hsaWdodDogImVzcHJlc3NvIg0KLS0tDQoNCiFbXShDOlxcVGVjXFw2dG8gc2VtZXN0cmVcXE1vZHVsbyAyXGlyaXMuanBnKQ0KDQojIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPlRlb3JpYTwvc3Bhbj4NCkVsIHBhcXVldGUgY2FyZXQoKkNsYXNzaWZpY2F0aW9uIEFuZCBSRWdyZXNzaW9uIFRyYWluaW5nKikgZXMgdW5hIGhlcnJhbWllbnRhIHBvZGVyb3NhIHBhcmEgbGEgaW1wbGVtZW50YWNpb24gZGUgbW9kZWxvcyBkZSAqKk1hY2hpbmUgTGVhcm5pbmcqKi4NCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5JbnN0YWxhciBwYXF1ZXRlcyB5IGxpYnJlcmlhczwvc3Bhbj4NCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQojaW5zdGFsbC5wYWNrYWdlcygiY2FyZXQiKSAjQWxnb3JpdG1vcyBkZSBhcHJlbmRpemFqZXMgYXV0b21hdGljbw0KbGlicmFyeShjYXJldCkNCiNpbnN0YWxsLnBhY2thZ2VzKCJkYXRhc2V0cyIpICNQYXJhIHVzYXIgbGEgYmFzZSBkZSBkYXRvcyBkZSAiSVJJUyINCmxpYnJhcnkoZGF0YXNldHMpDQojaW5zdGFsbC5wYWNrYWdlcygiZ2dwbG90MiIpICNHcmFmaWNhcyBjb24gbWVqb3IgZGlzZcOxbw0KbGlicmFyeShnZ3Bsb3QyKQ0KI2luc3RhbGwucGFja2FnZXMoImxhdHRpY2UiKSAjIENyZWFyIGdyYWZpY29zDQpsaWJyYXJ5KGxhdHRpY2UpDQojaW5zdGFsbC5wYWNrYWdlcygiRGF0YUV4cGxvcmVyIikgI0FuYWxpc2lzIERlc2NyaXB0aXZvDQpsaWJyYXJ5KERhdGFFeHBsb3JlcikNCiNpbnN0YWxsLnBhY2thZ2VzKCJrZXJubGFiIikgI01ldG9kbyBkZSBBcHJlbmRpemFqZSBBdXRvbWF0aWNvDQpsaWJyYXJ5KGtlcm5sYWIpDQpgYGANCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5JbXBvcnRhciBiYXNlIGRlIGRhdG9zPC9zcGFuPg0KYGBge3J9DQpkZiA8LSBkYXRhLmZyYW1lKGlyaXMpDQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPkFuYWxpc2lzIERlc2NyaXB0aXZvPC9zcGFuPg0KYGBge3J9DQojIGNyZWF0ZV9yZXBvcnQoZGYpDQpwbG90X21pc3NpbmcoZGYpDQpwbG90X2hpc3RvZ3JhbShkZikNCnBsb3RfY29ycmVsYXRpb24oZGYpDQpgYGANCg0KKiogTk9UQTogTGEgdmFyaWFibGUgcXVlIHF1ZXJlbW9zIHByZWRlY2lyIGRlYmUgdGVuZXIgZm9ybWF0byBkZSBGQUNUT1IuICoqDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5QYXJ0aXIgbG9zIGRhdG9zIDgwLTIwPC9zcGFuPg0KYGBge3J9DQpzZXQuc2VlZCgxMjMpDQpyZW5nbG9uZXNfZW50cmVuYW1pZW50byA8LSBjcmVhdGVEYXRhUGFydGl0aW9uKGRmJFNwZWNpZXMsIHA9MC44LCBsaXN0PUZBTFNFKQ0KZW50cmVuYW1pZW50byA8LSBpcmlzW3Jlbmdsb25lc19lbnRyZW5hbWllbnRvLCBdDQpwcnVlYmEgPC0gaXJpc1stcmVuZ2xvbmVzX2VudHJlbmFtaWVudG8sIF0NCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOyI+RGlzdGludG9zIHRpcG9zIGRlIE1ldG9kb3MgcGFyYSBtb2RlbGFyPC9zcGFuPg0KTG9zIG1ldG9kb3MgbWFzIHV0aWxpemFkb3MgcGFyYSBtb2RlbGFyIGFwcmVuZGl6YWplIGF1dG9tYXRpY28gc29uOg0KDQoqICoqU1ZNKio6ICpTdXBwb3J0IFZlY3RvciBNYWNoaW5lKiBvIE1hcXVpbmEgZGUgVmVjdG9yZXMgZGUgU29wb3J0ZS4gSGF5IHZhcmlvcyBzdWJ0aXBvczogTGluZWFsIChzdm1MaW5lYXIpOyBSYWRpYWwgKHN2bVJhZGlhbCksIFBvbGlub21pY28oc3ZtUG9seSksIGV0Yy4NCiogKipBcmJvbCBkZSBEZWNpc2lvbioqOiBycGFydA0KKiAqKlJlZGVzIE5ldXJvbmFsZXMqKjogbm5ldA0KKiAqKlJhbmRvbSBGb3Jlc3QqKiBvIEJvc3F1ZXMgQWxlYXRvcmlvczogcmYNCg0KTGEgKip2YWxpZGFjaW9uIGNydXphZGEqKiAoKkNyb3NzIHZhbGlkYXRpb24qLCBDVikgZXMgdW5hIHRlY25pY2EgcGFyYSBldmFsdWFyIGVsIHJlbmRpbWllbnRvIGRlIHVuIG1vZGVsbywgZGl2aWRpZW5kbyBsb3MgZGF0b3MgZW4gbXV0bGlwbGVzIHN1YmNvbmp1bnRvcywgcGVybWl0aWVuZG8gbWVkaXIgc3UgY2FwYWNpZGFkIGRlIGdlbmVyYWxpemFjaW9uIHkgZXZpdGFyIHNvYnJlYWp1c3RlICgqb3ZlcmZpdHRpbmcqKS4NCg0KTGEgKiptbWF0cml6IGRlIGNvbmZ1c2lvbioqKCpDb25mdXNpb24gbWF0cml4KikgcGVybWl0ZSBhbmFsaXphciBxdWUgdGFuIGJpZW4gZnVuY2lvbmEgdW4gbW9kZWxvIHkgcXVlIHRpcG9zIGRlIGVycm9yZXMgY29tZXRlLiBMbyBxdWUgaGFjZXMgZXMgY29tcGFyYXIgbGFzIHByZWRpY2Npb25lcyBkZWwgbW9kZWxvIGNvbiBsb3MgdmFscmVzIHJlYWxlcyBkZSBsYSB2YXJpYWJsZSBvYmpldGl2bw0KDQpTaSBsYSBwcmVjaXNpb24gZXMgbXV5IGFsdGEgZW4gZW50cmVuYW1pZW50byAoOTUtMTAwJSksIHBlcm8gYmFqYSBlbiBwcnVlYmEgKDYwLTcwJSksIGVzIHVuYSBzZcOxYWwgZGUgKipzb2JyZWFqdXN0ZSoqDQoNCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPk1vZGVsbyAxLiAgU1ZNIExpbmVhbDwvc3Bhbj4NCmBgYHtyfQ0KbW9kZWxvMSA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YT1lbnRyZW5hbWllbnRvLCANCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bUxpbmVhciIsICMgQ2FtYmlhcg0KICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLA0KICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlcj0xMCksDQogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZShDPTEpICMgQ2FtYmlhciBoaXBlcnBhcsOhbWV0cm9zDQogICAgICAgICAgICAgICAgICkNCg0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG8xIDwtIHByZWRpY3QobW9kZWxvMSwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmExIDwtIHByZWRpY3QobW9kZWxvMSwgcHJ1ZWJhKQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvDQptY3JlMSA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG8xLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlMQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQ0KbWNycDEgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmExLCBwcnVlYmEkU3BlY2llcykNCm1jcnAxDQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPk1vZGVsbyAyLiAgU1ZNIFJhZGlhbDwvc3Bhbj4NCmBgYHtyfQ0KbW9kZWxvMiA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YT1lbnRyZW5hbWllbnRvLCANCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bVJhZGlhbCIsICMgQ2FtYmlhcg0KICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLA0KICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlcj0xMCksDQogICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZShzaWdtYT0xLCBDPTEpICMgQ2FtYmlhciBoaXBlcnBhcsOhbWV0cm9zDQogICAgICAgICAgICAgICAgICkNCg0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG8yIDwtIHByZWRpY3QobW9kZWxvMiwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmEyIDwtIHByZWRpY3QobW9kZWxvMiwgcHJ1ZWJhKQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvDQptY3JlMiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG8yLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlMg0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQ0KbWNycDIgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmEyLCBwcnVlYmEkU3BlY2llcykNCm1jcnAyDQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPk1vZGVsbyAzLiAgU1ZNIFBvbGlub21pY288L3NwYW4+DQpgYGB7cn0NCm1vZGVsbzMgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGE9ZW50cmVuYW1pZW50bywgDQogICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJzdm1Qb2x5IiwgIyBDYW1iaWFyDQogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksDQogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2Q9ImN2IiwgbnVtYmVyPTEwKSwNCiAgICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKGRlZ3JlZT0xLCBzY2FsZT0xLCBDPTEpICMgQ2FtYmlhciBoaXBlcnBhcsOhbWV0cm9zDQogICAgICAgICAgICAgICAgICkNCg0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG8zIDwtIHByZWRpY3QobW9kZWxvMywgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmEzIDwtIHByZWRpY3QobW9kZWxvMywgcHJ1ZWJhKQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvDQptY3JlMyA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG8zLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlMw0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQ0KbWNycDMgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmEzLCBwcnVlYmEkU3BlY2llcykNCm1jcnAzDQpgYGANCg0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOyI+TW9kZWxvIDQuICBBcmJvbCBkZSBkZWNpc2lvbmVzPC9zcGFuPg0KYGBge3J9DQptb2RlbG80IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAicnBhcnQiLCAjIENhbWJpYXINCiAgICAgICAgICAgICAgICAgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsICJjZW50ZXIiKSwNCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXI9MTApLA0KICAgICAgICAgICAgICAgICB0dW5lTGVuZ3RoID0gMTAgIyBDYW1iaWFyIGhpcGVycGFyw6FtZXRyb3MNCiAgICAgICAgICAgICAgICAgKQ0KDQpyZXN1bHRhZG9fZW50cmVuYW1pZW50bzQgPC0gcHJlZGljdChtb2RlbG80LCBlbnRyZW5hbWllbnRvKQ0KcmVzdWx0YWRvX3BydWViYTQgPC0gcHJlZGljdChtb2RlbG80LCBwcnVlYmEpDQoNCiNNYXRyaXogZGUgQ29uZnVzaW9uIGRlbCBSZXN1bHRhZG8gZGVsIEVudHJlbmFtaWVudG8NCm1jcmU0IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fZW50cmVuYW1pZW50bzQsIGVudHJlbmFtaWVudG8kU3BlY2llcykNCm1jcmU0DQoNCiNNYXRyaXogZGUgQ29uZnVzaW9uIGRlbCBSZXN1bHRhZG8gZGUgbGEgUHJ1ZWJhDQptY3JwNCA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTQsIHBydWViYSRTcGVjaWVzKQ0KbWNycDQNCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOyI+TW9kZWxvIDUuICBSZWRlcyBuZXVyb25hbGVzPC9zcGFuPg0KYGBge3J9DQptb2RlbG81IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAibm5ldCIsICMgQ2FtYmlhcg0KICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLA0KICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kPSJjdiIsIG51bWJlcj0xMCksDQogICAgICAgICAgICAgICAgICAjIENhbWJpYXIgaGlwZXJwYXLDoW1ldHJvcw0KICAgICAgICAgICAgICAgICB0cmFjZT1GQUxTRSAjIFBhcmEgb2N1bHRhciByZXN1bHRhZG9zDQogICAgICAgICAgICAgICAgICkNCg0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG81IDwtIHByZWRpY3QobW9kZWxvNSwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmE1IDwtIHByZWRpY3QobW9kZWxvNSwgcHJ1ZWJhKQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlbCBFbnRyZW5hbWllbnRvDQptY3JlNSA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG81LCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlNQ0KDQojTWF0cml6IGRlIENvbmZ1c2lvbiBkZWwgUmVzdWx0YWRvIGRlIGxhIFBydWViYQ0KbWNycDUgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmE1LCBwcnVlYmEkU3BlY2llcykNCm1jcnA1DQpgYGANCg0KIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPk1vZGVsbyA2LiBSYW5kb20gRm9yZXN0PC9zcGFuPg0KYGBge3J9DQptb2RlbG82IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIA0KICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmYiLCAjIENhbWJpYXINCiAgICAgICAgICAgICAgICAgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsICJjZW50ZXIiKSwNCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZD0iY3YiLCBudW1iZXI9MTApLA0KICAgICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBleHBhbmQuZ3JpZChtdHJ5PSBjKDIsNCw2KSkjIENhbWJpYXIgaGlwZXJwYXLDoW1ldHJvcw0KICAgICAgICAgICAgICAgICApDQoNCnJlc3VsdGFkb19lbnRyZW5hbWllbnRvNiA8LSBwcmVkaWN0KG1vZGVsbzYsIGVudHJlbmFtaWVudG8pDQpyZXN1bHRhZG9fcHJ1ZWJhNiA8LSBwcmVkaWN0KG1vZGVsbzYsIHBydWViYSkNCg0KIyBNYXRyaXogZGUgQ29uZnVzacOzbiBkZWwgRW50cmVuYW1pZW50bw0KbWNyZTYgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvNiwNCiAgICAgICAgICAgICAgICAgICAgICAgICBlbnRyZW5hbWllbnRvJFNwZWNpZXMpDQptY3JlNg0KDQojIE1hdHJpeiBkZSBDb25mdXNpw7NuIGRlbCBSZXN1bHRhZG8gZGUgbGEgUHJ1ZWJhDQptY3JwNiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTYsIHBydWViYSRTcGVjaWVzKQ0KbWNycDYNCmBgYA0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjogcHVycGxlOyI+UmVzdW1lbiBkZSBSZXN1bHRhZG9zPC9zcGFuPg0KYGBge3J9DQpyZXN1bHRhZG9zIDwtIGRhdGEuZnJhbWUoDQogICJTVk0gTGluZWFsIiA9IGMobWNyZTEkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDEkb3ZlcmFsbFsiQWNjdXJhY3kiXSksIA0KICAiU1ZNIFJhZGlhbCIgPSBjKG1jcmUyJG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnAyJG92ZXJhbGxbIkFjY3VyYWN5Il0pLCANCiAgIlNWTSBQb2xpbm9taWNvIiA9IGMobWNyZTMkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDMkb3ZlcmFsbFsiQWNjdXJhY3kiXSksIA0KICAiQXJib2wgZGUgRGVjaXNpb24iID0gYyhtY3JlNCRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNCRvdmVyYWxsWyJBY2N1cmFjeSJdKSwgDQogICJSZWRlcyBOZXVyb25hbGVzIiAgPSBjKG1jcmU1JG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA1JG92ZXJhbGxbIkFjY3VyYWN5Il0pLCANCiAgIkJvc3F1ZXMgQWxlYXRvcmlvcyIgPSBjKG1jcmU2JG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA2JG92ZXJhbGxbIkFjY3VyYWN5Il0pDQopDQpyb3duYW1lcyhyZXN1bHRhZG9zKSA8LSBjKCJQcmVjaXNpb24gZGUgRW50cmVuYW1pZW50byIsICJQcmVjaXNpb24gZGUgUHJ1ZWJhIikNCnJlc3VsdGFkb3MNCmBgYA0KDQo=