Teoría

El paquete caret(Classifciation and Regression Training) es una herramienta poderosa para la implementación de modelos de Machine Learning.

Paquetes y librerias

#install.packages("caret")
library(caret)#algoritmos de aprendiaje automártico
#install.packages("datasets")
library(datasets) # PAra usar la base de datos "Iris"
library(ggplot2)
#install.packages("lattice")
library(lattice) #Crear base de datos
#install.packages("DataExplorer")
library(DataExplorer)

Importar base de datos

df <- data.frame(iris)

Análisis descriptivo

# create_report(df)
plot_histogram(df)

plot_missing(df)

plot_correlation(df)

** NOTA: La variable que queremos predecir tiene que tener formato de FACTOR.**

División de datos 80/20

set.seed(123)
r_train <- createDataPartition(df$Species, p=0.8, list=FALSE)
train<- iris[r_train, ]
test <- iris [-r_train,]

Moelos

Los metodos mas utilizados para entrenar aprendizaje automático son:

  • SVM: Support Vector Machine a Máquina de Vectores de Soporte. Hay varios subtipos: Lineal(svmLinear), Radial(svmRadial), Polinomico (svmPoly), etc.
  • Arból de decision: rpart
  • Redes neuronales: nnet
  • Random Forest: Bosques Aleatorios: rf La validación cruzada (CV) es una técnica para evaluar el rendimiento de un modelo, dividiendo los datos en multiples subconjuntos, permitiendo medir su capacidad de generalización y evitar sobreajuste (overfitting)

La Matriz de confusión (Confusion Matrix) permite analizar qué tan bien funciona un modelo y qué tipps de errores comete. Lo que hace es comparar las predicciones del modelo con los valores reales de la variable objetivo.

Si la precsión es muy alta en entrenamiento (95% - 100%), pero baja en prueba (60- 70%), es una señal de sobreajuste

Modelo 1. SVM Lineal

modelo1 <- train(
  Species ~ ., 
  data = train, 
  method = "svmLinear",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  tuneGrid = data.frame(C = 1)  # Se separa correctamente de trainControl
)

Predicciones

resultado_train <- predict(modelo1, train)
resultado_test <- predict(modelo1, test)

Matriz de Confusión

mcrp1<- confusionMatrix(resultado_train, train$Species)
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp2<- confusionMatrix(resultado_test, test$Species)
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

**Modelo 2. SVM Radial*

modelo2 <- train(
  Species ~ ., 
  data = train, 
  method = "svmRadial",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  tuneGrid = data.frame(sigma=1,C = 1)  # Se separa correctamente de trainControl
)

Predicciones

resultado_train2 <- predict(modelo2, train)
resultado_test2 <- predict(modelo2, test)

Matriz de Confusión

mcrp3<- confusionMatrix(resultado_train2, train$Species)
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp4<- confusionMatrix(resultado_test2, test$Species)
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 3. SVM Polinomial

modelo3 <- train(
  Species ~ ., 
  data = train, 
  method = "svmPoly",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  tuneGrid = data.frame(degree=1, scale=1,C = 1)  # Se separa correctamente de trainControl
)

Predicciones

resultado_train3 <- predict(modelo3, train)
resultado_test3 <- predict(modelo3, test)

Matriz de Confusión

mcrp5<- confusionMatrix(resultado_train3, train$Species)
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp6<- confusionMatrix(resultado_test3, test$Species)
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

Modelo 4. .Arból de Decisió

modelo4 <- train(
  Species ~ ., 
  data = train, 
  method = "rpart",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  tuneLength=10  # Se separa correctamente de trainControl
)

Predicciones

resultado_train4 <- predict(modelo4, train)
resultado_test4<- predict(modelo4, test)

Matriz de Confusión

mcrp7<- confusionMatrix(resultado_train4, train$Species)
mcrp7
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
mcrp8<- confusionMatrix(resultado_test4, test$Species)
mcrp8
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Modelo 5. Redes Neuronales 

modelo5 <- train(
  Species ~ ., 
  data = train, 
  method = "nnet",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  trace=FALSE
)

**Predicciones*

resultado_train5 <- predict(modelo5, train)
resultado_test5 <- predict(modelo5, test)

Matriz de Confusión

mcrp9<- confusionMatrix(resultado_train5, train$Species)
mcrp9
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
mcrp10<- confusionMatrix(resultado_test5, test$Species)
mcrp10
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

Modelo 6. Random Forest 

modelo6 <- train(
  Species ~ ., 
  data = train, 
  method = "rf",  # Cambiar si deseas otro método
  preProcess = c("scale", "center"),
  trControl = trainControl(method = "cv", number = 10),  # Se cierra correctamente el paréntesis
  tuneLength=10  # Se separa correctamente de trainControl
)
## note: only 3 unique complexity parameters in default grid. Truncating the grid to 3 .

Predicciones

resultado_train6 <- predict(modelo6, train)
resultado_test6 <- predict(modelo6, test)

Matriz de Confusión

mcrp11<- confusionMatrix(resultado_train6, train$Species)
mcrp11
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
mcrp12<- confusionMatrix(resultado_test6, test$Species)
mcrp12
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Resultados de los modelos

resultados <- data.frame(
  "SVM Lineal" = c(mcrp1$overall["Accuracy"], mcrp2$overall["Accuracy"]),
  
  "SVM Radial" = c(mcrp3$overall["Accuracy"], mcrp4$overall["Accuracy"]),
  
  "SVM Polinómico" = c(mcrp5$overall["Accuracy"], mcrp6$overall["Accuracy"]),
  
  "Árbol de decisión" = c(mcrp7$overall["Accuracy"], mcrp8$overall["Accuracy"]),
  
  "Redes Neuronales" = c(mcrp9$overall["Accuracy"], mcrp10$overall["Accuracy"]),
  
  "Bosques Aleatorios" = c(mcrp11$overall["Accuracy"], mcrp12$overall["Accuracy"])
  
)
rownames(resultados)<- c("Precisión de Entrenamiento","Precision de prueba")
resultados
##                            SVM.Lineal SVM.Radial SVM.Polinómico
## Precisión de Entrenamiento  0.9916667  0.9916667      0.9916667
## Precision de prueba         0.9666667  0.9333333      0.9666667
##                            Árbol.de.decisión Redes.Neuronales
## Precisión de Entrenamiento         0.9666667        0.9666667
## Precision de prueba                0.9333333        0.9666667
##                            Bosques.Aleatorios
## Precisión de Entrenamiento          1.0000000
## Precision de prueba                 0.9333333
LS0tDQp0aXRsZTogIkNBUkVUIg0KYXV0aG9yOiAiQW5kcmV1IFNhcnJldGEiDQpkYXRlOiAiMjAyNS0wMi0yMCINCm91dHB1dDogDQogIGh0bWxfZG9jdW1lbnQ6DQogICAgdG9jOiBUUlVFDQogICAgdG9jX2Zsb2F0OiBUUlVFDQogICAgY29kZV9kb3dubG9hZDogVFJVRQ0KICAgIHRoZW1lOiAidW5pdGVkIg0KICAgIGhpZ2hsaWdodDogImVzcHJlc3NvIg0KLS0tDQoNCiANCiFbXShEOi9UZWMvU2V4dG8gU2VtZXN0cmUvSUEgY29uY2VudHJhY2lvbi9Sc3R1ZGlvL01vZHVsbyAyL2lyaXMuanBlZykNCg0KDQojIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKlRlb3LDrWEqKjwvc3Bhbj4NCkVsIHBhcXVldGUgY2FyZXQoKkNsYXNzaWZjaWF0aW9uIGFuZCBSZWdyZXNzaW9uIFRyYWluaW5nKikNCmVzIHVuYSBoZXJyYW1pZW50YSBwb2Rlcm9zYSBwYXJhIGxhIGltcGxlbWVudGFjacOzbiBkZSBtb2RlbG9zIGRlDQoqKk1hY2hpbmUgTGVhcm5pbmcqKi4NCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqUGFxdWV0ZXMgeSBsaWJyZXJpYXMqKjwvc3Bhbj4NCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQojaW5zdGFsbC5wYWNrYWdlcygiY2FyZXQiKQ0KbGlicmFyeShjYXJldCkjYWxnb3JpdG1vcyBkZSBhcHJlbmRpYWplIGF1dG9tw6FydGljbw0KI2luc3RhbGwucGFja2FnZXMoImRhdGFzZXRzIikNCmxpYnJhcnkoZGF0YXNldHMpICMgUEFyYSB1c2FyIGxhIGJhc2UgZGUgZGF0b3MgIklyaXMiDQpsaWJyYXJ5KGdncGxvdDIpDQojaW5zdGFsbC5wYWNrYWdlcygibGF0dGljZSIpDQpsaWJyYXJ5KGxhdHRpY2UpICNDcmVhciBiYXNlIGRlIGRhdG9zDQojaW5zdGFsbC5wYWNrYWdlcygiRGF0YUV4cGxvcmVyIikNCmxpYnJhcnkoRGF0YUV4cGxvcmVyKQ0KYGBgDQoNCiMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKkltcG9ydGFyIGJhc2UgZGUgZGF0b3MqKjwvc3Bhbj4NCmBgYHtyfQ0KZGYgPC0gZGF0YS5mcmFtZShpcmlzKQ0KYGBgDQoNCiMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKkFuw6FsaXNpcyBkZXNjcmlwdGl2byoqPC9zcGFuPg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCiMgY3JlYXRlX3JlcG9ydChkZikNCnBsb3RfaGlzdG9ncmFtKGRmKQ0KcGxvdF9taXNzaW5nKGRmKQ0KcGxvdF9jb3JyZWxhdGlvbihkZikNCg0KYGBgDQoNCioqIE5PVEE6IExhIHZhcmlhYmxlIHF1ZSBxdWVyZW1vcyBwcmVkZWNpciB0aWVuZSBxdWUgdGVuZXIgZm9ybWF0byBkZSBGQUNUT1IuKioNCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqRGl2aXNpw7NuIGRlIGRhdG9zIDgwLzIwKio8L3NwYW4+DQpgYGB7cn0NCnNldC5zZWVkKDEyMykNCnJfdHJhaW4gPC0gY3JlYXRlRGF0YVBhcnRpdGlvbihkZiRTcGVjaWVzLCBwPTAuOCwgbGlzdD1GQUxTRSkNCnRyYWluPC0gaXJpc1tyX3RyYWluLCBdDQp0ZXN0IDwtIGlyaXMgWy1yX3RyYWluLF0NCg0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTW9lbG9zKio8L3NwYW4+DQpMb3MgbWV0b2RvcyBtYXMgdXRpbGl6YWRvcyBwYXJhIGVudHJlbmFyIGFwcmVuZGl6YWplIGF1dG9tw6F0aWNvIHNvbjogDQoNCiogKipTVk0qKjogKlN1cHBvcnQgVmVjdG9yIE1hY2hpbmUqIGEgTcOhcXVpbmEgZGUgVmVjdG9yZXMgZGUgU29wb3J0ZS4gSGF5IHZhcmlvcyANCnN1YnRpcG9zOiBMaW5lYWwoc3ZtTGluZWFyKSwgUmFkaWFsKHN2bVJhZGlhbCksIFBvbGlub21pY28gKHN2bVBvbHkpLCBldGMuDQoqICoqQXJiw7NsIGRlIGRlY2lzaW9uKio6IHJwYXJ0DQoqICoqUmVkZXMgbmV1cm9uYWxlcyoqOiBubmV0DQoqICoqUmFuZG9tIEZvcmVzdCoqOiBCb3NxdWVzIEFsZWF0b3Jpb3M6IHJmDQpMYSB2YWxpZGFjacOzbiBjcnV6YWRhIChDVikgZXMgdW5hIHTDqWNuaWNhIHBhcmEgZXZhbHVhciBlbCByZW5kaW1pZW50byBkZSB1biBtb2RlbG8sIGRpdmlkaWVuZG8gbG9zIGRhdG9zIGVuIG11bHRpcGxlcyBzdWJjb25qdW50b3MsIHBlcm1pdGllbmRvIG1lZGlyIHN1IGNhcGFjaWRhZCBkZSBnZW5lcmFsaXphY2nDs24geSBldml0YXIgc29icmVhanVzdGUgKG92ZXJmaXR0aW5nKQ0KDQpMYSAqKk1hdHJpeiBkZSBjb25mdXNpw7NuKiogKCpDb25mdXNpb24gTWF0cml4KikgcGVybWl0ZSBhbmFsaXphciBxdcOpIHRhbiBiaWVuIGZ1bmNpb25hIA0KdW4gbW9kZWxvIHkgcXXDqSB0aXBwcyBkZSBlcnJvcmVzIGNvbWV0ZS4gTG8gcXVlIGhhY2UgZXMgY29tcGFyYXIgbGFzIHByZWRpY2Npb25lcyBkZWwgbW9kZWxvIGNvbiBsb3MgdmFsb3JlcyByZWFsZXMgZGUgbGEgdmFyaWFibGUgb2JqZXRpdm8uDQoNClNpIGxhIHByZWNzacOzbiBlcyBtdXkgYWx0YSBlbiBlbnRyZW5hbWllbnRvICg5NSUgLSAxMDAlKSwgcGVybyBiYWphIGVuIHBydWViYSAoNjAtIDcwJSksIGVzIHVuYSBzZcOxYWwgZGUgKipzb2JyZWFqdXN0ZSoqDQoNCiMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKk1vZGVsbyAxLiBTVk0gTGluZWFsKio8L3NwYW4+DQpgYGB7cn0NCg0KbW9kZWxvMSA8LSB0cmFpbigNCiAgU3BlY2llcyB+IC4sIA0KICBkYXRhID0gdHJhaW4sIA0KICBtZXRob2QgPSAic3ZtTGluZWFyIiwgICMgQ2FtYmlhciBzaSBkZXNlYXMgb3RybyBtw6l0b2RvDQogIHByZVByb2Nlc3MgPSBjKCJzY2FsZSIsICJjZW50ZXIiKSwNCiAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKSwgICMgU2UgY2llcnJhIGNvcnJlY3RhbWVudGUgZWwgcGFyw6ludGVzaXMNCiAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKEMgPSAxKSAgIyBTZSBzZXBhcmEgY29ycmVjdGFtZW50ZSBkZSB0cmFpbkNvbnRyb2wNCikNCg0KYGBgDQoNCiMjIyA8c3BhbiBzdHlsZT0iY29sb3I6cmVkOyI+KipQcmVkaWNjaW9uZXMqKjwvc3Bhbj4NCmBgYHtyfQ0KcmVzdWx0YWRvX3RyYWluIDwtIHByZWRpY3QobW9kZWxvMSwgdHJhaW4pDQpyZXN1bHRhZG9fdGVzdCA8LSBwcmVkaWN0KG1vZGVsbzEsIHRlc3QpDQpgYGANCg0KIyMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKk1hdHJpeiBkZSBDb25mdXNpw7NuKio8L3NwYW4+DQpgYGB7cn0NCm1jcnAxPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb190cmFpbiwgdHJhaW4kU3BlY2llcykNCm1jcnAxDQoNCm1jcnAyPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb190ZXN0LCB0ZXN0JFNwZWNpZXMpDQptY3JwMg0KYGBgDQoNCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTW9kZWxvIDIuIFNWTSBSYWRpYWwqPC9zcGFuPg0KYGBge3J9DQoNCm1vZGVsbzIgPC0gdHJhaW4oDQogIFNwZWNpZXMgfiAuLCANCiAgZGF0YSA9IHRyYWluLCANCiAgbWV0aG9kID0gInN2bVJhZGlhbCIsICAjIENhbWJpYXIgc2kgZGVzZWFzIG90cm8gbcOpdG9kbw0KICBwcmVQcm9jZXNzID0gYygic2NhbGUiLCAiY2VudGVyIiksDQogIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCksICAjIFNlIGNpZXJyYSBjb3JyZWN0YW1lbnRlIGVsIHBhcsOpbnRlc2lzDQogIHR1bmVHcmlkID0gZGF0YS5mcmFtZShzaWdtYT0xLEMgPSAxKSAgIyBTZSBzZXBhcmEgY29ycmVjdGFtZW50ZSBkZSB0cmFpbkNvbnRyb2wNCikNCg0KYGBgDQoNCiMjIyA8c3BhbiBzdHlsZT0iY29sb3I6cmVkOyI+KipQcmVkaWNjaW9uZXMqKjwvc3Bhbj4NCmBgYHtyfQ0KcmVzdWx0YWRvX3RyYWluMiA8LSBwcmVkaWN0KG1vZGVsbzIsIHRyYWluKQ0KcmVzdWx0YWRvX3Rlc3QyIDwtIHByZWRpY3QobW9kZWxvMiwgdGVzdCkNCmBgYA0KDQojIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTWF0cml6IGRlIENvbmZ1c2nDs24qKjwvc3Bhbj4NCmBgYHtyfQ0KbWNycDM8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3RyYWluMiwgdHJhaW4kU3BlY2llcykNCm1jcnAzDQoNCm1jcnA0PC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb190ZXN0MiwgdGVzdCRTcGVjaWVzKQ0KbWNycDQNCmBgYA0KDQoNCiMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKk1vZGVsbyAzLiBTVk0gUG9saW5vbWlhbCoqPC9zcGFuPg0KYGBge3J9DQoNCm1vZGVsbzMgPC0gdHJhaW4oDQogIFNwZWNpZXMgfiAuLCANCiAgZGF0YSA9IHRyYWluLCANCiAgbWV0aG9kID0gInN2bVBvbHkiLCAgIyBDYW1iaWFyIHNpIGRlc2VhcyBvdHJvIG3DqXRvZG8NCiAgcHJlUHJvY2VzcyA9IGMoInNjYWxlIiwgImNlbnRlciIpLA0KICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTApLCAgIyBTZSBjaWVycmEgY29ycmVjdGFtZW50ZSBlbCBwYXLDqW50ZXNpcw0KICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoZGVncmVlPTEsIHNjYWxlPTEsQyA9IDEpICAjIFNlIHNlcGFyYSBjb3JyZWN0YW1lbnRlIGRlIHRyYWluQ29udHJvbA0KKQ0KDQpgYGANCg0KIyMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKlByZWRpY2Npb25lcyoqPC9zcGFuPg0KYGBge3J9DQpyZXN1bHRhZG9fdHJhaW4zIDwtIHByZWRpY3QobW9kZWxvMywgdHJhaW4pDQpyZXN1bHRhZG9fdGVzdDMgPC0gcHJlZGljdChtb2RlbG8zLCB0ZXN0KQ0KYGBgDQoNCiMjIyA8c3BhbiBzdHlsZT0iY29sb3I6cmVkOyI+KipNYXRyaXogZGUgQ29uZnVzacOzbioqPC9zcGFuPg0KYGBge3J9DQptY3JwNTwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fdHJhaW4zLCB0cmFpbiRTcGVjaWVzKQ0KbWNycDUNCg0KbWNycDY8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3Rlc3QzLCB0ZXN0JFNwZWNpZXMpDQptY3JwNg0KYGBgDQoNCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTW9kZWxvIDQuIC5BcmLDs2wgZGUgRGVjaXNpw7MqKjwvc3Bhbj4NCmBgYHtyfQ0KDQptb2RlbG80IDwtIHRyYWluKA0KICBTcGVjaWVzIH4gLiwgDQogIGRhdGEgPSB0cmFpbiwgDQogIG1ldGhvZCA9ICJycGFydCIsICAjIENhbWJpYXIgc2kgZGVzZWFzIG90cm8gbcOpdG9kbw0KICBwcmVQcm9jZXNzID0gYygic2NhbGUiLCAiY2VudGVyIiksDQogIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCksICAjIFNlIGNpZXJyYSBjb3JyZWN0YW1lbnRlIGVsIHBhcsOpbnRlc2lzDQogIHR1bmVMZW5ndGg9MTAgICMgU2Ugc2VwYXJhIGNvcnJlY3RhbWVudGUgZGUgdHJhaW5Db250cm9sDQopDQoNCmBgYA0KDQojIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqUHJlZGljY2lvbmVzKio8L3NwYW4+DQpgYGB7cn0NCnJlc3VsdGFkb190cmFpbjQgPC0gcHJlZGljdChtb2RlbG80LCB0cmFpbikNCnJlc3VsdGFkb190ZXN0NDwtIHByZWRpY3QobW9kZWxvNCwgdGVzdCkNCmBgYA0KDQojIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTWF0cml6IGRlIENvbmZ1c2nDs24qKjwvc3Bhbj4NCmBgYHtyfQ0KbWNycDc8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3RyYWluNCwgdHJhaW4kU3BlY2llcykNCm1jcnA3DQoNCm1jcnA4PC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb190ZXN0NCwgdGVzdCRTcGVjaWVzKQ0KbWNycDgNCmBgYA0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTW9kZWxvIDUuIFJlZGVzIE5ldXJvbmFsZXMgKio8L3NwYW4+DQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KDQptb2RlbG81IDwtIHRyYWluKA0KICBTcGVjaWVzIH4gLiwgDQogIGRhdGEgPSB0cmFpbiwgDQogIG1ldGhvZCA9ICJubmV0IiwgICMgQ2FtYmlhciBzaSBkZXNlYXMgb3RybyBtw6l0b2RvDQogIHByZVByb2Nlc3MgPSBjKCJzY2FsZSIsICJjZW50ZXIiKSwNCiAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKSwgICMgU2UgY2llcnJhIGNvcnJlY3RhbWVudGUgZWwgcGFyw6ludGVzaXMNCiAgdHJhY2U9RkFMU0UNCikNCg0KYGBgDQoNCiMjIyA8c3BhbiBzdHlsZT0iY29sb3I6cmVkOyI+KipQcmVkaWNjaW9uZXMqPC9zcGFuPg0KYGBge3J9DQpyZXN1bHRhZG9fdHJhaW41IDwtIHByZWRpY3QobW9kZWxvNSwgdHJhaW4pDQpyZXN1bHRhZG9fdGVzdDUgPC0gcHJlZGljdChtb2RlbG81LCB0ZXN0KQ0KYGBgDQoNCiMjIyA8c3BhbiBzdHlsZT0iY29sb3I6cmVkOyI+KipNYXRyaXogZGUgQ29uZnVzacOzbioqPC9zcGFuPg0KYGBge3J9DQptY3JwOTwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fdHJhaW41LCB0cmFpbiRTcGVjaWVzKQ0KbWNycDkNCg0KbWNycDEwPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb190ZXN0NSwgdGVzdCRTcGVjaWVzKQ0KbWNycDEwDQpgYGANCg0KIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqTW9kZWxvIDYuIFJhbmRvbSBGb3Jlc3QgKio8L3NwYW4+DQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KDQptb2RlbG82IDwtIHRyYWluKA0KICBTcGVjaWVzIH4gLiwgDQogIGRhdGEgPSB0cmFpbiwgDQogIG1ldGhvZCA9ICJyZiIsICAjIENhbWJpYXIgc2kgZGVzZWFzIG90cm8gbcOpdG9kbw0KICBwcmVQcm9jZXNzID0gYygic2NhbGUiLCAiY2VudGVyIiksDQogIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCksICAjIFNlIGNpZXJyYSBjb3JyZWN0YW1lbnRlIGVsIHBhcsOpbnRlc2lzDQogIHR1bmVMZW5ndGg9MTAgICMgU2Ugc2VwYXJhIGNvcnJlY3RhbWVudGUgZGUgdHJhaW5Db250cm9sDQopDQoNCmBgYA0KDQojIyMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqUHJlZGljY2lvbmVzKio8L3NwYW4+DQpgYGB7cn0NCnJlc3VsdGFkb190cmFpbjYgPC0gcHJlZGljdChtb2RlbG82LCB0cmFpbikNCnJlc3VsdGFkb190ZXN0NiA8LSBwcmVkaWN0KG1vZGVsbzYsIHRlc3QpDQpgYGANCg0KIyMjIDxzcGFuIHN0eWxlPSJjb2xvcjpyZWQ7Ij4qKk1hdHJpeiBkZSBDb25mdXNpw7NuKio8L3NwYW4+DQpgYGB7cn0NCm1jcnAxMTwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fdHJhaW42LCB0cmFpbiRTcGVjaWVzKQ0KbWNycDExDQoNCm1jcnAxMjwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fdGVzdDYsIHRlc3QkU3BlY2llcykNCm1jcnAxMg0KYGBgDQoNCiMgPHNwYW4gc3R5bGU9ImNvbG9yOnJlZDsiPioqUmVzdWx0YWRvcyBkZSBsb3MgbW9kZWxvcyoqPC9zcGFuPg0KYGBge3J9DQpyZXN1bHRhZG9zIDwtIGRhdGEuZnJhbWUoDQogICJTVk0gTGluZWFsIiA9IGMobWNycDEkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDIkb3ZlcmFsbFsiQWNjdXJhY3kiXSksDQogIA0KICAiU1ZNIFJhZGlhbCIgPSBjKG1jcnAzJG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA0JG92ZXJhbGxbIkFjY3VyYWN5Il0pLA0KICANCiAgIlNWTSBQb2xpbsOzbWljbyIgPSBjKG1jcnA1JG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnA2JG92ZXJhbGxbIkFjY3VyYWN5Il0pLA0KICANCiAgIsOBcmJvbCBkZSBkZWNpc2nDs24iID0gYyhtY3JwNyRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwOCRvdmVyYWxsWyJBY2N1cmFjeSJdKSwNCiAgDQogICJSZWRlcyBOZXVyb25hbGVzIiA9IGMobWNycDkkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDEwJG92ZXJhbGxbIkFjY3VyYWN5Il0pLA0KICANCiAgIkJvc3F1ZXMgQWxlYXRvcmlvcyIgPSBjKG1jcnAxMSRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwMTIkb3ZlcmFsbFsiQWNjdXJhY3kiXSkNCiAgDQopDQpgYGANCg0KYGBge3J9DQpyb3duYW1lcyhyZXN1bHRhZG9zKTwtIGMoIlByZWNpc2nDs24gZGUgRW50cmVuYW1pZW50byIsIlByZWNpc2lvbiBkZSBwcnVlYmEiKQ0KcmVzdWx0YWRvcw0KYGBgDQoNCg==