Teoría

El paquete Caret (Clasificación And REgresion Training) es un paquete integral con una amplia variedad de algoritmos para el aprendizaje automático.

Instalación de paquetes y librerías

#install.packages("caret")    
library(caret)
#install.packages("ggplot2")  #Para poder graficar con mejor diseño
library(ggplot2)
#install.packages("lattice")  #Para crear gráficos
library(lattice)
#install.packages("datasets") #Para poder usar la base de datos "Iris"
library(datasets)

Crear base de datos

df<- data.frame(iris)

Análisis exploratorio

summary(df)
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
## 
str(df)
## 'data.frame':    150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
#create_report(df)

** Nota_ La variable que queremos predecir debe terner dormato de FACTOR.**

Partir datos 80-20

set.seed(123)
entrenamiento_renglones<- createDataPartition(df$Species, p=0.8, list = FALSE)
entrenamiento<- iris[entrenamiento_renglones, ]
prueba<- iris[-entrenamiento_renglones, ]

Distintos tipos de Métodos para Modelar

Los métodos más utilizados para modelar aprendizaje automático son:

  • SVM: Support Vector Machine o Máquina Vectores de Soport. Hay varios subtipos: Lineal (svmLineal), Radial (svmRadial), Polinómico (svmPoly), etc.
  • Árbol de decisión: rpart
  • Redes Neuronal: nnet
  • Random Forest: rf

1.Modelo con el Método svmLinear

modelo1 <- train(Species ~ ., data=entrenamiento, method= "svmLinear", preProcess=c("scale", "center"), trControl = trainControl(method = "cv", number=10), tuneGrid= data.frame(C=1) #Cuando es svmLinear
                )
resultado_entrenamiento1 <- predict(modelo1, entrenamiento)
resultado_prueba1 <- predict(modelo1, prueba)

##Matriz de Confusión

mcre1 <- confusionMatrix(resultado_entrenamiento1, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp1 <- confusionMatrix(resultado_prueba1, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

2.Modelo con el Método svmRadial

modelo2 <- train(Species ~ ., data=entrenamiento, method= "svmRadial", preProcess=c("scale", "center"), trControl = trainControl(method = "cv", number=10), tuneGrid= data.frame(sigma=1, C=1) #Cambiar
                )
 
resultado_entrenamiento2 <- predict(modelo2, entrenamiento)
resultado_prueba2 <- predict(modelo2, prueba)

##Matriz de Confusión

mcre2 <- confusionMatrix(resultado_entrenamiento2, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp2 <- confusionMatrix(resultado_prueba2, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

3.Modelo con el Método svmPoly

modelo3 <- train(Species ~ ., data=entrenamiento, method= "svmPoly", preProcess=c("scale", "center"), trControl = trainControl(method = "cv", number=10), tuneGrid= data.frame(degree=1, scale=1, C=1) #Cambiar
                )
 
resultado_entrenamiento3 <- predict(modelo3, entrenamiento)
resultado_prueba3 <- predict(modelo3, prueba)

##Matriz de Confusión

mcre3 <- confusionMatrix(resultado_entrenamiento3, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp3 <- confusionMatrix(resultado_prueba3, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500

4.Modelo con el Método rpart

modelo4 <- train(Species ~ ., data=entrenamiento, method= "rpart", preProcess=c("scale", "center"), trControl = trainControl(method = "cv", number=10), tuneLength=10 #Cambiar
                )
 
resultado_entrenamiento4 <- predict(modelo4, entrenamiento)
resultado_prueba4 <- predict(modelo4, prueba)

##Matriz de Confusión

mcre4 <- confusionMatrix(resultado_entrenamiento4, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
mcrp4 <- confusionMatrix(resultado_prueba4, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

5.Modelo con el Método nnet

##Matriz de Confusión

mcre5 <- confusionMatrix(resultado_entrenamiento5, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
mcrp5 <- confusionMatrix(resultado_prueba5, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750

6.Modelo con el Método rf

modelo6 <- train(Species ~ ., data=entrenamiento, method= "rf", preProcess=c("scale", "center"), trControl = trainControl(method = "cv", number=10), tuneGrid= expand.grid(mtry=c(2,4,6)) #Cambiar
                )
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
resultado_entrenamiento6 <- predict(modelo6, entrenamiento)
resultado_prueba6 <- predict(modelo6, prueba)

##Matriz de Confusión

mcre6 <- confusionMatrix(resultado_entrenamiento6, entrenamiento$Species) #matriz de confusión del resultado del entrenamiento
mcre6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
mcrp6 <- confusionMatrix(resultado_prueba6, prueba$Species) #matriz de confusión del resultado de la prueba
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000

Resumen de resultados

resultados<- data.frame(
  "1. smvLinear"= c(mcre1$overall["Accuracy"], mcrp1$overall["Accuracy"]),
  "1. smvRadial"= c(mcre2$overall["Accuracy"], mcrp2$overall["Accuracy"])
  
)
LS0tDQp0aXRsZTogJ01hY2hpbmUgTGVhcm5pbmc6IFBhY2thZ2UgQ2FyZXQnDQphdXRob3I6ICJEYXZpZCBTw6FuY2hleiBBMDEyNzU2NTUiDQpkYXRlOiAiMi8yOC8yMDI0Ig0Kb3V0cHV0Og0KICBodG1sX2RvY3VtZW50Og0KICAgIHRvYzogeWVzDQogICAgdG9jX2Zsb2F0OiB5ZXMNCiAgICBjb2RlX2Rvd25sb2FkOiB5ZXMNCiAgICB0aGVtZTogeWV0aQ0KICBwZGZfZG9jdW1lbnQ6DQogICAgdG9jOiB5ZXMNCi0tLQ0KDQogICFbXShDOlxcVXNlcnNcXEx1aXNEXFxEb2N1bWVudHNcXENvbmNlbnRyYWNpw7NuXFxpcmlzLmpmaWYpDQoNCiAgDQojIFRlb3LDrWENCkVsIHBhcXVldGUgKkNhcmV0IChDbGFzaWZpY2FjacOzbiBBbmQgUkVncmVzaW9uIFRyYWluaW5nKSogZXMgdW4gcGFxdWV0ZSBpbnRlZ3JhbCBjb24gdW5hIGFtcGxpYSB2YXJpZWRhZCBkZSBhbGdvcml0bW9zIHBhcmEgZWwgYXByZW5kaXphamUgYXV0b23DoXRpY28uICANCg0KIyBJbnN0YWxhY2nDs24gZGUgcGFxdWV0ZXMgeSBsaWJyZXLDrWFzDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KI2luc3RhbGwucGFja2FnZXMoImNhcmV0IikgICAgDQpsaWJyYXJ5KGNhcmV0KQ0KI2luc3RhbGwucGFja2FnZXMoImdncGxvdDIiKSAgI1BhcmEgcG9kZXIgZ3JhZmljYXIgY29uIG1lam9yIGRpc2XDsW8NCmxpYnJhcnkoZ2dwbG90MikNCiNpbnN0YWxsLnBhY2thZ2VzKCJsYXR0aWNlIikgICNQYXJhIGNyZWFyIGdyw6FmaWNvcw0KbGlicmFyeShsYXR0aWNlKQ0KI2luc3RhbGwucGFja2FnZXMoImRhdGFzZXRzIikgI1BhcmEgcG9kZXIgdXNhciBsYSBiYXNlIGRlIGRhdG9zICJJcmlzIg0KbGlicmFyeShkYXRhc2V0cykNCg0KYGBgDQoNCiMgQ3JlYXIgYmFzZSBkZSBkYXRvcw0KYGBge3J9DQpkZjwtIGRhdGEuZnJhbWUoaXJpcykNCmBgYA0KDQojIEFuw6FsaXNpcyBleHBsb3JhdG9yaW8NCmBgYHtyfQ0Kc3VtbWFyeShkZikNCnN0cihkZikNCiNjcmVhdGVfcmVwb3J0KGRmKQ0KYGBgDQoNCioqIE5vdGFfIExhIHZhcmlhYmxlIHF1ZSBxdWVyZW1vcyBwcmVkZWNpciBkZWJlIHRlcm5lciBkb3JtYXRvIGRlIEZBQ1RPUi4qKg0KDQojIFBhcnRpciBkYXRvcyA4MC0yMA0KYGBge3J9DQpzZXQuc2VlZCgxMjMpDQplbnRyZW5hbWllbnRvX3Jlbmdsb25lczwtIGNyZWF0ZURhdGFQYXJ0aXRpb24oZGYkU3BlY2llcywgcD0wLjgsIGxpc3QgPSBGQUxTRSkNCmVudHJlbmFtaWVudG88LSBpcmlzW2VudHJlbmFtaWVudG9fcmVuZ2xvbmVzLCBdDQpwcnVlYmE8LSBpcmlzWy1lbnRyZW5hbWllbnRvX3Jlbmdsb25lcywgXQ0KYGBgDQoNCiMgRGlzdGludG9zIHRpcG9zIGRlIE3DqXRvZG9zIHBhcmEgTW9kZWxhcg0KTG9zIG3DqXRvZG9zIG3DoXMgdXRpbGl6YWRvcyBwYXJhIG1vZGVsYXIgYXByZW5kaXphamUgYXV0b23DoXRpY28gc29uOiAgDQoNCiogKipTVk0qKjogKlN1cHBvcnQgVmVjdG9yIE1hY2hpbmUqIG8gTcOhcXVpbmEgVmVjdG9yZXMgZGUgU29wb3J0LiBIYXkgdmFyaW9zIHN1YnRpcG9zOiBMaW5lYWwgKHN2bUxpbmVhbCksIFJhZGlhbCAoc3ZtUmFkaWFsKSwgUG9saW7Ds21pY28gKHN2bVBvbHkpLCBldGMuDQoqICoqw4FyYm9sIGRlIGRlY2lzacOzbioqOiBycGFydA0KKiAqKlJlZGVzIE5ldXJvbmFsKio6IG5uZXQNCiogKipSYW5kb20gRm9yZXN0Kio6IHJmDQoNCiMgMS5Nb2RlbG8gY29uIGVsIE3DqXRvZG8gc3ZtTGluZWFyDQpgYGB7cn0NCm1vZGVsbzEgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGE9ZW50cmVuYW1pZW50bywgbWV0aG9kPSAic3ZtTGluZWFyIiwgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsICJjZW50ZXIiKSwgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlcj0xMCksIHR1bmVHcmlkPSBkYXRhLmZyYW1lKEM9MSkgI0N1YW5kbyBlcyBzdm1MaW5lYXINCiAgICAgICAgICAgICAgICApDQpgYGANCiAgICANCmBgYHtyfQ0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG8xIDwtIHByZWRpY3QobW9kZWxvMSwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmExIDwtIHByZWRpY3QobW9kZWxvMSwgcHJ1ZWJhKQ0KYGBgDQoNCiMjTWF0cml6IGRlIENvbmZ1c2nDs24NCmBgYHtyfQ0KbWNyZTEgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvMSwgZW50cmVuYW1pZW50byRTcGVjaWVzKSAjbWF0cml6IGRlIGNvbmZ1c2nDs24gZGVsIHJlc3VsdGFkbyBkZWwgZW50cmVuYW1pZW50bw0KbWNyZTENCm1jcnAxIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhMSwgcHJ1ZWJhJFNwZWNpZXMpICNtYXRyaXogZGUgY29uZnVzacOzbiBkZWwgcmVzdWx0YWRvIGRlIGxhIHBydWViYQ0KbWNycDENCmBgYA0KDQojIDIuTW9kZWxvIGNvbiBlbCBNw6l0b2RvIHN2bVJhZGlhbA0KYGBge3J9DQptb2RlbG8yIDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIG1ldGhvZD0gInN2bVJhZGlhbCIsIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXI9MTApLCB0dW5lR3JpZD0gZGF0YS5mcmFtZShzaWdtYT0xLCBDPTEpICNDYW1iaWFyDQogICAgICAgICAgICAgICAgKQ0KIA0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG8yIDwtIHByZWRpY3QobW9kZWxvMiwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmEyIDwtIHByZWRpY3QobW9kZWxvMiwgcHJ1ZWJhKQ0KYGBgDQoNCiMjTWF0cml6IGRlIENvbmZ1c2nDs24NCmBgYHtyfQ0KbWNyZTIgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvMiwgZW50cmVuYW1pZW50byRTcGVjaWVzKSAjbWF0cml6IGRlIGNvbmZ1c2nDs24gZGVsIHJlc3VsdGFkbyBkZWwgZW50cmVuYW1pZW50bw0KbWNyZTINCm1jcnAyIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhMiwgcHJ1ZWJhJFNwZWNpZXMpICNtYXRyaXogZGUgY29uZnVzacOzbiBkZWwgcmVzdWx0YWRvIGRlIGxhIHBydWViYQ0KbWNycDINCmBgYA0KDQoNCiMgMy5Nb2RlbG8gY29uIGVsIE3DqXRvZG8gc3ZtUG9seQ0KYGBge3J9DQptb2RlbG8zIDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIG1ldGhvZD0gInN2bVBvbHkiLCBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyPTEwKSwgdHVuZUdyaWQ9IGRhdGEuZnJhbWUoZGVncmVlPTEsIHNjYWxlPTEsIEM9MSkgI0NhbWJpYXINCiAgICAgICAgICAgICAgICApDQogDQpyZXN1bHRhZG9fZW50cmVuYW1pZW50bzMgPC0gcHJlZGljdChtb2RlbG8zLCBlbnRyZW5hbWllbnRvKQ0KcmVzdWx0YWRvX3BydWViYTMgPC0gcHJlZGljdChtb2RlbG8zLCBwcnVlYmEpDQpgYGANCg0KIyNNYXRyaXogZGUgQ29uZnVzacOzbg0KYGBge3J9DQptY3JlMyA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX2VudHJlbmFtaWVudG8zLCBlbnRyZW5hbWllbnRvJFNwZWNpZXMpICNtYXRyaXogZGUgY29uZnVzacOzbiBkZWwgcmVzdWx0YWRvIGRlbCBlbnRyZW5hbWllbnRvDQptY3JlMw0KbWNycDMgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmEzLCBwcnVlYmEkU3BlY2llcykgI21hdHJpeiBkZSBjb25mdXNpw7NuIGRlbCByZXN1bHRhZG8gZGUgbGEgcHJ1ZWJhDQptY3JwMw0KYGBgDQoNCiMgNC5Nb2RlbG8gY29uIGVsIE3DqXRvZG8gcnBhcnQNCmBgYHtyfQ0KbW9kZWxvNCA8LSB0cmFpbihTcGVjaWVzIH4gLiwgZGF0YT1lbnRyZW5hbWllbnRvLCBtZXRob2Q9ICJycGFydCIsIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXI9MTApLCB0dW5lTGVuZ3RoPTEwICNDYW1iaWFyDQogICAgICAgICAgICAgICAgKQ0KIA0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG80IDwtIHByZWRpY3QobW9kZWxvNCwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmE0IDwtIHByZWRpY3QobW9kZWxvNCwgcHJ1ZWJhKQ0KYGBgDQoNCiMjTWF0cml6IGRlIENvbmZ1c2nDs24NCmBgYHtyfQ0KbWNyZTQgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvNCwgZW50cmVuYW1pZW50byRTcGVjaWVzKSAjbWF0cml6IGRlIGNvbmZ1c2nDs24gZGVsIHJlc3VsdGFkbyBkZWwgZW50cmVuYW1pZW50bw0KbWNyZTQNCm1jcnA0IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhNCwgcHJ1ZWJhJFNwZWNpZXMpICNtYXRyaXogZGUgY29uZnVzacOzbiBkZWwgcmVzdWx0YWRvIGRlIGxhIHBydWViYQ0KbWNycDQNCmBgYA0KDQojIDUuTW9kZWxvIGNvbiBlbCBNw6l0b2RvIG5uZXQNCmBgYHtyIGluY2x1ZGU9RkFMU0V9DQptb2RlbG81IDwtIHRyYWluKFNwZWNpZXMgfiAuLCBkYXRhPWVudHJlbmFtaWVudG8sIG1ldGhvZD0gIm5uZXQiLCBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyPTEwKSAjQ2FtYmlhcg0KICAgICAgICAgICAgICAgICkNCiANCnJlc3VsdGFkb19lbnRyZW5hbWllbnRvNSA8LSBwcmVkaWN0KG1vZGVsbzUsIGVudHJlbmFtaWVudG8pDQpyZXN1bHRhZG9fcHJ1ZWJhNSA8LSBwcmVkaWN0KG1vZGVsbzUsIHBydWViYSkNCmBgYA0KDQojI01hdHJpeiBkZSBDb25mdXNpw7NuDQpgYGB7cn0NCm1jcmU1IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fZW50cmVuYW1pZW50bzUsIGVudHJlbmFtaWVudG8kU3BlY2llcykgI21hdHJpeiBkZSBjb25mdXNpw7NuIGRlbCByZXN1bHRhZG8gZGVsIGVudHJlbmFtaWVudG8NCm1jcmU1DQptY3JwNSA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTUsIHBydWViYSRTcGVjaWVzKSAjbWF0cml6IGRlIGNvbmZ1c2nDs24gZGVsIHJlc3VsdGFkbyBkZSBsYSBwcnVlYmENCm1jcnA1DQpgYGANCg0KDQojIDYuTW9kZWxvIGNvbiBlbCBNw6l0b2RvIHJmDQpgYGB7cn0NCm1vZGVsbzYgPC0gdHJhaW4oU3BlY2llcyB+IC4sIGRhdGE9ZW50cmVuYW1pZW50bywgbWV0aG9kPSAicmYiLCBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyPTEwKSwgdHVuZUdyaWQ9IGV4cGFuZC5ncmlkKG10cnk9YygyLDQsNikpICNDYW1iaWFyDQogICAgICAgICAgICAgICAgKQ0KIA0KcmVzdWx0YWRvX2VudHJlbmFtaWVudG82IDwtIHByZWRpY3QobW9kZWxvNiwgZW50cmVuYW1pZW50bykNCnJlc3VsdGFkb19wcnVlYmE2IDwtIHByZWRpY3QobW9kZWxvNiwgcHJ1ZWJhKQ0KYGBgDQoNCiMjTWF0cml6IGRlIENvbmZ1c2nDs24NCmBgYHtyfQ0KbWNyZTYgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19lbnRyZW5hbWllbnRvNiwgZW50cmVuYW1pZW50byRTcGVjaWVzKSAjbWF0cml6IGRlIGNvbmZ1c2nDs24gZGVsIHJlc3VsdGFkbyBkZWwgZW50cmVuYW1pZW50bw0KbWNyZTYNCm1jcnA2IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhNiwgcHJ1ZWJhJFNwZWNpZXMpICNtYXRyaXogZGUgY29uZnVzacOzbiBkZWwgcmVzdWx0YWRvIGRlIGxhIHBydWViYQ0KbWNycDYNCg0KYGBgDQoNCiMgUmVzdW1lbiBkZSByZXN1bHRhZG9zDQpgYGB7cn0NCnJlc3VsdGFkb3M8LSBkYXRhLmZyYW1lKA0KICAiMS4gc212TGluZWFyIj0gYyhtY3JlMSRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwMSRvdmVyYWxsWyJBY2N1cmFjeSJdKSwNCiAgIjEuIHNtdlJhZGlhbCI9IGMobWNyZTIkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDIkb3ZlcmFsbFsiQWNjdXJhY3kiXSkNCiAgDQopDQpgYGANCg0K