Teoría

El paquete caret(Classification and REgression Training) es una herramienta poderosa para la implementacion de modelos de Machine Learning

Instalar paquetes y llamar librerias

Importar librerias de datos

df <- data.frame(iris)

Reporte de la base de datos

#create_report(df)
plot_missing(df)

plot_histogram(df)

plot_correlation(df)

plot_bar(df)

NOTA: La variable que queremos predecirdebe tener formato de FATOR, que este en caracter no quire decir que esta en factor

Partir los datos 80-20

set.seed(123) 
renglones_entrenamiento <- createDataPartition(df$Species, p=0.8, list=FALSE)
dt_entrenamiento <- iris[renglones_entrenamiento, ]
prueba <- iris[-renglones_entrenamiento, ]

Tipos de modelos de aprendizaje

Los metodosmas utilizados para modelar aprendizaje automatico son: * SVM: Support vector machine o Maquina de vectores de soporte. Hay varios subtipos: Lineal(svmlinear), Radial (svmRadial), Polinomico (svmPoly), etc.

  • Arbol de decisiones: rpart
  • Redes neuronales: nnet
  • Random forest o Bosques aleatorios: rf

Validacion cruzada (cross validation, CV) es una tecnica para evaluar el rendimiento de un modelo dividiendo los datos en multiples subconjuntos permitiendo medir su capacidad de generalizacion y evitar sobreajuste (Overfitting) *Matriz de confusion** permite analizar que tan buen funcuina un modelo uu qye tipos de errores comete. Lo que hace es comparar la predicciones del modelo con los valores reales de la variable objetvo

Modelo 1

modelo1 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "svmLinear", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 tuneGird = data.frame(C=1) #Cambiar hiperparametros
                 )
resultados_entrenamiento1 <- predict(modelo1, dt_entrenamiento)
resultado_prueba1 <- predict(modelo1, prueba)

mcre1 <- confusionMatrix(resultados_entrenamiento1, dt_entrenamiento$Species)
mcre1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp1 <- confusionMatrix(resultado_prueba1, prueba$Species)
mcrp1
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500
#centrar los datos significa que el promedio de una variable se resta de los datos (quitar outliers)
#la cross validation es una tecnica para evalura el rendimiento de un modelo dividiendo los datos en multiples subconjuntos permitiendo mediro su capacidad de generalizacion y evtiar sobreajuste(overfitting)
modelo2 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "svmRadial", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 tuneGird = data.frame(sigma = 1, C=1) #Cambiar hiperparametros
                 )
resultados_entrenamiento2 <- predict(modelo2, dt_entrenamiento)
resultado_prueba2 <- predict(modelo2, prueba)

mcre2 <- confusionMatrix(resultados_entrenamiento2, dt_entrenamiento$Species)
mcre2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         0
##   virginica       0          1        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9917          
##                  95% CI : (0.9544, 0.9998)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9875          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           1.0000
## Specificity                 1.0000            1.0000           0.9875
## Pos Pred Value              1.0000            1.0000           0.9756
## Neg Pred Value              1.0000            0.9877           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3333
## Detection Prevalence        0.3333            0.3250           0.3417
## Balanced Accuracy           1.0000            0.9875           0.9938
mcrp2 <- confusionMatrix(resultado_prueba2, prueba$Species)
mcrp2
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000
modelo3 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "svmPoly", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 tuneGird = data.frame(degree=1 , scale=1, C=1) #Cambiar hiperparametros
                 )
resultados_entrenamiento3 <- predict(modelo3, dt_entrenamiento)
resultado_prueba3 <- predict(modelo3, prueba)

mcre3 <- confusionMatrix(resultados_entrenamiento3, dt_entrenamiento$Species)
mcre3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         4
##   virginica       0          0        36
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500
mcrp3 <- confusionMatrix(resultado_prueba3, prueba$Species)
mcrp3
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000
modelo4 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "rpart", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 tuneLength = 10 #Cambiar hiperparametros
                 )
resultados_entrenamiento4 <- predict(modelo4, dt_entrenamiento)
resultado_prueba4 <- predict(modelo1, prueba)

mcre4 <- confusionMatrix(resultados_entrenamiento4, dt_entrenamiento$Species)
mcre4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         39         3
##   virginica       0          1        37
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9750           0.9250
## Specificity                 1.0000            0.9625           0.9875
## Pos Pred Value              1.0000            0.9286           0.9737
## Neg Pred Value              1.0000            0.9872           0.9634
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3250           0.3083
## Detection Prevalence        0.3333            0.3500           0.3167
## Balanced Accuracy           1.0000            0.9688           0.9563
mcrp4 <- confusionMatrix(resultado_prueba4, prueba$Species)
mcrp4
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         1
##   virginica       0          0         9
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.9000
## Specificity                 1.0000            0.9500           1.0000
## Pos Pred Value              1.0000            0.9091           1.0000
## Neg Pred Value              1.0000            1.0000           0.9524
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3000
## Detection Prevalence        0.3333            0.3667           0.3000
## Balanced Accuracy           1.0000            0.9750           0.9500
modelo5 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "nnet", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 trace = FALSE #Cambiar hiperparametros
                 )
resultados_entrenamiento5 <- predict(modelo5, dt_entrenamiento)
resultado_prueba5 <- predict(modelo5, prueba)

mcre5 <- confusionMatrix(resultados_entrenamiento5, dt_entrenamiento$Species)
mcre5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         36         0
##   virginica       0          4        40
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.9169, 0.9908)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
mcrp5 <- confusionMatrix(resultado_prueba5, prueba$Species)
mcrp5
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0          9         0
##   virginica       0          1        10
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9667          
##                  95% CI : (0.8278, 0.9992)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 2.963e-13       
##                                           
##                   Kappa : 0.95            
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            0.9000           1.0000
## Specificity                 1.0000            1.0000           0.9500
## Pos Pred Value              1.0000            1.0000           0.9091
## Neg Pred Value              1.0000            0.9524           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3000           0.3333
## Detection Prevalence        0.3333            0.3000           0.3667
## Balanced Accuracy           1.0000            0.9500           0.9750
modelo6 <- train(Species~ ., data = dt_entrenamiento, 
                 method = "rf", #cambiar 
                 preProcess=c("scale", "center"), 
                 trControl = trainControl(method = "cv", number = 10), 
                 tuneGrid = expand.grid(mtry = c(2,4,6)),#Cambiar hiperparametros
                 trace=FALSE 
                 )
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
resultados_entrenamiento6 <- predict(modelo6, dt_entrenamiento)
resultado_prueba6 <- predict(modelo6, prueba)

mcre6 <- confusionMatrix(resultados_entrenamiento6, dt_entrenamiento$Species)
mcre6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         40          0         0
##   versicolor      0         40         0
##   virginica       0          0        40
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9697, 1)
##     No Information Rate : 0.3333     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##                                      
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           1.0000
## Specificity                 1.0000            1.0000           1.0000
## Pos Pred Value              1.0000            1.0000           1.0000
## Neg Pred Value              1.0000            1.0000           1.0000
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.3333
## Detection Prevalence        0.3333            0.3333           0.3333
## Balanced Accuracy           1.0000            1.0000           1.0000
mcrp6 <- confusionMatrix(resultado_prueba6, prueba$Species)
mcrp6
## Confusion Matrix and Statistics
## 
##             Reference
## Prediction   setosa versicolor virginica
##   setosa         10          0         0
##   versicolor      0         10         2
##   virginica       0          0         8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9333          
##                  95% CI : (0.7793, 0.9918)
##     No Information Rate : 0.3333          
##     P-Value [Acc > NIR] : 8.747e-12       
##                                           
##                   Kappa : 0.9             
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: setosa Class: versicolor Class: virginica
## Sensitivity                 1.0000            1.0000           0.8000
## Specificity                 1.0000            0.9000           1.0000
## Pos Pred Value              1.0000            0.8333           1.0000
## Neg Pred Value              1.0000            1.0000           0.9091
## Prevalence                  0.3333            0.3333           0.3333
## Detection Rate              0.3333            0.3333           0.2667
## Detection Prevalence        0.3333            0.4000           0.2667
## Balanced Accuracy           1.0000            0.9500           0.9000
resultados <- data.frame( 
  "SVM Lineal" = c(mcre1$overall["Accuracy"], mcrp1$overall["Accuracy"]),
  "svm Radial" = c(mcre2$overall["Accuracy"], mcrp2$overall["Accuracy"]),
  "svm Polinomico" = c(mcre3$overall["Accuracy"], mcrp3$overall["Accuracy"]),
  "Arbol de decisiones" = c(mcre4$overall["Accuracy"], mcrp4$overall["Accuracy"]),
  "Red neuronal" = c(mcre5$overall["Accuracy"], mcrp5$overall["Accuracy"]),
  "Random Forest" = c(mcre6$overall["Accuracy"], mcrp6$overall["Accuracy"])
)
rownames(resultados) <- c("Precision de entrenamiento", "Precision de prueba")
resultados
##                            SVM.Lineal svm.Radial svm.Polinomico
## Precision de entrenamiento  0.9916667  0.9916667      0.9666667
## Precision de prueba         0.9666667  0.9333333      0.9333333
##                            Arbol.de.decisiones Red.neuronal Random.Forest
## Precision de entrenamiento           0.9666667    0.9666667     1.0000000
## Precision de prueba                  0.9666667    0.9666667     0.9333333
LS0tCnRpdGxlOiAiYWN0aXZpZGFkIDMiCm91dHB1dDogCiAgaHRtbF9kb2N1bWVudDoKICAgIHRvYzogVFJVRQogICAgdG9jX2Zsb2F0OiBUUlVFCiAgICBjb2RlX2Rvd25sb2FkOiBUUlVFCiAgICB0aGVtZTogInVuaXRlZCIKICAgIGhpZ2hsaWdodDogImVzcHJlc3NvIgpkYXRlOiAiMjAyNS0wMi0yMCIKLS0tCgohW10oL1VzZXJzL2VtaWxpYW5vL0Rvd25sb2Fkcy9saXJpby5qcGVnKQoKIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPlRlb3LDrWE8L3NwYW4+CkVsIHBhcXVldGUgY2FyZXQoKkNsYXNzaWZpY2F0aW9uIGFuZCBSRWdyZXNzaW9uIFRyYWluaW5nKikgZXMgdW5hIGhlcnJhbWllbnRhIHBvZGVyb3NhIHBhcmEgbGEgaW1wbGVtZW50YWNpb24gZGUgbW9kZWxvcyBkZSAqKk1hY2hpbmUgTGVhcm5pbmcqKgoKIyMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5JbnN0YWxhciBwYXF1ZXRlcyB5IGxsYW1hciBsaWJyZXJpYXM8L3NwYW4+CmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQojaW5zdGFsbC5wYWNrYWdlcygiY2FyZXQiKSAjQWxnb3RpbXlvcyBkZSBhcHJlbmRpemFqZSBhdXRvbWF0aWNvCmxpYnJhcnkoY2FyZXQpCiNpbnN0YWxsLnBhY2thZ2VzKCJkYXRhc2V0cyIpICNwYXJhIGxhIGJhc2UgZGUgZGF0b3MgZGUgSXJpcwpsaWJyYXJ5KGRhdGFzZXRzKQpsaWJyYXJ5KGdncGxvdDIpCiNpbnN0YWxsLnBhY2thZ2VzKCJsYXR0aWNlIikgI2NyZWFyIGdyYWZpY29zCmxpYnJhcnkobGF0dGljZSkKI2luc3RhbGwucGFja2FnZXMoIkRhdGFFeHBsb3JlciIpICNBbmFsaXNpcyBkZXNjcmlwdGl2bwpsaWJyYXJ5KERhdGFFeHBsb3JlcikKYGBgCgojIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPkltcG9ydGFyIGxpYnJlcmlhcyBkZSBkYXRvczwvc3Bhbj4KYGBge3J9CmRmIDwtIGRhdGEuZnJhbWUoaXJpcykKYGBgCgojIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPlJlcG9ydGUgZGUgbGEgYmFzZSBkZSBkYXRvczwvc3Bhbj4KYGBge3J9CiNjcmVhdGVfcmVwb3J0KGRmKQpwbG90X21pc3NpbmcoZGYpCnBsb3RfaGlzdG9ncmFtKGRmKQpwbG90X2NvcnJlbGF0aW9uKGRmKQpwbG90X2JhcihkZikKYGBgCioqTk9UQTogTGEgdmFyaWFibGUgcXVlIHF1ZXJlbW9zIHByZWRlY2lyZGViZSB0ZW5lciBmb3JtYXRvIGRlIEZBVE9SLCBxdWUgZXN0ZSBlbiBjYXJhY3RlciBubyBxdWlyZSBkZWNpciBxdWUgZXN0YSBlbiBmYWN0b3IqKgoKIyMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5QYXJ0aXIgbG9zIGRhdG9zIDgwLTIwPC9zcGFuPgpgYGB7cn0Kc2V0LnNlZWQoMTIzKSAKcmVuZ2xvbmVzX2VudHJlbmFtaWVudG8gPC0gY3JlYXRlRGF0YVBhcnRpdGlvbihkZiRTcGVjaWVzLCBwPTAuOCwgbGlzdD1GQUxTRSkKZHRfZW50cmVuYW1pZW50byA8LSBpcmlzW3Jlbmdsb25lc19lbnRyZW5hbWllbnRvLCBdCnBydWViYSA8LSBpcmlzWy1yZW5nbG9uZXNfZW50cmVuYW1pZW50bywgXQpgYGAKCiMgPHNwYW4gc3R5bGU9ImNvbG9yOiBwdXJwbGU7Ij5UaXBvcyBkZSBtb2RlbG9zIGRlIGFwcmVuZGl6YWplPC9zcGFuPgpMb3MgbWV0b2Rvc21hcyB1dGlsaXphZG9zIHBhcmEgbW9kZWxhciBhcHJlbmRpemFqZSBhdXRvbWF0aWNvIHNvbjoKKiAqKlNWTSoqOiAqU3VwcG9ydCB2ZWN0b3IgbWFjaGluZSogbyBNYXF1aW5hIGRlIHZlY3RvcmVzIGRlIHNvcG9ydGUuIEhheSB2YXJpb3Mgc3VidGlwb3M6IExpbmVhbChzdm1saW5lYXIpLCBSYWRpYWwgKHN2bVJhZGlhbCksIFBvbGlub21pY28gKHN2bVBvbHkpLCBldGMuCgoqICoqQXJib2wgZGUgZGVjaXNpb25lcyoqOiBycGFydAoqICoqUmVkZXMgbmV1cm9uYWxlcyoqOiBubmV0CiogKipSYW5kb20gZm9yZXN0KiogbyBCb3NxdWVzIGFsZWF0b3Jpb3M6IHJmCgoqKlZhbGlkYWNpb24gY3J1emFkYSoqICgqKmNyb3NzIHZhbGlkYXRpb24qKiwgQ1YpIGVzIHVuYSB0ZWNuaWNhIHBhcmEgZXZhbHVhciBlbCByZW5kaW1pZW50byBkZSB1biBtb2RlbG8gZGl2aWRpZW5kbyBsb3MgZGF0b3MgZW4gbXVsdGlwbGVzIHN1YmNvbmp1bnRvcyBwZXJtaXRpZW5kbyBtZWRpciBzdSBjYXBhY2lkYWQgZGUgZ2VuZXJhbGl6YWNpb24geSBldml0YXIgc29icmVhanVzdGUgKCoqT3ZlcmZpdHRpbmcqKikKKk1hdHJpeiBkZSBjb25mdXNpb24qKiBwZXJtaXRlIGFuYWxpemFyIHF1ZSB0YW4gYnVlbiBmdW5jdWluYSB1biBtb2RlbG8gdXUgcXllIHRpcG9zIGRlIGVycm9yZXMgY29tZXRlLiBMbyBxdWUgaGFjZSBlcyBjb21wYXJhciBsYSBwcmVkaWNjaW9uZXMgZGVsIG1vZGVsbyBjb24gbG9zIHZhbG9yZXMgcmVhbGVzIGRlIGxhIHZhcmlhYmxlIG9iamV0dm8KCgojIyA8c3BhbiBzdHlsZT0iY29sb3I6IHB1cnBsZTsiPk1vZGVsbyAxPC9zcGFuPgpgYGB7cn0KbW9kZWxvMSA8LSB0cmFpbihTcGVjaWVzfiAuLCBkYXRhID0gZHRfZW50cmVuYW1pZW50bywgCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bUxpbmVhciIsICNjYW1iaWFyIAogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksIAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCksIAogICAgICAgICAgICAgICAgIHR1bmVHaXJkID0gZGF0YS5mcmFtZShDPTEpICNDYW1iaWFyIGhpcGVycGFyYW1ldHJvcwogICAgICAgICAgICAgICAgICkKcmVzdWx0YWRvc19lbnRyZW5hbWllbnRvMSA8LSBwcmVkaWN0KG1vZGVsbzEsIGR0X2VudHJlbmFtaWVudG8pCnJlc3VsdGFkb19wcnVlYmExIDwtIHByZWRpY3QobW9kZWxvMSwgcHJ1ZWJhKQoKbWNyZTEgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb3NfZW50cmVuYW1pZW50bzEsIGR0X2VudHJlbmFtaWVudG8kU3BlY2llcykKbWNyZTEKCm1jcnAxIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhMSwgcHJ1ZWJhJFNwZWNpZXMpCm1jcnAxCgojY2VudHJhciBsb3MgZGF0b3Mgc2lnbmlmaWNhIHF1ZSBlbCBwcm9tZWRpbyBkZSB1bmEgdmFyaWFibGUgc2UgcmVzdGEgZGUgbG9zIGRhdG9zIChxdWl0YXIgb3V0bGllcnMpCiNsYSBjcm9zcyB2YWxpZGF0aW9uIGVzIHVuYSB0ZWNuaWNhIHBhcmEgZXZhbHVyYSBlbCByZW5kaW1pZW50byBkZSB1biBtb2RlbG8gZGl2aWRpZW5kbyBsb3MgZGF0b3MgZW4gbXVsdGlwbGVzIHN1YmNvbmp1bnRvcyBwZXJtaXRpZW5kbyBtZWRpcm8gc3UgY2FwYWNpZGFkIGRlIGdlbmVyYWxpemFjaW9uIHkgZXZ0aWFyIHNvYnJlYWp1c3RlKG92ZXJmaXR0aW5nKQoKYGBgCmBgYHtyfQptb2RlbG8yIDwtIHRyYWluKFNwZWNpZXN+IC4sIGRhdGEgPSBkdF9lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAic3ZtUmFkaWFsIiwgI2NhbWJpYXIgCiAgICAgICAgICAgICAgICAgcHJlUHJvY2Vzcz1jKCJzY2FsZSIsICJjZW50ZXIiKSwgCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKSwgCiAgICAgICAgICAgICAgICAgdHVuZUdpcmQgPSBkYXRhLmZyYW1lKHNpZ21hID0gMSwgQz0xKSAjQ2FtYmlhciBoaXBlcnBhcmFtZXRyb3MKICAgICAgICAgICAgICAgICApCnJlc3VsdGFkb3NfZW50cmVuYW1pZW50bzIgPC0gcHJlZGljdChtb2RlbG8yLCBkdF9lbnRyZW5hbWllbnRvKQpyZXN1bHRhZG9fcHJ1ZWJhMiA8LSBwcmVkaWN0KG1vZGVsbzIsIHBydWViYSkKCm1jcmUyIDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9zX2VudHJlbmFtaWVudG8yLCBkdF9lbnRyZW5hbWllbnRvJFNwZWNpZXMpCm1jcmUyCgptY3JwMiA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvX3BydWViYTIsIHBydWViYSRTcGVjaWVzKQptY3JwMgpgYGAKCmBgYHtyfQptb2RlbG8zIDwtIHRyYWluKFNwZWNpZXN+IC4sIGRhdGEgPSBkdF9lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAic3ZtUG9seSIsICNjYW1iaWFyIAogICAgICAgICAgICAgICAgIHByZVByb2Nlc3M9Yygic2NhbGUiLCAiY2VudGVyIiksIAogICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCksIAogICAgICAgICAgICAgICAgIHR1bmVHaXJkID0gZGF0YS5mcmFtZShkZWdyZWU9MSAsIHNjYWxlPTEsIEM9MSkgI0NhbWJpYXIgaGlwZXJwYXJhbWV0cm9zCiAgICAgICAgICAgICAgICAgKQpyZXN1bHRhZG9zX2VudHJlbmFtaWVudG8zIDwtIHByZWRpY3QobW9kZWxvMywgZHRfZW50cmVuYW1pZW50bykKcmVzdWx0YWRvX3BydWViYTMgPC0gcHJlZGljdChtb2RlbG8zLCBwcnVlYmEpCgptY3JlMyA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvc19lbnRyZW5hbWllbnRvMywgZHRfZW50cmVuYW1pZW50byRTcGVjaWVzKQptY3JlMwoKbWNycDMgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmEzLCBwcnVlYmEkU3BlY2llcykKbWNycDMKYGBgCmBgYHtyfQptb2RlbG80IDwtIHRyYWluKFNwZWNpZXN+IC4sIGRhdGEgPSBkdF9lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAicnBhcnQiLCAjY2FtYmlhciAKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCAKICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTApLCAKICAgICAgICAgICAgICAgICB0dW5lTGVuZ3RoID0gMTAgI0NhbWJpYXIgaGlwZXJwYXJhbWV0cm9zCiAgICAgICAgICAgICAgICAgKQpyZXN1bHRhZG9zX2VudHJlbmFtaWVudG80IDwtIHByZWRpY3QobW9kZWxvNCwgZHRfZW50cmVuYW1pZW50bykKcmVzdWx0YWRvX3BydWViYTQgPC0gcHJlZGljdChtb2RlbG8xLCBwcnVlYmEpCgptY3JlNCA8LSBjb25mdXNpb25NYXRyaXgocmVzdWx0YWRvc19lbnRyZW5hbWllbnRvNCwgZHRfZW50cmVuYW1pZW50byRTcGVjaWVzKQptY3JlNAoKbWNycDQgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb19wcnVlYmE0LCBwcnVlYmEkU3BlY2llcykKbWNycDQKYGBgCgpgYGB7cn0KbW9kZWxvNSA8LSB0cmFpbihTcGVjaWVzfiAuLCBkYXRhID0gZHRfZW50cmVuYW1pZW50bywgCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gIm5uZXQiLCAjY2FtYmlhciAKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCAKICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTApLCAKICAgICAgICAgICAgICAgICB0cmFjZSA9IEZBTFNFICNDYW1iaWFyIGhpcGVycGFyYW1ldHJvcwogICAgICAgICAgICAgICAgICkKcmVzdWx0YWRvc19lbnRyZW5hbWllbnRvNSA8LSBwcmVkaWN0KG1vZGVsbzUsIGR0X2VudHJlbmFtaWVudG8pCnJlc3VsdGFkb19wcnVlYmE1IDwtIHByZWRpY3QobW9kZWxvNSwgcHJ1ZWJhKQoKbWNyZTUgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb3NfZW50cmVuYW1pZW50bzUsIGR0X2VudHJlbmFtaWVudG8kU3BlY2llcykKbWNyZTUKCm1jcnA1IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhNSwgcHJ1ZWJhJFNwZWNpZXMpCm1jcnA1CmBgYAoKCmBgYHtyfQptb2RlbG82IDwtIHRyYWluKFNwZWNpZXN+IC4sIGRhdGEgPSBkdF9lbnRyZW5hbWllbnRvLCAKICAgICAgICAgICAgICAgICBtZXRob2QgPSAicmYiLCAjY2FtYmlhciAKICAgICAgICAgICAgICAgICBwcmVQcm9jZXNzPWMoInNjYWxlIiwgImNlbnRlciIpLCAKICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTApLCAKICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGV4cGFuZC5ncmlkKG10cnkgPSBjKDIsNCw2KSksI0NhbWJpYXIgaGlwZXJwYXJhbWV0cm9zCiAgICAgICAgICAgICAgICAgdHJhY2U9RkFMU0UgCiAgICAgICAgICAgICAgICAgKQoKcmVzdWx0YWRvc19lbnRyZW5hbWllbnRvNiA8LSBwcmVkaWN0KG1vZGVsbzYsIGR0X2VudHJlbmFtaWVudG8pCnJlc3VsdGFkb19wcnVlYmE2IDwtIHByZWRpY3QobW9kZWxvNiwgcHJ1ZWJhKQoKbWNyZTYgPC0gY29uZnVzaW9uTWF0cml4KHJlc3VsdGFkb3NfZW50cmVuYW1pZW50bzYsIGR0X2VudHJlbmFtaWVudG8kU3BlY2llcykKbWNyZTYKCm1jcnA2IDwtIGNvbmZ1c2lvbk1hdHJpeChyZXN1bHRhZG9fcHJ1ZWJhNiwgcHJ1ZWJhJFNwZWNpZXMpCm1jcnA2CmBgYApgYGB7cn0KcmVzdWx0YWRvcyA8LSBkYXRhLmZyYW1lKCAKICAiU1ZNIExpbmVhbCIgPSBjKG1jcmUxJG92ZXJhbGxbIkFjY3VyYWN5Il0sIG1jcnAxJG92ZXJhbGxbIkFjY3VyYWN5Il0pLAogICJzdm0gUmFkaWFsIiA9IGMobWNyZTIkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDIkb3ZlcmFsbFsiQWNjdXJhY3kiXSksCiAgInN2bSBQb2xpbm9taWNvIiA9IGMobWNyZTMkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDMkb3ZlcmFsbFsiQWNjdXJhY3kiXSksCiAgIkFyYm9sIGRlIGRlY2lzaW9uZXMiID0gYyhtY3JlNCRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNCRvdmVyYWxsWyJBY2N1cmFjeSJdKSwKICAiUmVkIG5ldXJvbmFsIiA9IGMobWNyZTUkb3ZlcmFsbFsiQWNjdXJhY3kiXSwgbWNycDUkb3ZlcmFsbFsiQWNjdXJhY3kiXSksCiAgIlJhbmRvbSBGb3Jlc3QiID0gYyhtY3JlNiRvdmVyYWxsWyJBY2N1cmFjeSJdLCBtY3JwNiRvdmVyYWxsWyJBY2N1cmFjeSJdKQopCnJvd25hbWVzKHJlc3VsdGFkb3MpIDwtIGMoIlByZWNpc2lvbiBkZSBlbnRyZW5hbWllbnRvIiwgIlByZWNpc2lvbiBkZSBwcnVlYmEiKQpyZXN1bHRhZG9zCmBgYAoK