ASAI 2020 Results

Results for:

LSTM Woodbrigde + Dense 512 + Batch 256 + maxlen = 75 + first 5 symbols included

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0  499   66
         1   34 1214
                                          
               Accuracy : 0.9448          
                 95% CI : (0.9333, 0.9549)
    No Information Rate : 0.706           
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.8694          
                                          
 Mcnemar's Test P-Value : 0.001935        
                                          
            Sensitivity : 0.9484          
            Specificity : 0.9362          
         Pos Pred Value : 0.9728          
         Neg Pred Value : 0.8832          
              Precision : 0.9728          
                 Recall : 0.9484          
                     F1 : 0.9604          
             Prevalence : 0.7060          
         Detection Rate : 0.6696          
   Detection Prevalence : 0.6884          
      Balanced Accuracy : 0.9423          
                                          
       'Positive' Class : 1               
                                          

RF mrty=2 + maxlen=complete + first 5 symbols included

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0  473   60
         1   72 1208
                                          
               Accuracy : 0.9272          
                 95% CI : (0.9143, 0.9387)
    No Information Rate : 0.6994          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.8258          
                                          
 Mcnemar's Test P-Value : 0.3384          
                                          
            Sensitivity : 0.9527          
            Specificity : 0.8679          
         Pos Pred Value : 0.9437          
         Neg Pred Value : 0.8874          
              Precision : 0.9437          
                 Recall : 0.9527          
                     F1 : 0.9482          
             Prevalence : 0.6994          
         Detection Rate : 0.6663          
   Detection Prevalence : 0.7060          
      Balanced Accuracy : 0.9103          
                                          
       'Positive' Class : 1               
                                          

Results for:

LSTM Woodbrigde + Dense 512 + Batch 256 + maxlen = 75 + first 5 symbols remove

caret::confusionMatrix(as.factor(test_results$predicted_class),as.factor(test_results$class), positive='1', mode="everything" )

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0  466   87
         1   72 1188
                                          
               Accuracy : 0.9123          
                 95% CI : (0.8983, 0.9249)
    No Information Rate : 0.7033          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.7916          
                                          
 Mcnemar's Test P-Value : 0.2669          
                                          
            Sensitivity : 0.9318          
            Specificity : 0.8662          
         Pos Pred Value : 0.9429          
         Neg Pred Value : 0.8427          
              Precision : 0.9429          
                 Recall : 0.9318          
                     F1 : 0.9373          
             Prevalence : 0.7033          
         Detection Rate : 0.6553          
   Detection Prevalence : 0.6950          
      Balanced Accuracy : 0.8990          
                                          
       'Positive' Class : 1               
                                          

RF mrty=2 + maxlen=complete + first 5 symbols removed

confusionMatrix(as.factor(ctu13_test_vectorized$class),as.factor(ifelse(rfpreds[,1]>0.5,0,1)),positive = "1", mode="everything" )

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0  490   43
         1  140 1140
                                          
               Accuracy : 0.8991          
                 95% CI : (0.8843, 0.9126)
    No Information Rate : 0.6525          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.7691          
                                          
 Mcnemar's Test P-Value : 1.279e-12       
                                          
            Sensitivity : 0.9637          
            Specificity : 0.7778          
         Pos Pred Value : 0.8906          
         Neg Pred Value : 0.9193          
              Precision : 0.8906          
                 Recall : 0.9637          
                     F1 : 0.9257          
             Prevalence : 0.6525          
         Detection Rate : 0.6288          
   Detection Prevalence : 0.7060          
      Balanced Accuracy : 0.8707          
                                          
       'Positive' Class : 1