NN Multi class no trehshold evaluation for DGA detection

ASAI2019 (no threshold)

Testing on complete dataset

dga_test_dataset<-read_csv("../datasets/results_1000_dga_vaclav_mc_no_thres_full_domain.csv")

Parsed with column specification:
cols(
  domain = col_character(),
  new_class_2 = col_double()
)

Binary DGA (CACIC 2018) results

caret::confusionMatrix(reference=as.factor(test_vaclav_results_binary_nn$real_class),data=as.factor(t$pred_class))

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0 1988  104
         1   12  896
                                          
               Accuracy : 0.9613          
                 95% CI : (0.9538, 0.9679)
    No Information Rate : 0.6667          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.911           
                                          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.9940          
            Specificity : 0.8960          
         Pos Pred Value : 0.9503          
         Neg Pred Value : 0.9868          
             Prevalence : 0.6667          
         Detection Rate : 0.6627          
   Detection Prevalence : 0.6973          
      Balanced Accuracy : 0.9450          
                                          
       'Positive' Class : 0               
                                          

MC NN (ASAI 2019) results

ASAI 2019 no threshold vs CACIC 2018

caret::confusionMatrix(data=as.factor(test_vaclav_results_mc_nn$pred_class),reference=as.factor(t$pred_class_2))

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0 1918   48
         1   73  961
                                         
               Accuracy : 0.9597         
                 95% CI : (0.952, 0.9664)
    No Information Rate : 0.6637         
    P-Value [Acc > NIR] : < 2e-16        
                                         
                  Kappa : 0.9102         
                                         
 Mcnemar's Test P-Value : 0.02912        
                                         
            Sensitivity : 0.9633         
            Specificity : 0.9524         
         Pos Pred Value : 0.9756         
         Neg Pred Value : 0.9294         
             Prevalence : 0.6637         
         Detection Rate : 0.6393         
   Detection Prevalence : 0.6553         
      Balanced Accuracy : 0.9579         
                                         
       'Positive' Class : 0              
                                         

RESULTS AFTER MC-NN RETRAIN with vaclav

Tested on 5000 “normal domains from FING”

RESULTS on 5000 FING domains using FING TRAINED MC-NN

DIFF between FING TRAINED NN and binary NN (CACIC 2018)

mc_nn<-test_fing_results_mc_nn %>% select(partial_domain, class) %>% unique()
names(mc_nn)[1]<-"domain"
binary_nn<-test_fing_results_mc_binary  %>% select(domain,class) %>% unique()


setdiff(mc_nn,binary_nn)

inner_join(mc_nn,binary_nn,by=c("domain"))  %>% filter(class.x!=class.y)

FING trained MC-NN vs binary (CACIC 2018)

caret::confusionMatrix(data=as.factor(mc_nn$class),reference=as.factor(binary_nn$class))

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0 2178   10
         1   21   15
                                          
               Accuracy : 0.9861          
                 95% CI : (0.9803, 0.9905)
    No Information Rate : 0.9888          
    P-Value [Acc > NIR] : 0.90117         
                                          
                  Kappa : 0.485           
                                          
 Mcnemar's Test P-Value : 0.07249         
                                          
            Sensitivity : 0.9905          
            Specificity : 0.6000          
         Pos Pred Value : 0.9954          
         Neg Pred Value : 0.4167          
             Prevalence : 0.9888          
         Detection Rate : 0.9793          
   Detection Prevalence : 0.9838          
      Balanced Accuracy : 0.7952          
                                          
       'Positive' Class : 0               
                                          

FING TRAINED MC-NN vs nothreshold-1700-vaclav MC-NN

caret::confusionMatrix(data=as.factor(test_fing_results_mc_nn$class),reference=as.factor(test_fing_results_mc_nn$mc_nn_notrehshold_class))

Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0 4821  119
         1   22   38
                                          
               Accuracy : 0.9718          
                 95% CI : (0.9668, 0.9762)
    No Information Rate : 0.9686          
    P-Value [Acc > NIR] : 0.103           
                                          
                  Kappa : 0.3387          
                                          
 Mcnemar's Test P-Value : 6.234e-16       
                                          
            Sensitivity : 0.9955          
            Specificity : 0.2420          
         Pos Pred Value : 0.9759          
         Neg Pred Value : 0.6333          
             Prevalence : 0.9686          
         Detection Rate : 0.9642          
   Detection Prevalence : 0.9880          
      Balanced Accuracy : 0.6187          
                                          
       'Positive' Class : 0