Evaluation of CN-CNN on full domains

library(dplyr)
library(readr)
library(ggplot2)
library(plotly)
library(keras)
library(abind)
library(jsonlite)
source("../../dga-wb-r/preprocess.R")

dga_dga_vaclav <- read_csv("../datasets/results_1000_dga_vaclav_mc_no_thres_full_domain.csv")
dga_normal_vaclav <- read_csv("../datasets/results_2000_normal_valclav_mc_no_thres_full_domain.csv")
dga_dga_vaclav$label<- 1
dga_normal_vaclav$label<-0
#dga_results<-stream_in(textConnection(readLines("../datasets/results_full_dga_binary.json",n=1000)))
#normal_results<-stream_in(textConnection(readLines("../datasets/results_full_normal_binary.json",n=2000)))
#write_csv(dga_results,"../datasets/results_full_dga_binary.csv")
#write_csv(normal_results,"../datasets/results_full_normal_binary.csv")
dga_results<-stream_in(textConnection(readLines("../datasets/results_full_dga_binary.json",n=1000)))
normal_results<-read_csv("../datasets/results_full_normal_binary.csv")
dga_results<-read_csv("../datasets/results_full_dga_binary.csv")

PCA VIZ for DGA and Normal full domains

EVALUATION of MC-CNN on 1000 full domains with tunneling top-level domains changed

Dataset contain 3 classes. Normal, DGA and TUNNELING

sample_test_mc_cnn_fulldomains <- read_csv("../datasets/sample_test_mc_cnn-fulldomain.csv")
results_test_mc_cnn_fulldomains <- read_csv("../datasets/results_test_mc_cnn-fulldomains.csv")

Confusion Matrix

caret::confusionMatrix(reference=as.factor(sample_test_mc_cnn_fulldomains$class),data=as.factor(results_test_mc_cnn_fulldomains$class))

longitud de objeto mayor no es múltiplo de la longitud de uno menorLevels are not in the same order for reference and data. Refactoring data to match.

Confusion Matrix and Statistics

          Reference
Prediction   0   1   2 class
     0     198   1  39     1
     1       2 299 276     0
     2       0   0 185     0
     class   0   0   0     0

Overall Statistics
                                          
               Accuracy : 0.6813          
                 95% CI : (0.6515, 0.7101)
    No Information Rate : 0.4995          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5363          
                                          
 Mcnemar's Test P-Value : NA              

Statistics by Class:

                     Class: 0 Class: 1 Class: 2 Class: class
Sensitivity            0.9900   0.9967   0.3700     0.000000
Specificity            0.9488   0.6034   1.0000     1.000000
Pos Pred Value         0.8285   0.5182   1.0000          NaN
Neg Pred Value         0.9974   0.9976   0.6140     0.999001
Prevalence             0.1998   0.2997   0.4995     0.000999
Detection Rate         0.1978   0.2987   0.1848     0.000000
Detection Prevalence   0.2388   0.5764   0.1848     0.000000
Balanced Accuracy      0.9694   0.8000   0.6850     0.500000

Analysis of the resuls of the MC-CNN considering the domain length

Tunneling domains uncorrectly detected as DGA

cbind(sample_test_mc_cnn_fulldomains, pred=results_test_mc_cnn_fulldomains$class) %>% filter(class==2 & pred==1)