DGA NN Model Comparison

library(dplyr)
library(readr)
library(ggplot2)

Results from evaluating new domains using cacic web service

CACIC service uses the default architecture from bitbucket repo

• Service located at http://catanuso.duckdns.org:8000/predict?domain=www.google.com

Dataset

The dataset contains 3465 domains names. These domains were obtained from 3 datasets sent by Vaclav:

1. 1500 domains names labeled as DGA by the original NN
2. 1295 domains labeled as DGA and Normal by the original NN
3. 1113 domains names labeled by the original NN as DGa and incorrectly classified as normal by Vaclav NN implementation

We merge the datasets and removed duplicated. Final dataset can be found here.

Confusion matrix

caret::confusionMatrix(as.factor(vaclav_results$class),as.factor(vaclav_results$label))

Confusion Matrix and Statistics

          Reference
Prediction  dga normal
    dga    2654     15
    normal  470    326
                                         
               Accuracy : 0.86           
                 95% CI : (0.848, 0.8714)
    No Information Rate : 0.9016         
    P-Value [Acc > NIR] : 1              
                                         
                  Kappa : 0.5053         
 Mcnemar's Test P-Value : <2e-16         
                                         
            Sensitivity : 0.8496         
            Specificity : 0.9560         
         Pos Pred Value : 0.9944         
         Neg Pred Value : 0.4095         
             Prevalence : 0.9016         
         Detection Rate : 0.7659         
   Detection Prevalence : 0.7703         
      Balanced Accuracy : 0.9028         
                                         
       'Positive' Class : dga            
                                         

FALSE NEGATIVES Only 470 domains labeled as DGA by the original NN service where not detected by the cacic web service

Interactive 3D representation of the TRUE POSITIVE (Blue) and FALSE (Orange) NEGATIVE DOMAINS

plotly::plot_ly(pca_data  , type="scatter3d", 
                x = ~PC1, y = ~PC2, z = ~PC3, color = ~label,
                colors = c('#BF382A', '#0C4B8E'), 
                opacity=0.5, marker = list(size = 2),text = ~paste(preds," ",domain))  

No scatter3d mode specifed:
  Setting the mode to markers
  Read more about this attribute -> https://plot.ly/r/reference/#scatter-mode
No scatter3d mode specifed:
  Setting the mode to markers
  Read more about this attribute -> https://plot.ly/r/reference/#scatter-mode

We can observe three clusters corresponding to the .hosting , .org and .feedback TLDs. TP and FN are observed in the three clusters. The table below shows the percent of FN for each cluster. The org TLD is the the cluster with the higher FN with a 23% followed by the hosting with 18% and feedback with 3%

char_dist<- function(tld_name){
  library(scales)
  
  charlist_FN=fn_domains %>% filter(grepl(tld_name,domain))  %>% do(charlist= unlist(sapply(.$domain, function(x) c(str_split(x,"")[1]))))
  charlist_TP=tp_domains %>% filter(grepl(tld_name,domain)) %>% do(charlist= unlist(sapply(.$domain, function(x) c(str_split(x,"")[1]))))
  
  charlist_FN=as.vector(unlist(charlist_FN %>% select(charlist)))
  charlist_TP=as.vector(unlist(charlist_TP %>% select(charlist)))
  
  tpplot<-ggplot(data.frame(charlist=charlist_TP),aes(x=charlist))+
    geom_bar(col="black",fill='white',aes(y = (..count..)/sum(..count..)))+
    scale_y_continuous(labels=percent)+ylab("Percent")+xlab("")+
    theme_bw()
  
  
  fnplot<-ggplot(data.frame(charlist=charlist_FN),aes(x=charlist))+
    geom_bar(col="black",fill='black',aes(y = (..count..)/sum(..count..)))+
    scale_y_continuous(labels=percent)+ylab("Percent")+xlab("")+
    theme_bw()
  
  gridExtra::grid.arrange(fnplot,tpplot,ncol=1)
  return(list(fnplot=fnplot,tpplot=tpplot))
}

Question: What is the difference between TP and FN in the org TLD?

The Character Frequency Histogram for TP (white) and FN (black)

Question: What is the difference between TP and FN in the hosting TLD?

The Character Frequency Histogram for TP (white) and FN (black)

org_plot<-char_dist("hosting")

Question: What is the difference between TP and FN in the feedback TLD?

The Character Frequency Histogram for TP (white) and FN (black)

org_plot<-char_dist("feedback")

CONCLUSIONS

No significant differences are observable between the FP and TP histograms for the three considered domains. A possible solution could be to include a portion of the non detected DGA into the dataset and retrain de model.

tensorboard("logs")