DNS queries results

Reading large dns data from whalebone (JSON) and sample

library(dplyr)
library(readr)
library(tidyr)
library(purrr)

library(jsonlite)
con_out <- file(tmp <- "/home/harpo/Dropbox/ongoing-work/git-repos/dns-threats/dataset/domains_passivedns-2019.04.01_sample.csv", open = "wb")

dga_results<-stream_in(file("/tmp/passivedns-2019.04.01.json"),
                       
  handler = function(df){
  df <- df %>% sample_n(1000)
  query <- df$query
  dga_domain <- df$dga$domain 
  stream_out(data.frame(dga_domain,query), con_out, pagesize = 1000)
}, pagesize = 100000)

close(con_out)
                   

Reading dns data from FING and sample

dns_queries <- read_delim("/home/harpo/passivedns.log.1.gz",delim = "|",col_names = FALSE)
dns_queries_sample<-dns_queries %>% sample_n((nrow(dns_queries)*10)/100)
write_delim(dns_queries_sample %>% select(X5),"/home/harpo/Dropbox/ongoing-work/git-repos/dns-threats/dataset/domains_fing.2019.05.17_sample.csv")

Analysis results

Analysis of FING queries using MC-NN classifier

0 for normal dns 1 for DGA 2 for tunnel

Save domains detected as DGA by MC-NN

many of the same domains detected as DGA by the MC-NN are correctly detected as normal by the Binary DGA (CACIC 2018)