Comparsion of bots detection software

Data description

We collcted 30 twitter accounts, where 15 are known to be bots and 15 are known to be human users. Ideally for testing purpose it would be better to have a larger sample size, but the data collection process is quite time-consuming because each account has to be identified by human. For privacy reasons, only user IDs (in integer format) are stored so their screen names won’t be shown here. To access the performance of bots detection methods, we will briefly compare their running time and misclassification rate (false positive and false negative).

tweetbotornot

Author: Michael W. Kearney (same as “rtweet”)

Description: https://github.com/mkearney/tweetbotornot

Interface: R package

load("data/users_bots_id_30.rda")
library(tweetbotornot)
system.time({
  bots.score.r <- tweetbotornot(bots.id)$prob_bot;
  users.score.r <- tweetbotornot(users.id)$prob_bot
})

   user  system elapsed 
  9.902   0.056  20.963 

tweetbotornot (fast mode)

Using normal mode we can only do 180 estimates per every 15 minutes due to Twitter’s REST API rate limits, while fast method uses only users-level data, which increases the maximum number of estimates per 15 minutes to 90,000.

system.time({
  bots.score.fast <- tweetbotornot(bots.id, fast = T)$prob_bot;
  users.score.fast <- tweetbotornot(users.id, fast = T)$prob_bot
})

   user  system elapsed 
  7.084   0.009  18.074 

Botometer

Author: Multiple, see https://botometer.iuni.iu.edu/#!/publications

Description: https://botometer.iuni.iu.edu/#!/

Interface: API, via official Python client libary

# the below script servers as a bridge between R environment and Python libary 
source("R/bom_py_client_wrapper.R")
system.time({
  bots.score.bom <- bom_score(bots.id);
  users.score.bom <- bom_score(users.id)
})

   user  system elapsed 
  3.904   0.338 285.540 

Botometer is substantially more time-consuming than tweetbotornot.

Classification result visualization

par(mfrow=c(1,3)) 
plot(users.score.r, col = "blue", ylim = c(0, 1), xlab = "", ylab = "Bots Probability", main = "bot-or-not")
points(bots.score.r, pch = 2, col = "red")

abline(h = 0.5, lty = 2)
plot(users.score.fast, col = "blue", ylim = c(0, 1), xlab = "User Index", ylab = "", main = "bot-or-not (fast mode)")

points(bots.score.fast, pch = 2, col = "red")
abline(h = 0.5, lty = 2)

plot(users.score.bom, col = "blue", ylim = c(0, 1), xlab = "", ylab = "", main = "Botometer")
points(bots.score.bom, pch = 2, col = "red")

abline(h = 0.5, lty = 2)

*Triangles stand for actual robot user, circle stand for actual human users.

Both methods return a score in range 0 to 1 to indicate the probability that one account is a bot. The score given by tweetbotornot seems to be more intuitive.

ROC curves

source("R/botcheck.R")
roc.r <- botcheck_roc(bots.score.r, users.score.r)
roc.fast <- botcheck_roc(bots.score.fast, users.score.fast)
roc.bom <- botcheck_roc(bots.score.bom, users.score.bom)
library(ggplot2)
p.roc <- ggplot() + 
  geom_path(data = roc.r, aes(x = fpr, y = tpr, color = "botornot")) +
  geom_path(data = roc.bom, aes(x = fpr, y = tpr, color = "Botometer")) +
  geom_path(data = roc.fast, aes(x = fpr, y = tpr, color = "fast")) +
  geom_segment(aes(x = 0, y = 0, xend = 1, yend = 1, color = "RandomGuess"), linetype="dotdash") +
  xlab('False Positive rate (Specificity)') +
  ylab('True Positive rate (Sensitivity)') +
  ggtitle("ROC curves for bots classification (sample size = 30)")
plot(p.roc)

We can see that tweetbotornot(normal mode) has the best AUC (area under the ROC curve) among all 3 methods, and both modes outperform Botometer at low false positive rate.