Data description
We collcted 30 twitter accounts, where 15 are known to be bots and 15 are known to be human users. Ideally for testing purpose it would be better to have a larger sample size, but the data collection process is quite time-consuming because each account has to be identified by human. For privacy reasons, only user IDs (in integer format) are stored so their screen names won’t be shown here. To access the performance of bots detection methods, we will briefly compare their running time and misclassification rate (false positive and false negative).
Botometer
Author: Multiple, see https://botometer.iuni.iu.edu/#!/publications
Description: https://botometer.iuni.iu.edu/#!/
Interface: API, via official Python client libary
# the below script servers as a bridge between R environment and Python libary
source("R/bom_py_client_wrapper.R")
system.time({
bots.score.bom <- bom_score(bots.id);
users.score.bom <- bom_score(users.id)
})
user system elapsed
3.904 0.338 285.540
Botometer is substantially more time-consuming than tweetbotornot.
Classification result visualization
par(mfrow=c(1,3))
plot(users.score.r, col = "blue", ylim = c(0, 1), xlab = "", ylab = "Bots Probability", main = "bot-or-not")
points(bots.score.r, pch = 2, col = "red")
abline(h = 0.5, lty = 2)
plot(users.score.fast, col = "blue", ylim = c(0, 1), xlab = "User Index", ylab = "", main = "bot-or-not (fast mode)")
points(bots.score.fast, pch = 2, col = "red")
abline(h = 0.5, lty = 2)
plot(users.score.bom, col = "blue", ylim = c(0, 1), xlab = "", ylab = "", main = "Botometer")
points(bots.score.bom, pch = 2, col = "red")
abline(h = 0.5, lty = 2)

*Triangles stand for actual robot user, circle stand for actual human users.
Both methods return a score in range 0 to 1 to indicate the probability that one account is a bot. The score given by tweetbotornot seems to be more intuitive.
ROC curves
source("R/botcheck.R")
roc.r <- botcheck_roc(bots.score.r, users.score.r)
roc.fast <- botcheck_roc(bots.score.fast, users.score.fast)
roc.bom <- botcheck_roc(bots.score.bom, users.score.bom)
library(ggplot2)
p.roc <- ggplot() +
geom_path(data = roc.r, aes(x = fpr, y = tpr, color = "botornot")) +
geom_path(data = roc.bom, aes(x = fpr, y = tpr, color = "Botometer")) +
geom_path(data = roc.fast, aes(x = fpr, y = tpr, color = "fast")) +
geom_segment(aes(x = 0, y = 0, xend = 1, yend = 1, color = "RandomGuess"), linetype="dotdash") +
xlab('False Positive rate (Specificity)') +
ylab('True Positive rate (Sensitivity)') +
ggtitle("ROC curves for bots classification (sample size = 30)")
plot(p.roc)

We can see that tweetbotornot(normal mode) has the best AUC (area under the ROC curve) among all 3 methods, and both modes outperform Botometer at low false positive rate.
LS0tCnRpdGxlOiAiQ29tcGFyc2lvbiBvZiBib3RzIGRldGVjdGlvbiBzb2Z0d2FyZSIKYXV0aG9yOiBTdGV2ZW4gTGl1Cm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCiMjIyBEYXRhIGRlc2NyaXB0aW9uCgpXZSBjb2xsY3RlZCAzMCB0d2l0dGVyIGFjY291bnRzLCB3aGVyZSAxNSBhcmUga25vd24gdG8gYmUgYm90cyBhbmQgMTUgYXJlIGtub3duIHRvIGJlIGh1bWFuIHVzZXJzLiBJZGVhbGx5IGZvciB0ZXN0aW5nIHB1cnBvc2UgaXQgd291bGQgYmUgYmV0dGVyIHRvIGhhdmUgYSBsYXJnZXIgc2FtcGxlIHNpemUsIGJ1dCB0aGUgZGF0YSBjb2xsZWN0aW9uIHByb2Nlc3MgaXMgcXVpdGUgdGltZS1jb25zdW1pbmcgYmVjYXVzZSBlYWNoIGFjY291bnQgaGFzIHRvIGJlIGlkZW50aWZpZWQgYnkgaHVtYW4uIEZvciBwcml2YWN5IHJlYXNvbnMsIG9ubHkgdXNlciBJRHMgKGluIGludGVnZXIgZm9ybWF0KSBhcmUgc3RvcmVkIHNvIHRoZWlyIHNjcmVlbiBuYW1lcyB3b24ndCBiZSBzaG93biBoZXJlLiBUbyBhY2Nlc3MgdGhlIHBlcmZvcm1hbmNlIG9mIGJvdHMgZGV0ZWN0aW9uIG1ldGhvZHMsIHdlIHdpbGwgYnJpZWZseSBjb21wYXJlIHRoZWlyIHJ1bm5pbmcgdGltZSBhbmQgbWlzY2xhc3NpZmljYXRpb24gcmF0ZSAoZmFsc2UgcG9zaXRpdmUgYW5kIGZhbHNlIG5lZ2F0aXZlKS4KCiMjIyB0d2VldGJvdG9ybm90CgpBdXRob3I6IE1pY2hhZWwgVy4gS2Vhcm5leSAoc2FtZSBhcyAicnR3ZWV0IikKCkRlc2NyaXB0aW9uOiBodHRwczovL2dpdGh1Yi5jb20vbWtlYXJuZXkvdHdlZXRib3Rvcm5vdAoKSW50ZXJmYWNlOiBSIHBhY2thZ2UKCmBgYHtyfQpsb2FkKCJkYXRhL3VzZXJzX2JvdHNfaWRfMzAucmRhIikKbGlicmFyeSh0d2VldGJvdG9ybm90KQpzeXN0ZW0udGltZSh7CiAgYm90cy5zY29yZS5yIDwtIHR3ZWV0Ym90b3Jub3QoYm90cy5pZCkkcHJvYl9ib3Q7CiAgdXNlcnMuc2NvcmUuciA8LSB0d2VldGJvdG9ybm90KHVzZXJzLmlkKSRwcm9iX2JvdAp9KQpgYGAKCiMjIyMgdHdlZXRib3Rvcm5vdCAoZmFzdCBtb2RlKQoKVXNpbmcgbm9ybWFsIG1vZGUgd2UgY2FuIG9ubHkgZG8gMTgwIGVzdGltYXRlcyBwZXIgZXZlcnkgMTUgbWludXRlcyBkdWUgdG8gVHdpdHRlcuKAmXMgUkVTVCBBUEkgcmF0ZSBsaW1pdHMsIHdoaWxlIGZhc3QgbWV0aG9kIHVzZXMgb25seSB1c2Vycy1sZXZlbCBkYXRhLCB3aGljaCBpbmNyZWFzZXMgdGhlIG1heGltdW0gbnVtYmVyIG9mIGVzdGltYXRlcyBwZXIgMTUgbWludXRlcyB0byA5MCwwMDAuIAoKYGBge3J9CnN5c3RlbS50aW1lKHsKICBib3RzLnNjb3JlLmZhc3QgPC0gdHdlZXRib3Rvcm5vdChib3RzLmlkLCBmYXN0ID0gVCkkcHJvYl9ib3Q7CiAgdXNlcnMuc2NvcmUuZmFzdCA8LSB0d2VldGJvdG9ybm90KHVzZXJzLmlkLCBmYXN0ID0gVCkkcHJvYl9ib3QKfSkKYGBgCgojIyMgQm90b21ldGVyCgpBdXRob3I6IE11bHRpcGxlLCBzZWUgaHR0cHM6Ly9ib3RvbWV0ZXIuaXVuaS5pdS5lZHUvIyEvcHVibGljYXRpb25zCgpEZXNjcmlwdGlvbjogaHR0cHM6Ly9ib3RvbWV0ZXIuaXVuaS5pdS5lZHUvIyEvCgpJbnRlcmZhY2U6IEFQSSwgdmlhIG9mZmljaWFsIFB5dGhvbiBjbGllbnQgbGliYXJ5CgpgYGB7cn0KIyB0aGUgYmVsb3cgc2NyaXB0IHNlcnZlcnMgYXMgYSBicmlkZ2UgYmV0d2VlbiBSIGVudmlyb25tZW50IGFuZCBQeXRob24gbGliYXJ5IApzb3VyY2UoIlIvYm9tX3B5X2NsaWVudF93cmFwcGVyLlIiKQoKc3lzdGVtLnRpbWUoewogIGJvdHMuc2NvcmUuYm9tIDwtIGJvbV9zY29yZShib3RzLmlkKTsKICB1c2Vycy5zY29yZS5ib20gPC0gYm9tX3Njb3JlKHVzZXJzLmlkKQp9KQpgYGAKCkJvdG9tZXRlciBpcyBzdWJzdGFudGlhbGx5IG1vcmUgdGltZS1jb25zdW1pbmcgdGhhbiB0d2VldGJvdG9ybm90LiAKCgojIyMgQ2xhc3NpZmljYXRpb24gcmVzdWx0IHZpc3VhbGl6YXRpb24KCmBgYHtyfQpwYXIobWZyb3c9YygxLDMpKSAKCnBsb3QodXNlcnMuc2NvcmUuciwgY29sID0gImJsdWUiLCB5bGltID0gYygwLCAxKSwgeGxhYiA9ICIiLCB5bGFiID0gIkJvdHMgUHJvYmFiaWxpdHkiLCBtYWluID0gImJvdC1vci1ub3QiKQpwb2ludHMoYm90cy5zY29yZS5yLCBwY2ggPSAyLCBjb2wgPSAicmVkIikKYWJsaW5lKGggPSAwLjUsIGx0eSA9IDIpCgpwbG90KHVzZXJzLnNjb3JlLmZhc3QsIGNvbCA9ICJibHVlIiwgeWxpbSA9IGMoMCwgMSksIHhsYWIgPSAiVXNlciBJbmRleCIsIHlsYWIgPSAiIiwgbWFpbiA9ICJib3Qtb3Itbm90IChmYXN0IG1vZGUpIikKcG9pbnRzKGJvdHMuc2NvcmUuZmFzdCwgcGNoID0gMiwgY29sID0gInJlZCIpCmFibGluZShoID0gMC41LCBsdHkgPSAyKQoKcGxvdCh1c2Vycy5zY29yZS5ib20sIGNvbCA9ICJibHVlIiwgeWxpbSA9IGMoMCwgMSksIHhsYWIgPSAiIiwgeWxhYiA9ICIiLCBtYWluID0gIkJvdG9tZXRlciIpCnBvaW50cyhib3RzLnNjb3JlLmJvbSwgcGNoID0gMiwgY29sID0gInJlZCIpCmFibGluZShoID0gMC41LCBsdHkgPSAyKQpgYGAKKlRyaWFuZ2xlcyBzdGFuZCBmb3IgYWN0dWFsIHJvYm90IHVzZXIsIGNpcmNsZSBzdGFuZCBmb3IgYWN0dWFsIGh1bWFuIHVzZXJzLiAKCkJvdGggbWV0aG9kcyByZXR1cm4gYSBzY29yZSBpbiByYW5nZSAwIHRvIDEgdG8gaW5kaWNhdGUgdGhlIHByb2JhYmlsaXR5IHRoYXQgb25lIGFjY291bnQgaXMgYSBib3QuIFRoZSBzY29yZSBnaXZlbiBieSB0d2VldGJvdG9ybm90IHNlZW1zIHRvIGJlIG1vcmUgaW50dWl0aXZlLgoKIyMjIyBST0MgY3VydmVzCmBgYHtyfQpzb3VyY2UoIlIvYm90Y2hlY2suUiIpCnJvYy5yIDwtIGJvdGNoZWNrX3JvYyhib3RzLnNjb3JlLnIsIHVzZXJzLnNjb3JlLnIpCnJvYy5mYXN0IDwtIGJvdGNoZWNrX3JvYyhib3RzLnNjb3JlLmZhc3QsIHVzZXJzLnNjb3JlLmZhc3QpCnJvYy5ib20gPC0gYm90Y2hlY2tfcm9jKGJvdHMuc2NvcmUuYm9tLCB1c2Vycy5zY29yZS5ib20pCgpsaWJyYXJ5KGdncGxvdDIpCnAucm9jIDwtIGdncGxvdCgpICsgCiAgZ2VvbV9wYXRoKGRhdGEgPSByb2MuciwgYWVzKHggPSBmcHIsIHkgPSB0cHIsIGNvbG9yID0gImJvdG9ybm90IikpICsKICBnZW9tX3BhdGgoZGF0YSA9IHJvYy5ib20sIGFlcyh4ID0gZnByLCB5ID0gdHByLCBjb2xvciA9ICJCb3RvbWV0ZXIiKSkgKwogIGdlb21fcGF0aChkYXRhID0gcm9jLmZhc3QsIGFlcyh4ID0gZnByLCB5ID0gdHByLCBjb2xvciA9ICJmYXN0IikpICsKICBnZW9tX3NlZ21lbnQoYWVzKHggPSAwLCB5ID0gMCwgeGVuZCA9IDEsIHllbmQgPSAxLCBjb2xvciA9ICJSYW5kb21HdWVzcyIpLCBsaW5ldHlwZT0iZG90ZGFzaCIpICsKICB4bGFiKCdGYWxzZSBQb3NpdGl2ZSByYXRlIChTcGVjaWZpY2l0eSknKSArCiAgeWxhYignVHJ1ZSBQb3NpdGl2ZSByYXRlIChTZW5zaXRpdml0eSknKSArCiAgZ2d0aXRsZSgiUk9DIGN1cnZlcyBmb3IgYm90cyBjbGFzc2lmaWNhdGlvbiAoc2FtcGxlIHNpemUgPSAzMCkiKQoKcGxvdChwLnJvYykKYGBgCgpXZSBjYW4gc2VlIHRoYXQgdHdlZXRib3Rvcm5vdChub3JtYWwgbW9kZSkgaGFzIHRoZSBiZXN0IEFVQyAoYXJlYSB1bmRlciB0aGUgUk9DIGN1cnZlKSBhbW9uZyBhbGwgMyBtZXRob2RzLCBhbmQgYm90aCBtb2RlcyBvdXRwZXJmb3JtIEJvdG9tZXRlciBhdCBsb3cgZmFsc2UgcG9zaXRpdmUgcmF0ZS4=