A bot identification model and tool based on GitHub activity sequences

• Published in: The Journal of systems and software 2025-03, Vol.221, p.112287, Article 112287
• Main Authors: Chidambaram, Natarajan, Decan, Alexandre, Mens, Tom
• Format: Article
• Language: English
• Subjects: Bot identification; Classification model; Collaborative software development; GitHub
• ISSN: 0164-1212
• DOI: 10.1016/j.jss.2024.112287

Identifying whether GitHub contributors are automated bots is important for empirical research on collaborative software development practices. Multiple such bot identification approaches have been proposed in the past. In this article, we identify the limitations of these approaches and we propose a new binary classification model, called BIMBAS, to overcome these limitations. To do so, we propose a new ground-truth dataset containing 1035 bots and 1115 humans on GitHub. We train BIMBAS on a wide range of features extracted from the activity sequences of these GitHub contributors. We show that the performance of BIMBAS (in terms of precision, recall, F1 score and AUC) is comparable to state-of-the-art bot identification approaches, while being able to identify bots engaged in a wider range of activity types. We implement RABBIT, an open-source command-line bot identification tool based on BIMBAS. We demonstrate its ability to be used at scale, and show that its efficiency outperforms the state-of-the-art. •A ground-truth dataset of 2,150 manually labelled bots and humans in GitHub.•A comparison of the accuracy and efficiency of four bot detection approaches.•BIMBAS is a classification model to detect bots based on GitHub activity sequences.•RABBIT is an open source tool implementing BIMBAS to accurately and efficiently detect bots.