Efficient on-the-fly Web bot detection

• Published in: Knowledge-based systems 2021-07, Vol.223, p.107074, Article 107074
• Main Authors: Suchacka, Grażyna, Cabri, Alberto, Rovetta, Stefano, Masulli, Francesco
• Format: Article
• Language: English
• Subjects: Classification; Early decision; Intelligent agents; Internet robot; Machine learning; Neural network; Neural networks; Performance evaluation; Probabilistic analysis; Real-time bot detection; Sequential analysis; Servers; Traffic information; Web bot
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2021.107074

A large fraction of traffic on present-day Web servers is generated by bots — intelligent agents able to traverse the Web and execute various advanced tasks. Since bots’ activity may raise concerns about server security and performance, many studies have investigated traffic features discriminating bots from human visitors and developed methods for automated traffic classification. Very few previous works, however, aim at identifying bots on-the-fly, trying to classify active sessions as early as possible. This paper proposes a novel method for binary classification of streams of Web server requests in order to label each active session as “bot” or “human”. A machine learning approach has been developed to discover traffic patterns from historical usage data. The model, built on a neural network, is used to classify each incoming HTTP request and a sequential probabilistic analysis approach is then applied to capture relationships between subsequent HTTP requests in an ongoing session to assess the likelihood of the session being generated by a bot or a human, as soon as possible. A performance evaluation study with real server traffic data confirmed the effectiveness of the proposed classifier in discriminating bots from humans at early stages of their visits, leaving very few of them undecided, with very low number of false positives.