A Game-based Adversarial DGA Detection Scheme using Multi-level Incremental Random Forest

• Published in: IEEE transactions on network science and engineering 2024-01, Vol.11 (1), p.1-13
• Main Authors: Nie, Lihai, Zhao, Laiping, Li, Keqiu, Shan, Xiaoyang, Qiu, Tie
• Format: Article
• Language: English
• Subjects: adversarial domain generation algorithm; Algorithms; Chatbots; Decision trees; Detectors; Domain names; Game theory; Games; incremental learning; Malicious domain name detection; Radio frequency; Reliability; Servers; Strategy; Training
• ISSN: 2327-4697; 2334-329X
• DOI: 10.1109/TNSE.2023.3308126

Security vendors can take down botnets by detecting the malicious domain names crafted by attackers. However, the adversarial Domain Generation Algorithms ( DGAs ) greatly challenge the existing domain detection schemes, in particular, adversarial DGAs can actively compromise arbitrarily specified domain detection systems by crafting the adversarial domain names. To resist adversarial DGAs, we propose a game theory-based defending strategy, which launches adversarial DGA and trains an incremental domain detector alternately. While we find the game-based strategy cannot achieve the expected detection accuracy due to two problems: the failure of incremental training and the problem of the catastrophic forgetting . To this end, we propose a multi-level incremental random forest model, which settles the above problems by splitting the leaf nodes of the decision trees and increasing the levels of the original random forest. The experimental results on the real-life dataset demonstrate the proposed detection method significantly outperforms the competing schemes when detecting adversarial DGAs (improves the detection AUC by 42%) and presents comparable performance when defending against non-adversarial DGAs.