QLLog: A log anomaly detection method based on Q-learning algorithm

• Published in: Information processing & management 2021-05, Vol.58 (3), p.102540, Article 102540
• Main Authors: Duan, Xiaoyu, Ying, Shi, Yuan, Wanli, Cheng, Hailong, Yin, Xiang
• Format: Article
• Language: English
• Subjects: Accuracy; Algorithms; Anomalies; Construction; Data analysis; Datasets; Log anomaly detection; Machine learning; Mathematical models; Model testing; Q-learning; Reinforcement learning; Scalability; Sensors
• ISSN: 0306-4573; 1873-5371
• DOI: 10.1016/j.ipm.2021.102540

Most of the existing log anomaly detection methods suffer from scalability and numerous false positives. Besides, they cannot rank the severity level of abnormal events. This paper proposes a log anomaly detection based on Q-learning, namely QLLog, which can detect multiple types of system anomalies and rank the severity level of abnormal events. We first build a mathematical model of log anomaly detection, proving that log anomaly detection is a sequential decision problem. Second, we use the Q-learning algorithm to build the core of the anomaly detection model. This allows QLLog to automatically learn directed acyclic graph log patterns from normal execution and adjust the training model according to the reward value. Then, QLLog combines the advantages of the Q-learning algorithm and the specially designed rules to detect anomalies when log patterns deviate from the model trained from log data under normal execution. Besides, we provide a feedback mechanism and build an abnormal level table. Therefore, QLLog can adapt to new log states and log patterns. Experiments on real datasets show that the method can quickly and effectively detect system anomalies. Compared with the state of the art, QLLog can detect numerous real problems with high accuracy 95%, and its scalability outperforms other existing log-based anomaly detection methods. •For all we know, this paper is the first successful application of the Q-learning algorithm in the field of log anomaly detection and has achieved good detection results.•QLLog can detect multiple types of log anomalies to reduce the false negative rate.•QLLog provides a feedback mechanism to update the detection model and the abnormal level of abnormal logs.•We summarize the existing log anomaly detection methods, compare and analyze the advantages and disadvantages of them. The experimental result proves the superiority of QLLog.