Mining Causality of Network Events in Log Data

• Published in: IEEE eTransactions on network and service management 2018-03, Vol.15 (1), p.53-67
• Main Authors: Kobayashi, Satoru, Otomo, Kazuki, Fukuda, Kensuke, Esaki, Hiroshi
• Format: Article
• Language: English
• Subjects: Algorithm design and analysis; Causal inference; Correlation; Data mining; Hidden Markov models; Image edge detection; Inference algorithms; log data; network management; PC algorithm; Reliability; root cause analysis
• ISSN: 1932-4537; 1932-4537
• DOI: 10.1109/TNSM.2017.2778096

Network log messages (e.g., syslog) are expected to be valuable and useful information to detect unexpected or anomalous behavior in large scale networks. However, because of the huge amount of system log data collected in daily operation, it is not easy to extract pinpoint system failures or to identify their causes. In this paper, we propose a method for extracting the pinpoint failures and identifying their causes from network syslog data. The methodology proposed in this paper relies on causal inference that reconstructs causality of network events from a set of time series of events. Causal inference can filter out accidentally correlated events, thus it outputs more plausible causal events than traditional cross-correlation-based approaches can. We apply our method to 15 months' worth of network syslog data obtained from a nationwide academic network in Japan. The proposed method significantly reduces the number of pseudo correlated events compared with the traditional methods. Also, through three case studies and comparison with trouble ticket data, we demonstrate the effectiveness of the proposed method for practical network operation.