Applying Event Stream Processing to Network Online Failure Prediction

• Published in: IEEE communications magazine 2018-01, Vol.56 (1), p.166-170
• Main Authors: Duenas, Juan C., Navarro, Jose M., Parada G., Hugo A., Andion, Javier, Cuadrado, Felix
• Format: Magazinearticle
• Language: English
• Subjects: Analytics; Computer networks; Failure; Failure analysis; Forest management; Machine learning; Mathematical models; Media streaming; Online services; Predictions; Predictive models; Radio frequency
• ISSN: 0163-6804; 1558-1896
• DOI: 10.1109/MCOM.2018.1601135

Predicting failures on networks and systems is critical in order to maintain high uptime rates. Online failure prediction (OFP) techniques use machine learning and predictive analytics to generate failure models that can be applied to computer network data. These techniques can be provisioned on state-of-the-art stream processing systems, such as Spark Streaming, in order to cope with the scalability challenges from the base data. A big challenge with OFP is selecting the right information to process, as well as the appropriate features in order to achieve high accuracy in predicting failures on complex, interconnected systems. In this article we describe an OFP system built over Apache Spark that takes a repository of network management events, trains a Random Forest model, and uses this model to predict the appearance of future events in near real time. We show through our experiments the usefulness of network management events for accurate predictions, and the advantages of the proposed system in terms of predictive quality, cost, and ease of deployment.