Anomaly detection in the CERN cloud infrastructure

• Published in: EPJ Web of conferences 2021, Vol.251, p.2011
• Main Authors: Giordano, Domenico, Paltenghi, Matteo, Metaj, Stiven, Dvorak, Antonin
• Format: Article
• Language: English
• Subjects: Air flow; Algorithms; Anomalies; CERN; Cloud computing; Data analysis; Datasets; Deep learning; Infrastructure; Machine learning; Monitoring; Performance measurement; Source code
• ISSN: 2100-014X; 2101-6275; 2100-014X
• DOI: 10.1051/epjconf/202125102011

Anomaly detection in the CERN OpenStack cloud is a challenging task due to the large scale of the computing infrastructure and, consequently, the large volume of monitoring data to analyse. The current solution to spot anomalous servers in the cloud infrastructure relies on a threshold-based alarming system carefully set by the system managers on the performance metrics of each infrastructure’s component. This contribution explores fully automated, unsupervised machine learning solutions in the anomaly detection field for time series metrics, by adapting both traditional and deep learning approaches. The paper describes a novel end-to-end data analytics pipeline implemented to digest the large amount of monitoring data and to expose anomalies to the system managers. The pipeline relies solely on open-source tools and frameworks, such as Spark, Apache Airflow, Kubernetes, Grafana, Elasticsearch . In addition, an approach to build annotated datasets from the CERN cloud monitoring data is reported. Finally, a preliminary performance of a number of anomaly detection algorithms is evaluated by using the aforementioned annotated datasets.