Integrating HPC into an agile and cloud-focused environment at CERN

CERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on...

Full description

Saved in:
Bibliographic Details
Published in:EPJ Web of Conferences 2019, Vol.214, p.7025
Main Authors: Llopis, Pablo, Lindqvist, Carolina, Høimyr, Nils, van der Ster, Dan, Ganz, Philippe
Format: Article
Language:English
Subjects:
Backfill
CERN
Cloud computing
Computer simulation
Data centers
Data processing
Infrastructure
Large Hadron Collider
Quantum chromodynamics
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:CERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on MPI-enabled infrastructure. This contribution addresses the implementation of HPC infrastructure at CERN for Lattice QCD application development, as well as for different types of simulations for the accelerator and technology sector at CERN. Our approach has been to integrate the HPC facilities as far as possible with the HTC services in our data centre, and to take advantage of an agile infrastructure for updates, configuration and deployment. The HPC cluster has been orchestrated with the OpenStack Ironic component, and is hence managed with the same tools as the CERN internal OpenStack cloud. Experience and benchmarks of MPI applications across Infiniband with shared storage on CephFS is discussed, as well the setup of the SLURM scheduler for HPC jobs with a provision for backfill of HTC workloads.
ISSN:2100-014X
2101-6275
2100-014X
DOI:10.1051/epjconf/201921407025