Copernicus, a hybrid dataflow and peer-to-peer scientific computing platform for efficient large-scale ensemble sampling

• Published in: Future generation computer systems 2017-06, Vol.71, p.18-31
• Main Authors: Pouya, Iman, Pronk, Sander, Lundborg, Magnus, Lindahl, Erik
• Format: Article
• Language: English
• Subjects: Dataflow programming; Distributed computing; Job resiliency; Peer-too-peer; Scientific computing
• ISSN: 0167-739X; 1872-7115; 1872-7115
• DOI: 10.1016/j.future.2016.11.004

Compute-intensive applications have gradually changed focus from massively parallel supercomputers to capacity as a resource obtained on-demand. This is particularly true for the large-scale adoption of cloud computing and MapReduce in industry, while it has been difficult for traditional high-performance computing (HPC) usage in scientific and engineering computing to exploit this type of resources. However, with the strong trend of increasing parallelism rather than faster processors, a growing number of applications target parallelism already on the algorithm level with loosely coupled approaches based on sampling and ensembles. While these cannot trivially be formulated as MapReduce, they are highly amenable to throughput computing. There are many general and powerful frameworks, but in particular for sampling-based algorithms in scientific computing there are some clear advantages from having a platform and scheduler that are highly aware of the underlying physical problem. Here, we present how these challenges are addressed with combinations of dataflow programming, peer-to-peer techniques and peer-to-peer networks in the Copernicus platform. This allows automation of sampling-focused workflows, task generation, dependency tracking, and not least distributing these to a diverse set of compute resources ranging from supercomputers to clouds and distributed computing (across firewalls and fragile networks). Workflows are defined from modules using existing programs, which makes them reusable without programming requirements. The system achieves resiliency by handling node failures transparently with minimal loss of computing time due to checkpointing, and a single server can manage hundreds of thousands of cores e.g. for computational chemistry applications. •Hybrid dataflow and peer-to-peer computing to fully automated ensemble sampling.•The platform automatically distributes workloads and manages them resiliently•Problems are defined as workflow by reusing existing software and scripts.•Portability in networks where parts are behind firewalls.