High Performance MPI over the Slingshot Interconnect

• Published in: Journal of computer science and technology 2023-02, Vol.38 (1), p.128-145
• Main Authors: Khorassani, Kawthar Shafie, Chen, Chen-Chun, Ramesh, Bharath, Shafi, Aamir, Subramoni, Hari, Panda, Dhabaleswar K.
• Format: Article
• Language: English
• Subjects: Adaptive control; Artificial Intelligence; Central processing units; Computer Science; CPUs; Data Structures and Information Theory; Evaluation; Graphics processing units; Information Systems Applications (incl.Internet); Libraries; Message passing; Microprocessors; Nodes; Rankings; Regular Paper; Software Engineering; Supercomputers; Theory of Computation
• ISSN: 1000-9000; 1860-4749
• DOI: 10.1007/s11390-023-2907-5

The Slingshot interconnect designed by HPE/Cray is becoming more relevant in high-performance computing with its deployment on the upcoming exascale systems. In particular, it is the interconnect empowering the first exascale and highest-ranked supercomputer in the world, Frontier. It offers various features such as adaptive routing, congestion control, and isolated workloads. The deployment of newer interconnects sparks interest related to performance, scalability, and any potential bottlenecks as they are critical elements contributing to the scalability across nodes on these systems. In this paper, we delve into the challenges the Slingshot interconnect poses with current state-of-the-art MPI (message passing interface) libraries. In particular, we look at the scalability performance when using Slingshot across nodes. We present a comprehensive evaluation using various MPI and communication libraries including Cray MPICH, Open- MPI + UCX, RCCL, and MVAPICH2 on CPUs and GPUs on the Spock system, an early access cluster deployed with Slingshot-10, AMD MI100 GPUs and AMD Epyc Rome CPUs to emulate the Frontier system. We also evaluate preliminary CPU-based support of MPI libraries on the Slingshot-11 interconnect.