Design and implementation of message-passing services for the Blue Gene/L supercomputer

• Published in: IBM journal of research and development 2005-03, Vol.49 (2-3), p.393-406
• Main Authors: Almasi, G., Archer, C., Castanos, J. G., Gunnels, J. A., Erway, C. C., Heidelberger, P., Martorell, X., Moreira, J. E., Pinnow, K., Ratterman, J., Steinmacher-Burow, B. D., Gropp, W., Toonen, B.
• Format: Article
• Language: English
• Subjects: Central processing units; Communication; Computer peripherals; CPUs; Design; Interfaces; Laboratories; Protocol; Random access memory; Software; Supercomputers; Systems design
• ISSN: 0018-8646; 0018-8646; 2151-8556
• DOI: 10.1147/rd.492.0393

The Blue Gene®/L (BG/L) supercomputer, with 65,536 dualprocessor compute nodes, was designed from the ground up to support efficient execution of massively parallel message-passing programs. Part of this support is an optimized implementation of the Message Passing Interface (MPI), which leverages the hardware features of BG/L. MPI for BG/L is implemented on top of a more basic message-passing infrastructure called the message layer. This message layer can be used both to implement other higher-level libraries and directly by applications. MPI and the message layer are used in the two BG/L modes of operation: the coprocessor mode and the virtual node mode. Performance measurements show that our message-passing services deliver performance close to the hardware limits of the machine. They also show that dedicating one of the processors of a node to communication functions (coprocessor mode) greatly improves the message-passing bandwidth, whereas running two processes per compute node (virtual node mode) can have a positive impact on application performance. [PUBLICATION ABSTRACT]