A lightweight RDMA connection protocol based on post-hoc confirmation

• Published in: Journal of parallel and distributed computing 2025-01, Vol.195, p.104991, Article 104991
• Main Authors: Wu, Ke, Dong, Dezun, Xu, Weixia
• Format: Article
• Language: English
• Subjects: Connection protocol; High-performance computing; RDMA networks
• ISSN: 0743-7315
• DOI: 10.1016/j.jpdc.2024.104991

With the increasing scale and complexity of high-performance computing systems, the rising failure rate poses significant challenges for RDMA networks that aim for high bandwidth and low latency. RDMA networks require hardware-level end-to-end reliable data transmission services to avoid the high cost of software failure recovery. Tianhe HPC interconnection network adopts a NIC-based RDMA reliable connection protocol, RCP. RCP establishes a connection for each message that enters the NIC and releases it after the transmission is complete. However, this introduces an additional round-trip time RTT connection overhead for each message, which severely impacts the performance of networks dominated by short messages in high-performance computing systems. We have found that utilization of receiver-side connection resources has been consistently low because maintaining message-grained connections on the NIC results in rapid release of connections. Therefore, we propose a lightweight RDMA connection protocol based on post-hoc confirmation, PCP. PCP assumes the receiver has connection resources by default and eliminates the need for confirmation from the receiver before sending a message, thus reducing the connection overhead of almost all messages by one RTT. At the same time, PCP also includes mechanisms to address the special case where the receiver lacks connection resources. Evaluation results demonstrate that PCP significantly optimizes short messages and applications dominated by short messages. Moreover, PCP further reduces the usage of receiver-side connection resources. Additionally, PCP does not experience performance degradation even under large-scale heavy loads and severe endpoint congestion. •The RDMA NIC architecture and RDMA connection protocol (RCP) of Tianhe are introduced in detail.•The load characteristics in the HPC interconnection network and the challenges of RCP are analyzed.•The Post-hoc Connection Protocol is proposed, and its implementation details on NIC are introduced.•The evaluation results show that PCP has a significant acceleration on HPC network load dominated by short messages.