COER: A Network Interface Offloading Architecture for RDMA and Congestion Control Protocol Codesign

• Published in: ACM transactions on architecture and code optimization 2024-09, Vol.21 (3), p.1-26, Article 49
• Main Authors: Wu, Ke, Dong, Dezun, Xu, Weixia
• Format: Article
• Language: English
• Subjects: Communication hardware, interfaces and storage; Hardware; Network protocols; Networks
• ISSN: 1544-3566; 1544-3973
• DOI: 10.1145/3660525

RDMA (Remote Direct Memory Access) networks require efficient congestion control to maintain their high throughput and low latency characteristics. However, congestion control protocols deployed at the software layer suffer from slow response times due to the communication overhead between host hardware and software. This limitation has hindered their ability to meet the demands of high-speed networks and applications. Harnessing the capabilities of rapidly advancing Network Interface Cards (NICs) can drive progress in congestion control. Some simple congestion control protocols have been offloaded to RDMA NICs to enable faster detection and processing of congestion. However, offloading congestion control to the RDMA NIC faces a significant challenge in integrating the RDMA transport protocol with advanced congestion control protocols that involve complex mechanisms. We have observed that reservation-based proactive congestion control protocols share strong similarities with RDMA transport protocols, allowing them to integrate seamlessly and combine the functionalities of the transport layer and network layer. In this article, we present COER, an RDMA NIC architecture that leverages the functional components of RDMA to perform reservations and completes the scheduling of congestion control during the scheduling process of the RDMA protocol. COER facilitates the streamlined development of offload strategies for congestion control techniques —specifically, proactive congestion control —on RDMA NICs. We use COER to design offloading schemes for 11 congestion control protocols, which we implement and evaluate using a network emulator with a cycle-accurate RDMA NIC model that can load Message Passing Interface (MPI) programs. The evaluation results demonstrate that the architecture of COER does not compromise the original characteristics of the congestion control protocols. Compared with a layered protocol stack approach, COER enables the performance of RDMA networks to reach new heights.