Performance trade-offs in reconfigurable networks for HPC

• Published in: Journal of optical communications and networking 2022-06, Vol.14 (6), p.454-468
• Main Authors: Teh, Min Yee, Wu, Zhenguo, Glick, Madeleine, Rumley, Sebastien, Ghobadi, Manya, Bergman, Keren
• Format: Article
• Language: English
• Subjects: Bandwidths; Circuits; Communication; Computation; Computer architecture; Data centers; Design; Fanout; Network design; Network latency; Network topology; Optical switches; Power consumption; Proposals; Racks; Reconfiguration; Routing; Switches; Switching circuits; Throughput; Topology; Tradeoffs; Workload; Workloads
• ISSN: 1943-0620; 1943-0639
• DOI: 10.1364/JOCN.451760

Designing efficient interconnects to support high-bandwidth and low-latency communication is critical toward realizing high performance computing (HPC) and data center (DC) systems in the exascale era. At extreme computing scales, providing the requisite bandwidth through overprovisioning becomes impractical. These challenges have motivated studies exploring reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. Despite the plethora of proposed architectures, surprisingly little is known about the relative performances and trade-offs among different reconfigurable network designs. We aim to bridge this gap by tackling two key issues in reconfigurable network design. First, we study how cost, power consumption, network performance, and scalability vary based on optical circuit switch (OCS) placement in the physical topology. Specifically, we consider two classes of reconfigurable architectures: one that places OCSs between top-of-rack (ToR) switches—ToR-reconfigurable networks (TRNs)—and one that places OCSs between pods of racks—pod-reconfigurable networks (PRNs). Second, we tackle the effects of reconfiguration frequency on network performance. Our results, based on network simulations driven by real HPC and DC workloads, show that while TRNs are optimized for low fan-out communication patterns, they are less suited for carrying high fan-out workloads. PRNs exhibit better overall trade-off, capable of performing comparably to a fully non-blocking fat tree for low fan-out workloads, and significantly outperform TRNs for high fan-out communication patterns.