A Multi-agent Reinforcement Learning Perspective on Distributed Traffic Engineering

• Main Authors: Geng, Nan, Lan, Tian, Aggarwal, Vaneet, Yang, Yuan, Xu, Mingwei
• Format: Conference Proceeding
• Language: English
• Subjects: Convergence; Heuristic algorithms; Peer-to-peer computing; Real-time systems; Reinforcement learning; Routing; Scalability
• ISSN: 2643-3303
• DOI: 10.1109/ICNP49622.2020.9259413

Traffic engineering (TE) in multi-region networks is a challenging problem due to the requirement that each region must independently compute its routing decisions based on local observations, yet with the goal of optimizing global TE objectives. Traditional approaches often lack the agility to adapt to changing traffic patterns and thus may suffer hefty performance loss under highly dynamic traffic demands. In this paper, we propose a data-driven framework for multi-region TE problems, which makes novel use of multi-agent deep reinforcement learning. In particular, we propose two reinforcement learning agents for each region, namely T-agents and O-agents, to control the terminal traffic and outgoing traffic, respectively. These distributed agents collect local link utilization statistics within their regions, optimize local routing decisions, and observe the resulting congestion-related reward. To facilitate these agents for optimizing global TE objectives, we tailor the agent design carefully including input, output, and reward functions. The proposed framework is evaluated extensively using real-world network topologies (e.g., Telstra and Google Cloud) and synthetic traffic patterns (e.g., the Gravity model). Numerical results show that comparing with existing protocols and single-agent learning algorithms, our solution can significantly reduce congestion and achieve nearly-optimal performance with both superior scalability and robustness. Throughout our simulations, over 90% of tests limit congestion within 1.2 times the global optimal solution. 1