Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning

• Published in: IEEE journal on selected areas in communications 2019-06, Vol.37 (6), p.1325-1336
• Main Authors: Xu, Zhiyuan, Tang, Jian, Yin, Chengxiang, Wang, Yanzhi, Xue, Guoliang
• Format: Article
• Language: English
• Subjects: Active control; Algorithms; Congestion; congestion control; Control algorithms; deep learning; experience-driven control; Heuristic algorithms; Machine learning; Mathematical model; multi-path TCP; Packet switched networks; Protocols; Recurrent neural networks; Reinforcement learning; Representations; Resource management; Runtime; TCP; TCP (protocol)
• ISSN: 0733-8716; 1558-0008
• DOI: 10.1109/JSAC.2019.2904358

In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.