Unified Automatic Control of Vehicular Systems With Reinforcement Learning

• Published in: IEEE transactions on automation science and engineering 2023-04, Vol.20 (2), p.789-804
• Main Authors: Yan, Zhongxia, Kreidieh, Abdul Rahman, Vinitsky, Eugene, Bayen, Alexandre M., Wu, Cathy
• Format: Article
• Language: English
• Subjects: automated vehicles; Automatic control; Automation; Control systems; Deep learning; Dynamical systems; Empirical analysis; Lane changing; Markov processes; Methodology; multi-agent systems; Nonlinear systems; Optimal control; Optimization; Primary topics: Mobile traffic control; Reinforcement learning; reinforcement learning Secondary topic keywords: Mixed autonomy; Robot kinematics; Simulation; Simulator fidelity; System effectiveness; Tuning; Vehicle dynamics; Vehicles
• ISSN: 1545-5955; 1558-3783
• DOI: 10.1109/TASE.2022.3168621

Emerging vehicular systems with increasing proportions of automated components present opportunities for optimal control to mitigate congestion and increase efficiency. There has been a recent interest in applying deep reinforcement learning (DRL) to these nonlinear dynamical systems for the automatic design of effective control strategies. Despite conceptual advantages of DRL being model-free, studies typically nonetheless rely on training setups that are painstakingly specialized to specific vehicular systems. This is a key challenge to efficient analysis of diverse vehicular and mobility systems. To this end, this article contributes a streamlined methodology for vehicular microsimulation and discovers high performance control strategies with minimal manual design. A variable-agent, multi-task approach is presented for optimization of vehicular Partially Observed Markov Decision Processes. The methodology is experimentally validated on mixed autonomy traffic systems, where fractions of vehicles are automated; empirical improvement, typically 15-60% over a human driving baseline, is observed in all configurations of six diverse open or closed traffic systems. The study reveals numerous emergent behaviors resembling wave mitigation, traffic signaling, and ramp metering. Finally, the emergent behaviors are analyzed to produce interpretable control strategies, which are validated against the learned control strategies. Note to Practitioners-As vehicular systems such as real-world traffic systems and robotic warehouses become increasingly automated, optimizing vehicle movements sees an increasing potential to reduce congestion and increase efficiency. For many vehicular systems, simulations of varying fidelity are commonly used for analysis and optimization without the need to deploy real vehicles. This article describes a unified and practical approach for optimal control of vehicles in arbitrary simulated vehicular systems while permitting partial automation, where the behavior of fractions of vehicles at given times can be modelled but not controlled. As illustrated by the diverse traffic systems considered in this article, the presented methodology emphasizes ease of application within any simulated vehicular system while minimizing manual efforts by the practitioner. The control inputs consist of local information around each automated vehicle, while the control outputs are commands for longitudinal acceleration and lateral lane change. Experimental res