HMDRL: Hierarchical Mixed Deep Reinforcement Learning to Balance Vehicle Supply and Demand

• Published in: IEEE transactions on intelligent transportation systems 2022-11, Vol.23 (11), p.21861-21872
• Main Authors: Xi, Jinhao, Zhu, Fenghua, Ye, Peijun, Lv, Yisheng, Tang, Haina, Wang, Fei-Yue
• Format: Article
• Language: English
• Subjects: Car sharing; Deep learning; Deep reinforcement learning; Heuristic algorithms; hierarchical repositioning framework; Idling; mixed state; online ride-hailing system; parallel coordination mechanism; Pricing; Reinforcement learning; Supply & demand; Supply and demand; Surges; Vehicle dynamics; Vehicles
• ISSN: 1524-9050; 1558-0016
• DOI: 10.1109/TITS.2022.3191752

The imbalance of vehicle supply and demand is a common phenomenon that influences the efficiency of online ride-hailing systems greatly. To address this problem, a three-level hierarchical mixed deep reinforcement learning method (HMDRL) is proposed to reposition idle vehicles. A manager operates at the top level, where action-abstraction is conducted from the time dimension and is adaptive for spatially scalable and time-varying systems. Coordinators locate at the middle level and a parallel coordination mechanism that is independent of the decision order is designed to improve the efficiency of the repositioning. The bottom level is composed of executive workers to reposition vehicles with mixed states and the states contain spatiotemporal information of agents' neighbor areas. Two reward functions are designed for the manager and the coordinators, respectively, aiming to improve the training effect by avoiding sparse rewards. A simulator based on real orders is designed and HMDRL is compared with six methods. Experimental results demonstrate that HMDRL outperforms all the other methods. In three comparison experiments, the order response rate is increased by 0.62% to 8.29%, 1.5% to 7.78%, 1.18% to 4.75%, respectively.