An optimized on-ramp metering method for urban expressway based on reinforcement learning

• Published in: Journal of intelligent & fuzzy systems 2020-01, Vol.38 (3), p.2703-2715
• Main Authors: Chai, Gan, Cao, Jinde, Xu, Shaosheng
• Format: Article
• Language: English
• Subjects: Algorithms; Computer simulation; Delay; Highways; Machine learning; Occupancy; Queues; Queuing theory; Roads; Roads & highways; Traffic congestion; Traffic models; Traffic volume; Travel time
• ISSN: 1064-1246; 1875-8967
• DOI: 10.3233/JIFS-179556

Aiming at the problem of mainline congestion and ramp queue spill in urban expressway, an optimized on-ramp control method based on reinforcement learning is put forward. Online reinforcement learning algorithm is used to optimize on-ramp control regulation by taking the metering rate as the action, the length of ramp queue, the throughput and the occupancy rate of the interweaving area as the state, and the volume of the road network as the reward function. By iterating the value function with actual behavior, the proposed method can avoid the establishment of an accurate traffic model and the reliance on prior knowledge. Meanwhile, the real-time update of the value function Q compensates for the defect of control hysteresis. Compared with the classical method in simulation scenarios of Nanjing Kazimen Expressway, the average delay of the proposed method is reduced by 16.83%, the total delay reduced by 15.83%, the average speed enhanced by 6.80%, and the total travel time decreased by 5.22%; the average queue length of the on-ramp decreased by 89%; the average occupancy rate of the weaving area is decreased by 2.42% at rush hours, and the average traffic volume increased by 109veh/h.