Heterogeneous Crowd Simulation Using Parametric Reinforcement Learning

• Published in: IEEE transactions on visualization and computer graphics 2023-04, Vol.29 (4), p.2036-2052
• Main Authors: Hu, Kaidong, Haworth, Brandon, Berseth, Glen, Pavlovic, Vladimir, Faloutsos, Petros, Kapadia, Mubbasir
• Format: Article
• Language: English
• Subjects: Animation; Collision avoidance; Computational modeling; crowd simulation; Flight simulators; Laminar flow; Learning; Mathematical models; Multi-agent navigation; Multiagent systems; Navigation; Neural networks; Parameters; parametric policy learning; Policies; Predictive models; Reinforcement learning; Simulation; Steering; Training; Virtual humans
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2021.3139031

Agent-based synthetic crowd simulation affords the cost-effective large-scale simulation and animation of interacting digital humans. Model-based approaches have successfully generated a plethora of simulators with a variety of foundations. However, prior approaches have been based on statically defined models predicated on simplifying assumptions, limited video-based datasets, or homogeneous policies. Recent works have applied reinforcement learning to learn policies for navigation. However, these approaches may learn static homogeneous rules, are typically limited in their generalization to trained scenarios, and limited in their usability in synthetic crowd domains. In this article, we present a multi-agent reinforcement learning-based approach that learns a parametric predictive collision avoidance and steering policy. We show that training over a parameter space produces a flexible model across crowd configurations. That is, our goal-conditioned approach learns a parametric policy that affords heterogeneous synthetic crowds. We propose a model-free approach without centralization of internal agent information, control signals, or agent communication. The model is extensively evaluated. The results show policy generalization across unseen scenarios, agent parameters, and out-of-distribution parameterizations. The learned model has comparable computational performance to traditional methods. Qualitatively the model produces both expected (laminar flow, shuffling, bottleneck) and unexpected (side-stepping) emergent qualitative behaviours, and quantitatively the approach is performant across measures of movement quality.