SocNavBench: A Grounded Simulation Testing Framework for Evaluating Social Navigation

• Published in: ACM transactions on human-robotic interaction 2022-07, Vol.11 (3), p.1-24, Article 26
• Main Authors: Biswas, Abhijat, Wang, Allan, Silvera, Gustavo, Steinfeld, Aaron, Admoni, Henny
• Format: Article
• Language: English
• Subjects: Computing methodologies; HCI design and evaluation methods; Human-centered computing; Simulation types and techniques; Social navigation
• ISSN: 2573-9522; 2573-9522
• DOI: 10.1145/3476413

The human-robot interaction community has developed many methods for robots to navigate safely and socially alongside humans. However, experimental procedures to evaluate these works are usually constructed on a per-method basis. Such disparate evaluations make it difficult to compare the performance of such methods across the literature. To bridge this gap, we introduce SocNavBench, a simulation framework for evaluating social navigation algorithms. SocNavBench comprises a simulator with photo-realistic capabilities and curated social navigation scenarios grounded in real-world pedestrian data. We also provide an implementation of a suite of metrics to quantify the performance of navigation algorithms on these scenarios. Altogether, SocNavBench provides a test framework for evaluating disparate social navigation methods in a consistent and interpretable manner. To illustrate its use, we demonstrate testing three existing social navigation methods and a baseline method on SocNavBench, showing how the suite of metrics helps infer their performance trade-offs. Our code is open-source, allowing the addition of new scenarios and metrics by the community to help evolve SocNavBench to reflect advancements in our understanding of social navigation.