Crowd3D++: Robust Monocular Crowd Reconstruction with Upright Space

• Published in: arXiv.org 2024-11
• Main Authors: Huang, Jing, Wen, Hao, Zhou, Tianyi, Lin, Haozhe, Yu-Kun, Lai, Li, Kun
• Format: Article
• Language: English
• Subjects: Cameras; Image reconstruction; Localization; Parameters; Robustness; Source code; Virtual reality
• ISSN: 2331-8422

This paper aims to reconstruct hundreds of people's 3D poses, shapes, and locations from a single image with unknown camera parameters. Due to the small and highly varying 2D human scales, depth ambiguity, and perspective distortion, no existing methods can achieve globally consistent reconstruction and accurate reprojection. To address these challenges, we first propose Crowd3D, which leverages a new concept, Human-scene Virtual Interaction Point (HVIP), to convert the complex 3D human localization into 2D-pixel localization with robust camera and ground estimation to achieve globally consistent reconstruction. To achieve stable generalization on different camera FoVs without test-time optimization, we propose an extended version, Crowd3D++, which eliminates the influence of camera parameters and the cropping operation by the proposed canonical upright space and ground-aware normalization transform. In the defined upright space, Crowd3D++ also designs an HVIPNet to regress 2D HVIP and infer the depths. Besides, we contribute two benchmark datasets, LargeCrowd and SyntheticCrowd, for evaluating crowd reconstruction in large scenes. The source code and data will be made publicly available after acceptance.