LASOR: Learning Accurate 3D Human Pose and Shape via Synthetic Occlusion-Aware Data and Neural Mesh Rendering

• Published in: IEEE transactions on image processing 2022, Vol.31, p.1938-1948
• Main Authors: Yang, Kaibing, Gu, Renshu, Wang, Maoyu, Toyoura, Masahiro, Xu, Gang
• Format: Article
• Language: English
• Subjects: 2D keypoint; 3D human pose and shape estimation; Biological system modeling; Cameras; Datasets; Humans; Imaging, Three-Dimensional - methods; neural mesh renderer; Occlusion; occlusion-aware; Pose estimation; Shape; silhouette; Solid modeling; Surgical Mesh; Synthesis; Three-dimensional displays; Training; Training data
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2022.3149229

A key challenge in the task of human pose and shape estimation is occlusion, including self-occlusions, object-human occlusions, and inter-person occlusions. The lack of diverse and accurate pose and shape training data becomes a major bottleneck, especially for scenes with occlusions in the wild. In this paper, we focus on the estimation of human pose and shape in the case of inter-person occlusions, while also handling object-human occlusions and self-occlusion. We propose a novel framework that synthesizes occlusion-aware silhouette and 2D keypoints data and directly regress to the SMPL pose and shape parameters. A neural 3D mesh renderer is exploited to enable silhouette supervision on the fly, which contributes to great improvements in shape estimation. In addition, keypoints-and-silhouette-driven training data in panoramic viewpoints are synthesized to compensate for the lack of viewpoint diversity in any existing dataset. Experimental results show that we are among the state-of-the-art on the 3DPW and 3DPW-Crowd datasets in terms of pose estimation accuracy. The proposed method evidently outperforms Mesh Transformer, 3DCrowdNet and ROMP in terms of shape estimation. Top performance is also achieved on SSP-3D in terms of shape prediction accuracy. Demo and code will be available at https://igame-lab.github.io/LASOR/ .