Confidence sharing adaptation for out-of-domain human pose and shape estimation

• Published in: Computer vision and image understanding 2024-09, Vol.246, p.104051, Article 104051
• Main Authors: Yue, Tianyi, Ren, Keyan, Shi, Yu, Zhao, Hu, Bian, Qingyun
• Format: Article
• Language: English
• Subjects: Monocular human pose and shape estimation; Out-of-domain adaptation; Rendering loss function; Self-attention
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2024.104051

3D human pose and shape estimation is often impacted by distribution bias in real-world scenarios due to factors such as bone length, camera parameters, background, and occlusion. To address this issue, we propose the Confidence Sharing Adaptation (CSA) algorithm, which corrects model bias using unlabeled images from the test domain before testing. However, the lack of annotation constraints in the adaptive training process poses a significant challenge, making it susceptible to model collapse. CSA utilizes a decoupled dual-branch learning framework to provide pseudo-labels and remove noise samples based on the confidence scores of the inference results. By sharing the most confident prior knowledge between the dual-branch networks, CSA effectively mitigates distribution bias. CSA is also remarkably adaptable to severely occluded scenes, thanks to two auxiliary techniques: a self-attentive parametric regressor that ensures robustness to occlusion of local body parts and a rendered surface texture loss that regulates the relationship between occlusion of human joint positions. Evaluation results show that CSA successfully adapts to scenarios beyond the training domain and achieves state-of-the-art performance on both occlusion-specific and general benchmarks. Code and pre-trained models are available for research at https://github.com/bodymapper/csa.git •Novel confidence-sharing adaptation (CSA) method for 3D human pose and shape estimation.•CSA adapts to new scenario using monocular images and reduces domain gap.•CSA includes a self-attention regressor to improve robustness to external occlusion.•CSA includes a rendering normal texture constraint to clear joint self-occlusion.•CSA achieves state-of-the-art performance on both occlusion-specific and general benchmarks.