RoHM: Robust Human Motion Reconstruction via Diffusion

We propose RoHM, an approach for robust 3D human motion reconstruction from monocular RGB(-D) videos in the presence of noise and occlusions. Most previous approaches either train neural networks to directly regress motion in 3D or learn data-driven motion priors and com-bine them with optimization...

Full description

Saved in:
Bibliographic Details
Main Authors: Zhang, Siwei, Bhatnagar, Bharat Lal, Xu, Yuanlu, Winkler, Alexander, Kadlecek, Petr, Tang, Siyu, Bogo, Federica
Format: Conference Proceeding
Language:English
Subjects:
3d human pose estimation
Correlation
diffusion
Diffusion models
human mesh recovery
Image reconstruction
Iterative methods
motion capture
motion prior
motion reconstruction
Noise
Noise reduction
Three-dimensional displays
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose RoHM, an approach for robust 3D human motion reconstruction from monocular RGB(-D) videos in the presence of noise and occlusions. Most previous approaches either train neural networks to directly regress motion in 3D or learn data-driven motion priors and com-bine them with optimization at test time. The former do not recover globally coherent motion and fail under occlusions; the latter are time-consuming, prone to local minima, and require manual tuning. To overcome these shortcomings, we exploit the iterative, denoising nature of diffusion models. RoHM is a novel diffusion-based motion model that, conditioned on noisy and occluded input data, reconstructs complete, plausible motions in consistent global co-ordinates. Given the complexity of the problem - requiring one to address different tasks (denoising and infilling) in different solution spaces (local and global motion) - we de-compose it into two sub-tasks and learn two models, one for global trajectory and one for local motion. To capture the correlations between the two, we then introduce a novel conditioning module, combining it with an iterative inference scheme. We apply RoHM to a variety of tasks from motion reconstruction and denoising to spatial and temporal infilling. Extensive experiments on three popular datasets show that our method outperforms state-of-the-art approaches qualitatively and quantitatively, while being faster at test time. The code is available at https://sanweiliti.github.io/ROHM/ROHM.html.
ISSN:2575-7075
DOI:10.1109/CVPR52733.2024.01384