Controllable Motion Synthesis and Reconstruction with Autoregressive Diffusion Models

Data-driven and controllable human motion synthesis and prediction are active research areas with various applications in interactive media and social robotics. Challenges remain in these fields for generating diverse motions given past observations and dealing with imperfect poses. This paper intro...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-04
Main Authors: Yin, Wenjie, Tu, Ruibo, Yin, Hang, Kragic, Danica, Kjellström, Hedvig, Björkman, Mårten
Format: Article
Language:English
Subjects:
Autoregressive models
Coders
Controllability
Diffusion
Human motion
Locomotion
Reconstruction
Robotics
Robustness
Synthesis
Transformers
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Data-driven and controllable human motion synthesis and prediction are active research areas with various applications in interactive media and social robotics. Challenges remain in these fields for generating diverse motions given past observations and dealing with imperfect poses. This paper introduces MoDiff, an autoregressive probabilistic diffusion model over motion sequences conditioned on control contexts of other modalities. Our model integrates a cross-modal Transformer encoder and a Transformer-based decoder, which are found effective in capturing temporal correlations in motion and control modalities. We also introduce a new data dropout method based on the diffusion forward process to provide richer data representations and robust generation. We demonstrate the superior performance of MoDiff in controllable motion synthesis for locomotion with respect to two baselines and show the benefits of diffusion data dropout for robust synthesis and reconstruction of high-fidelity motion close to recorded data.
ISSN:2331-8422