MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion

In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-05
Main Authors: Chang, Di, Shi, Yichun, Gao, Quankai, Fu, Jessica, Xu, Hongyi, Song, Guoxian, Yan, Qing, Zhu, Yizhe, Yang, Xiao, Soleymani, Mohammad
Format: Article
Language:English
Subjects:
Animation
Dance
Datasets
Human motion
Robust control
Two dimensional models
Video
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance
ISSN:2331-8422