3D Visual Speech Animation Using 2D Videos

In visual speech animation, lip motion accuracy is of paramount importance for speech intelligibility, especially for the hard of hearing or foreign language learners. We present an approach for visual speech animation that uses tracked lip motion in front-view 2D videos of a real speaker to drive t...

Full description

Saved in:
Bibliographic Details
Main Authors: Algadhy, Rabab, Gotoh, Yoshihiko, Maddock, Steve
Format: Conference Proceeding
Language:English
Subjects:
3D morphable model
Animation
Head
lip motion
Shape
Three-dimensional displays
Two dimensional displays
Videos
visual speech animation
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In visual speech animation, lip motion accuracy is of paramount importance for speech intelligibility, especially for the hard of hearing or foreign language learners. We present an approach for visual speech animation that uses tracked lip motion in front-view 2D videos of a real speaker to drive the lip motion of a synthetic 3D head. This makes use of a 3D morphable model (3DMM), built using 3D synthetic head poses, with corresponding landmarks identified in the 2D videos and the 3DMM. We show that using a wider range of synthetic head poses for different phoneme intensities to create a 3DMM, as well as a combination of front and side photographs of the real speakers rather than just front photographs to produce initial neutral 3D synthetic head poses, gives better animation results when compared to ground truth data consisting of front-view 2D videos of real speakers.
ISSN:2379-190X
DOI:10.1109/ICASSP.2019.8682455