Exploring Phonetic Context-Aware Lip-Sync for Talking Face Generation

• Main Authors: Park, Se Jin, Kim, Minsu, Choi, Jeongsoo, Ro, Yong Man
• Format: Conference Proceeding
• Language: English
• Subjects: Acoustics; Audio-visual Synchronization; Context learning; Face synthesis; Lips; Phonetics; Signal processing; Speech processing; Synchronization; Talking face generation; Task analysis
• ISSN: 2379-190X
• DOI: 10.1109/ICASSP48485.2024.10447284

Talking face generation is the challenging task of synthesizing a natural and realistic face that requires accurate synchronization with a given audio. Due to co-articulation, where an isolated phone is influenced by the preceding or following phones, the articulation of a phone varies upon the phonetic context. Therefore, modeling lip motion with the phonetic context can generate more spatio-temporally aligned lip movement. In this respect, we investigate the phonetic context in generating lip motion for talking face generation. We propose Context-Aware Lip-Sync framework (CALS), which explicitly leverages phonetic context to generate lip movement of the target face. CALS is comprised of an Audio-to-Lip module and a Lip-to-Face module. The former is pretrained based on masked learning to map each phone to a contextualized lip motion unit. The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion. From extensive experiments, we verify that simply exploiting the phonetic context in the proposed CALS framework effectively enhances spatio-temporal alignment. We also demonstrate the extent to which the phonetic context assists in lip synchronization and find the effective window size for lip generation to be approximately 1.2 seconds.