Towards Generating Ultra-High Resolution Talking-Face Videos with Lip synchronization

• Main Authors: Gupta, Anchit, Mukhopadhyay, Rudrabha, Balachandra, Sindhu, Khan, Faizan Farooq, Namboodiri, Vinay P., Jawahar, C. V.
• Format: Conference Proceeding
• Language: English
• Subjects: Algorithms: Vision + language and/or other modalities; Commercial/retail; Computer vision; Education; Generators; Industries; Lips; Motion pictures; Shape; Synchronization
• ISSN: 2642-9381
• DOI: 10.1109/WACV56688.2023.00518

Talking-face video generation works have achieved state-of-the-art results in synthesizing videos with lip synchronization. However, most of the previous works deal with low-resolution talking-face videos (up to 256×256 pixels), thus, generating extremely high-resolution videos still remains a challenge. We take a giant leap in this work and propose a novel method to synthesize talking-face videos at resolutions as high as 4K! Our task presents several key challenges: (i) Scaling the existing methods to such high resolutions is resource-constrained, both in terms of compute and the availability of very high-resolution datasets, (ii) The synthesized videos need to be spatially and temporally coherent. The sheer number of pixels that the model needs to generate while maintaining the temporal consistency at the video level makes this task non-trivial and has never been attempted before in literature. To address these issues, we propose to train the lip-sync generator in a compact Vector Quantized (VQ) space for the first time. Our core idea to encode the faces in a compact 16× 16 representation allows us to model high-resolution videos. In our framework, we learn the lip movements in the quantized space on the newly collected 4K Talking Faces (4KTF) dataset. Our approach is speaker agnostic and can handle various languages and voices. We benchmark our technique against several competitive works and show that we can achieve a remarkable 64-times more pixels than the current state-of-the-art! Our supplementary demo video depicts additional qualitative results, comparisons, and several real-world applications, like professional movie editing enabled by our model.