FlexLip: A Controllable Text-to-Lip System

• Published in: Sensors (Basel, Switzerland) Switzerland), 2022-05, Vol.22 (11), p.4104
• Main Authors: Oneață, Dan, Lőrincz, Beáta, Stan, Adriana, Cucu, Horia
• Format: Article
• Language: English
• Subjects: Adaptation; Audio data; Computer architecture; Conversion; Deep learning; Evaluation; generative models; Modular systems; Modularity; Mouth; Multimedia; Neural networks; Speech recognition; speech synthesis; speech-to-lip; Speeches; text-to-lip; text-to-speech; zero-shot adaptation
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22114104

The task of converting text input into video content is becoming an important topic for synthetic media generation. Several methods have been proposed with some of them reaching close-to-natural performances in constrained tasks. In this paper, we tackle a subissue of the text-to-video generation problem, by converting the text into lip landmarks. However, we do this using a modular, controllable system architecture and evaluate each of its individual components. Our system, entitled FlexLip, is split into two separate modules: text-to-speech and speech-to-lip, both having underlying controllable deep neural network architectures. This modularity enables the easy replacement of each of its components, while also ensuring the fast adaptation to new speaker identities by disentangling or projecting the input features. We show that by using as little as 20 min of data for the audio generation component, and as little as 5 min for the speech-to-lip component, the objective measures of the generated lip landmarks are comparable with those obtained when using a larger set of training samples. We also introduce a series of objective evaluation measures over the complete flow of our system by taking into consideration several aspects of the data and system configuration. These aspects pertain to the quality and amount of training data, the use of pretrained models, and the data contained therein, as well as the identity of the target speaker; with regard to the latter, we show that we can perform zero-shot lip adaptation to an unseen identity by simply updating the shape of the lips in our model.