Humanoid Audio-Visual Avatar With Emotive Text-to-Speech Synthesis

• Published in: IEEE transactions on multimedia 2008-10, Vol.10 (6), p.969-981
• Main Authors: Hao Tang, Yun Fu, Jilin Tu, Hasegawa-Johnson, M., Huang, T.S.
• Format: Article
• Language: English
• Subjects: 3-D face modeling and animation; Acoustics; Applied sciences; Artificial intelligence; Audio-visual avatar; Avatars; Computer science; control theory; systems; Computer systems and distributed systems. User interface; emotive speech synthesis; Exact sciences and technology; Face; Facial animation; Fundamental areas of phenomenology (including applications); Government; human-computer interaction; Humans; Intelligent agent; Man machine systems; multimodal system; Pattern recognition. Digital image processing. Computational geometry; Physics; Prototypes; Software; Speech and sound recognition and synthesis. Linguistics; Speech synthesis; Synthesizers; Transduction; acoustical devices for the generation and reproduction of sound; TTS
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2008.2001355

Emotive audio-visual avatars are virtual computer agents which have the potential of improving the quality of human-machine interaction and human-human communication significantly. However, the understanding of human communication has not yet advanced to the point where it is possible to make realistic avatars that demonstrate interactions with natural-sounding emotive speech and realistic-looking emotional facial expressions. In this paper, We propose the various technical approaches of a novel multimodal framework leading to a text-driven emotive audio-visual avatar. Our primary work is focused on emotive speech synthesis, realistic emotional facial expression animation, and the co-articulation between speech gestures (i.e., lip movements) and facial expressions. A general framework of emotive text-to-speech (TTS) synthesis using a diphone synthesizer is designed and integrated into a generic 3-D avatar face model. Under the guidance of this framework, we therefore developed a realistic 3-D avatar prototype. A rule-based emotive TTS synthesis system module based on the Festival-MBROLA architecture has been designed to demonstrate the effectiveness of the framework design. Subjective listening experiments were carried out to evaluate the expressiveness of the synthetic talking avatar.