Emotional voice conversion: Theory, databases and ESD

• Published in: Speech communication 2022-02, Vol.137, p.1-18
• Main Authors: Zhou, Kun, Sisman, Berrak, Liu, Rui, Li, Haizhou
• Format: Article
• Language: English
• Subjects: Chinese languages; Conversion; Emotional speech database (ESD); Emotional voice conversion; Emotions; Overview; Speech; State-of-the-art reviews; Voice; Voice conversion
• ISSN: 0167-6393; 1872-7182
• DOI: 10.1016/j.specom.2021.11.006

In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database11https://hltsingapore.github.io/ESD/. is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 h of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release. •This paper provides a comprehensive overview of the recent emotional voice conversion research and existing emotional speech databases. To our best knowledge, this paper is the first overview paper that covers emotional voice conversion research and databases in recent years.•We release the Emotional Speech Database (ESD) and make it publicly available, which represents one of the largest emotional speech databases, and is suitable for multi-speaker and cross-lingual emotional voice conversion, and other speech synthesis studies.•The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 h of speech data were recorded in a controlled acoustic environment.•By reporting several experiments on the ESD database, this paper provides a reference benchmark for emotional voice conversion studies that represent the state-of-the-art. All the codes and speech samples are publicly available.