Articulatory Control of HMM-Based Parametric Speech Synthesis Using Feature-Space-Switched Multiple Regression

• Published in: IEEE transactions on audio, speech, and language processing speech, and language processing, 2013-01, Vol.21 (1), p.207-219
• Main Authors: Zhen-Hua Ling, Richmond, K., Yamagishi, J.
• Format: Article
• Language: English
• Subjects: Acoustics; Applied sciences; Articulatory features; Context; Exact sciences and technology; Gaussian mixture model; Hidden Markov models; Information, signal and communications theory; Mathematical models; multiple-regression hidden Markov model; Pattern recognition; Regression; Signal processing; Speech; Speech processing; Speech synthesis; Studies; Synthesizers; Tasks; Telecommunications and information theory; Transforms; Vowels
• ISSN: 1558-7916; 2329-9290; 1558-7924; 2329-9304
• DOI: 10.1109/TASL.2012.2215600

In previous work we proposed a method to control the characteristics of synthetic speech flexibly by integrating articulatory features into a hidden Markov model (HMM) based parametric speech synthesizer. In this method, a unified acoustic-articulatory model is trained, and context-dependent linear transforms are used to model the dependency between the two feature streams. In this paper, we go significantly further and propose a feature-space-switched multiple regression HMM to improve the performance of articulatory control. A multiple regression HMM (MRHMM) is adopted to model the distribution of acoustic features, with articulatory features used as exogenous "explanatory" variables. A separate Gaussian mixture model (GMM) is introduced to model the articulatory space, and articulatory-to-acoustic regression matrices are trained for each component of this GMM, instead of for the context-dependent states in the HMM. Furthermore, we propose a task-specific context feature tailoring method to ensure compatibility between state context features and articulatory features that are manipulated at synthesis time. The proposed method is evaluated on two tasks, using a speech database with acoustic waveforms and articulatory movements recorded in parallel by electromagnetic articulography (EMA). In a vowel identity modification task, the new method achieves better performance when reconstructing target vowels by varying articulatory inputs than our previous approach. A second vowel creation task shows our new method is highly effective at producing a new vowel from appropriate articulatory representations which, even though no acoustic samples for this vowel are present in the training data, is shown to sound highly natural.