Applying Multi- and Cross-Lingual Stochastic Phone Space Transformations to Non-Native Speech Recognition

• Published in: IEEE transactions on audio, speech, and language processing speech, and language processing, 2013-08, Vol.21 (8), p.1713-1726
• Main Authors: Imseng, D., Bourlard, H., Dines, J., Garner, P. N., Magimai-Doss, Mathew
• Format: Article
• Language: English
• Subjects: Acoustics; Adaptation; Algorithms; Applied sciences; Artificial intelligence; Bayesian analysis; Computer science; control theory; systems; Connectionism. Neural networks; Cost function; Exact sciences and technology; Hidden Markov models; Information, signal and communications theory; Mapping; multilingual acoustic modeling; Non-native speech recognition; Signal processing; Speech; Speech processing; Speech recognition; Stochastic processes; Stochasticity; Telecommunications and information theory; Telephones; Training; Transformations; universal phone set
• ISSN: 1558-7916; 2329-9290; 1558-7924; 2329-9304
• DOI: 10.1109/TASL.2013.2260150

In the context of hybrid HMM/MLP Automatic Speech Recognition (ASR), this paper describes an investigation into a new type of stochastic phone space transformation, which maps "source" phone (or phone HMM state) posterior probabilities (as obtained at the output of a Multilayer Perceptron/MLP) into "destination" phone (HMM phone state) posterior probabilities. The resulting stochastic matrix transformation can be used within the same language to automatically adapt to different phone formats (e.g., IPA) or across languages. Additionally, as shown here, it can also be applied successfully to non-native speech recognition. In the same spirit as MLLR adaptation, or MLP adaptation, the approach proposed here is directly mapping posterior distributions, and is trained by optimizing on a small amount of adaptation data a Kullback-Leibler based cost function, along a modified version of an iterative EM algorithm. On a non-native English database (HIWIRE), and comparing with multiple setups (monophone and triphone mapping, MLLR adaptation) we show that the resulting posterior mapping yields state-of-the-art results using very limited amounts of adaptation data in mono-, cross- and multi-lingual setups. We also show that "universal" phone posteriors, trained on a large amount of multilingual data, can be transformed to English phone posteriors, resulting in an ASR system that significantly outperforms a system trained on English data only. Finally, we demonstrate that the proposed approach outperforms alternative data-driven, as well as a knowledge-based, mapping techniques.