A Unified Framework for Multilingual Speech Recognition in Air Traffic Control Systems

• Published in: IEEE transaction on neural networks and learning systems 2021-08, Vol.32 (8), p.3608-3620
• Main Authors: Lin, Yi, Guo, Dongyue, Zhang, Jianwei, Chen, Zhengmao, Yang, Bo
• Format: Article
• Language: English
• Subjects: Acoustic model (AM); Acoustics; Air traffic control; air traffic control (ATC); Algorithms; Artificial neural networks; Atmospheric modeling; Aviation; Coders; Computer Systems; Control systems; Decoding; English language; Hidden Markov models; Language; Machine translation; machine translation pronunciation model (PM); multilingual; Multilingualism; multiscale CNN (MCNN); Neural networks; Neural Networks, Computer; Phonemes; Radio transmission; Real-time systems; Recurrent neural networks; Reproducibility of Results; robust speech recognition; Speech; Speech recognition; Speech Recognition Software; Task analysis; Translation; Translations; Vocabulary; Voice recognition; Words (language)
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2020.3015830

This work focuses on robust speech recognition in air traffic control (ATC) by designing a novel processing paradigm to integrate multilingual speech recognition into a single framework using three cascaded modules: an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The AM converts ATC speech into phoneme-based text sequences that the PM then translates into a word-based sequence, which is the ultimate goal of this research. The LM corrects both phoneme- and word-based errors in the decoding results. The AM, including the convolutional neural network (CNN) and recurrent neural network (RNN), considers the spatial and temporal dependences of the speech features and is trained by the connectionist temporal classification loss. To cope with radio transmission noise and diversity among speakers, a multiscale CNN architecture is proposed to fit the diverse data distributions and improve the performance. Phoneme-to-word translation is addressed via a proposed machine translation PM with an encoder-decoder architecture. RNN-based LMs are trained to consider the code-switching specificity of the ATC speech by building dependences with common words. We validate the proposed approach using large amounts of real Chinese and English ATC recordings and achieve a 3.95% label error rate on Chinese characters and English words, outperforming other popular approaches. The decoding efficiency is also comparable to that of the end-to-end model, and its generalizability is validated on several open corpora, making it suitable for real-time approaches to further support ATC applications, such as ATC prediction and safety checking.