Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition

• Published in: arXiv.org 2022-10
• Main Authors: Rathod, Jash, Nauman Dawalatabad, Singh, Shatrughan, Gowda, Dhananjaya
• Format: Article
• Language: English
• Subjects: Automatic speech recognition; Coders; Degradation; Distillation; Electronic devices; Memory devices; Teachers; Voice recognition
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2210.00169

The smaller memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. To obtain a smaller model, one can employ the model compression techniques. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size with relatively lesser degradation in the model performance. In this approach, knowledge is distilled from a trained large size teacher model to a smaller size student model. Also, the transducer based models have recently shown to perform well for on-device streaming ASR task, while the conformer models are efficient in handling long term dependencies. Hence in this work we employ a streaming transducer architecture with conformer as the encoder. We propose a multi-stage progressive approach to compress the conformer transducer model using KD. We progressively update our teacher model with the distilled student model in a multi-stage setup. On standard LibriSpeech dataset, our experimental results have successfully achieved compression rates greater than 60% without significant degradation in the performance compared to the larger teacher model.