COLLD: Contrastive Layer-to-Layer Distillation for Compressing Multilingual Pre-Trained Speech Encoders

Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studie...

Full description

Saved in:
Bibliographic Details
Main Authors: Chang, Heng-Jui, Dong, Ning, Mavlyutov, Ruslan, Popuri, Sravya, Chung, Yu-An
Format: Conference Proceeding
Language:English
Subjects:
Benchmark testing
Buildings
Costs
knowledge distillation
model compression
multilingual speech translation
Predictive models
Self-supervised learning
Signal processing
Speech recognition
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.
ISSN:2379-190X
DOI:10.1109/ICASSP48485.2024.10446637