Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition

This paper addresses the challenge of integrating low-resource languages into multilingual automatic speech recognition (ASR) systems. We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets, to facilitate the integration of low-resource languages into pre-...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-09
Main Authors: Piñeiro-Martín, Andrés, García-Mateo, Carmen, Docío-Fernández, Laura, María del Carmen López-Pérez, Rehm, Georg
Format: Article
Language:English
Subjects:
Automatic speech recognition
Data augmentation
Entropy
Error reduction
Languages
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper addresses the challenge of integrating low-resource languages into multilingual automatic speech recognition (ASR) systems. We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets, to facilitate the integration of low-resource languages into pre-trained multilingual ASR models within the context of continual multilingual learning. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language, employing language-weighted dynamic cross-entropy and data augmentation. The results show a remarkable 6.69% word error rate (WER) reduction for the low-resource language compared to the fine-tuned model without applying our approach, and a 48.86% WER reduction compared to the original Whisper model. In addition, our approach yields an average WER reduction of 3.29% across the six languages, showing no degradation for the high-resource languages.
ISSN:2331-8422
DOI:10.48550/arxiv.2409.16954