Efficient Extraction of Noise-Robust Discrete Units from Self-Supervised Speech Models

Continuous speech can be converted into a discrete sequence by deriving discrete units from the hidden features of self-supervised learned (SSL) speech models. Although SSL models are becoming larger and trained on more data, they are often sensitive to real-life distortions like additive noise or r...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-09
Main Authors: Poncelet, Jakob, Wang, Yujun, Hugo Van hamme
Format: Article
Language:English
Subjects:
Encoders-Decoders
Feature recognition
Noise generation
Noise sensitivity
Parameter robustness
Parameter sensitivity
Robustness
Speech recognition
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Continuous speech can be converted into a discrete sequence by deriving discrete units from the hidden features of self-supervised learned (SSL) speech models. Although SSL models are becoming larger and trained on more data, they are often sensitive to real-life distortions like additive noise or reverberation, which translates to a shift in discrete units. We propose a parameter-efficient approach to generate noise-robust discrete units from pre-trained SSL models by training a small encoder-decoder model, with or without adapters, to simultaneously denoise and discretise the hidden features of the SSL model. The model learns to generate a clean discrete sequence for a noisy utterance, conditioned on the SSL features. The proposed denoiser outperforms several pre-training methods on the tasks of noisy discretisation and noisy speech recognition, and can be finetuned to the target environment with a few recordings of unlabeled target data.
ISSN:2331-8422