St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-lev...

Full description

Saved in:
Bibliographic Details
Main Authors: Kim, Minjeong, Kim, Gyuwan, Lee, Sang-Woo, Ha, Jung-Woo
Format: Conference Proceeding
Language:English
Subjects:
Benchmark testing
Cross-modal language model
Cross-modal pre-training
Data models
Degradation
Performance gain
Semantics
Signal processing
Spoken language understanding
Training data
Citations: Items that cite this one
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.
ISSN:2379-190X
DOI:10.1109/ICASSP39728.2021.9414558