DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalit...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-06
Main Authors: Ke-Han, Lu, Chen, Zhehuai, Szu-Wei Fu, Huang, He, Ginsburg, Boris, Yu-Chiang, Frank Wang, Hung-yi, Lee
Format: Article
Language:English
Subjects:
Alignment
Large language models
Linguistics
Speech
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions.
ISSN:2331-8422