Cascaded cross-modal transformer for audio–textual classification

Speech classification tasks often require powerful language understanding models to grasp useful features, which becomes problematic when limited training data is available. To attain superior classification performance, we propose to harness the inherent value of multimodal representations by trans...

Full description

Saved in:
Bibliographic Details
Published in:The Artificial intelligence review 2024-08, Vol.57 (9), p.225, Article 225
Main Authors: Ristea, Nicolae-Cătălin, Anghel, Andrei, Ionescu, Radu Tudor
Format: Article
Language:English
Subjects:
Artificial Intelligence
Audio data
Automatic speech recognition
Classification
Computational linguistics
Computer Science
Datasets
Fourier transforms
Imperative sentences
Language
Languages
Large language models
Machine translation
Multilingualism
Multimedia
Multimodality
Neural networks
Paralinguistics
Representations
Speech
Speech recognition
Transformers
Translating
Voice recognition
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Speech classification tasks often require powerful language understanding models to grasp useful features, which becomes problematic when limited training data is available. To attain superior classification performance, we propose to harness the inherent value of multimodal representations by transcribing speech using automatic speech recognition models and translating the transcripts into different languages via pretrained translation models. We thus obtain an audio–textual (multimodal) representation for each data sample. Subsequently, we combine language-specific Bidirectional Encoder Representations from Transformers with Wav2Vec2.0 audio features via a novel cascaded cross-modal transformer (CCMT). Our model is based on two cascaded transformer blocks. The first one combines text-specific features from distinct languages, while the second one combines acoustic features with multilingual features previously learned by the first transformer block. We employed our system in the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge. CCMT was declared the winning solution, obtaining an unweighted average recall of 65.41% and 85.87% for complaint and request detection, respectively. Moreover, we applied our framework on the Speech Commands v2 and HVB dialog data sets, surpassing previous studies reporting results on these benchmarks. Our code is freely available for download at: https://github.com/ristea/ccmt .
ISSN:1573-7462
0269-2821
1573-7462
DOI:10.1007/s10462-024-10869-1