Multi-task deep cross-attention networks for far-field speaker verification and keyword spotting

• Published in: EURASIP journal on audio, speech, and music processing speech, and music processing, 2023-07, Vol.2023 (1), p.28-16, Article 28
• Main Authors: Liang, Xingwei, Zhang, Zehua, Xu, Ruifeng
• Format: Article
• Language: English
• Subjects: Acoustics; Coders; Cost function; Customization; Decision theory; Discriminant analysis; Empirical Research; Engineering; Engineering Acoustics; Error detection; Flow attention; Keyword spotting; Keywords; Mathematics in Music; Modules; Multilayers; Music; Neural networks; Personalized voice trigger; Signal,Image and Speech Processing; Speaker verification; Speech; Verification; Voice
• ISSN: 1687-4722; 1687-4714; 1687-4722
• DOI: 10.1186/s13636-023-00293-8

Personalized voice triggering is a key technology in voice assistants and serves as the first step for users to activate the voice assistant. Personalized voice triggering involves keyword spotting (KWS) and speaker verification (SV). Conventional approaches to this task include developing KWS and SV systems separately. This paper proposes a single system called the multi-task deep cross-attention network (MTCANet) that simultaneously performs KWS and SV, while effectively utilizing information relevant to both tasks. The proposed framework integrates a KWS sub-network and an SV sub-network to enhance performance in challenging conditions such as noisy environments, short-duration speech, and model generalization. At the core of MTCANet are three modules: a novel deep cross-attention (DCA) module to integrate KWS and SV tasks, a multi-layer stacked shared encoder (SE) to reduce the impact of noise on the recognition rate, and soft attention (SA) modules to allow the model to focus on pertinent information in the middle layer while preventing gradient vanishing. Our proposed model demonstrates outstanding performance in the well-off test set, improving by 0.2%, 0.023, and 2.28% over the well-known SV model emphasized channel attention, propagation, and aggregation in time delay neural network (ECAPA-TDNN) and the advanced KWS model Convmixer in terms of equal error rate (EER), minimum detection cost function (minDCF), and accuracy (Acc), respectively.