Gated Recurrent Context: Softmax-Free Attention for Online Encoder-Decoder Speech Recognition

Recently, attention-based encoder-decoder (AED) models have shown state-of-the-art performance in automatic speech recognition (ASR). As the original AED models with global attentions are not capable of online inference, various online attention schemes have been developed to reduce ASR latency for...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on audio, speech, and language processing speech, and language processing, 2021, Vol.29, p.710-719
Main Authors: Lee, Hyeonseung, Kang, Woo Hyun, Cheon, Sung Jun, Kim, Hyeongju, Kim, Nam Soo
Format: Article
Language:English
Subjects:
Attention
Attention-based encoder-decoder model
Automatic speech recognition
Coders
Decoding
Encoders-Decoders
Error analysis
Hidden Markov models
Inference
Logic gates
online speech recognition
Predictive models
Speech recognition
Training
Tuning
Voice recognition
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recently, attention-based encoder-decoder (AED) models have shown state-of-the-art performance in automatic speech recognition (ASR). As the original AED models with global attentions are not capable of online inference, various online attention schemes have been developed to reduce ASR latency for better user experience. However, a common limitation of the conventional softmax-based online attention approaches is that they introduce an additional hyperparameter related to the length of the attention window, requiring multiple trials of model training for tuning the hyperparameter. In order to deal with this problem, we propose a novel softmax-free attention method and its modified formulation for online attention, which does not need any additional hyperparameter at the training phase. Through a number of ASR experiments, we demonstrate the tradeoff between the latency and performance of the proposed online attention technique can be controlled by merely adjusting a threshold at the test phase. Furthermore, the proposed methods showed competitive performance to the conventional global and online attentions in terms of word-error-rates (WERs).
ISSN:2329-9290
2329-9304
DOI:10.1109/TASLP.2021.3049344