Light Recurrent Unit: Towards an Interpretable Recurrent Neural Network for Modeling Long-Range Dependency

Recurrent neural networks (RNNs) play a pivotal role in natural language processing and computer vision. Long short-term memory (LSTM), as one of the most representative RNNs, is built upon relatively complex architecture with an excessive number of parameters, which results in large storage, high t...

Full description

Saved in:
Bibliographic Details
Published in:Electronics (Basel) 2024-08, Vol.13 (16), p.3204
Main Authors: Ye, Hong, Zhang, Yibing, Liu, Huizhou, Li, Xuannong, Chang, Jiaming, Zheng, Hui
Format: Article
Language:English
Subjects:
Accuracy
Algorithms
Analysis
Computational linguistics
Computer vision
Constraint modelling
Deep learning
Distance learning
Embedded systems
Language processing
Machine vision
Natural language interfaces
Natural language processing
Neural networks
Parameter modification
Real time
Recurrent neural networks
Sequences
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recurrent neural networks (RNNs) play a pivotal role in natural language processing and computer vision. Long short-term memory (LSTM), as one of the most representative RNNs, is built upon relatively complex architecture with an excessive number of parameters, which results in large storage, high training cost, and lousy interpretability. In this paper, we propose a lightweight network called Light Recurrent Unit (LRU). On the one hand, we designed an accessible gate structure, which has high interpretability and addresses the issue of gradient disappearance. On the other hand, we introduce the Stack Recurrent Cell (SRC) structure to modify the activation function, which not only expedites convergence rates but also enhances the interpretability of the network. Experimental results show that our proposed LRU has the advantages of fewer parameters, strong interpretability, and effective modeling ability for variable length sequences on several datasets. Consequently, LRU could be a promising alternative to traditional RNN models in real-time applications with space or time constraints, potentially reducing storage and training costs while maintaining high performance.
ISSN:2079-9292
2079-9292
DOI:10.3390/electronics13163204