Learning to Forget: Continual Prediction with LSTM

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences w...

Full description

Saved in:
Bibliographic Details
Published in:Neural computation 2000-10, Vol.12 (10), p.2451-2471
Main Authors: Gers, Felix A., Schmidhuber, Jürgen, Cummins, Fred
Format: Article
Language:English
Subjects:
Algorithms
Applied sciences
Artificial intelligence
Computer science
control theory
systems
Exact sciences and technology
Memory, Short-Term
Neural Networks (Computer)
Nonlinear Dynamics
Pattern recognition. Digital image processing. Computational geometry
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive “forget gate” that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.
ISSN:0899-7667
1530-888X
DOI:10.1162/089976600300015015