fastESN: Fast Echo State Network

Echo state networks (ESNs) are reservoir computing-based recurrent neural networks widely used in pattern analysis and machine intelligence applications. In order to achieve high accuracy with large model capacity, ESNs usually contain a large-sized internal layer (reservoir), making the evaluation...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2023-12, Vol.34 (12), p.10487-10501
Main Authors: Wang, Hai, Long, Xingyi, Liu, Xue-Xin
Format: Article
Language:English
Subjects:
Acceleration
Benchmarks
Biological neural networks
complexity reduction
Complexity theory
Compression ratio
echo state networks (ESNs)
Empirical analysis
Evaluation
Interpolation
Neural networks
Pattern analysis
Principal component analysis
Proper Orthogonal Decomposition
Recurrent neural networks
recurrent neural networks (RNNs)
reservoir computing
Reservoirs
Training
Very large scale integration
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Echo state networks (ESNs) are reservoir computing-based recurrent neural networks widely used in pattern analysis and machine intelligence applications. In order to achieve high accuracy with large model capacity, ESNs usually contain a large-sized internal layer (reservoir), making the evaluation process too slow for some applications. In this work, we speed up the evaluation of ESN by building a reduced network called the fast ESN (fastESN) and achieve an ESN evaluation complexity independent of the original ESN size for the first time. FastESN is generated using three techniques. First, the high-dimensional state of the original ESN is approximated by a low-dimensional state through proper orthogonal decomposition (POD)-based projection. Second, the activation function evaluation number is reduced through the discrete empirical interpolation method (DEIM). Third, we show the directly generated fastESN has instability problems and provide a stabilization scheme as a solution. Through experiments on four popular benchmarks, we show that fastESN is able to accelerate the sparse storage-based ESN evaluation with a high parameter compression ratio and a fast evaluation speed.
ISSN:2162-237X
2162-2388
DOI:10.1109/TNNLS.2022.3167466