Structural Properties of Recurrent Neural Networks

• Published in: Neural processing letters 2009-04, Vol.29 (2), p.75-88
• Main Authors: Dobnikar, Andrej, Šter, Branko
• Format: Article
• Language: English
• Subjects: Applied sciences; Artificial Intelligence; Clustering; Combinatorics; Combinatorics. Ordered structures; Complex Systems; Computational Intelligence; Computer Science; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Connectionism. Neural networks; Exact sciences and technology; Graph theory; Graphs; Information retrieval. Graph; Mathematics; Nodes; Recurrent neural networks; Sciences and techniques of general use; Software; Theoretical computing
• ISSN: 1370-4621; 1573-773X
• DOI: 10.1007/s11063-009-9096-2

In this article we research the impact of the adaptive learning process of recurrent neural networks (RNN) on the structural properties of the derived graphs. A trained fully connected RNN can be converted to a graph by defining edges between pairs od nodes having significant weights. We measured structural properties of the derived graphs, such as characteristic path lengths, clustering coefficients and degree distributions. The results imply that a trained RNN has significantly larger clustering coefficient than a random network with a comparable connectivity. Besides, the degree distributions show existence of nodes with a large degree or hubs, typical for scale-free networks. We also show analytically and experimentally that this type of degree distribution has increased entropy.