Energy-information trade-off induces continuous and discontinuous phase transitions in lateral predictive coding

• Published in: Science China. Physics, mechanics & astronomy mechanics & astronomy, 2024-06, Vol.67 (6), p.260511, Article 260511
• Main Authors: Huang, Zhen-Ye, Zhou, Ruyi, Huang, Miao, Zhou, Hai-Jun
• Format: Article
• Language: English
• Subjects: Astronomy; Classical and Continuum Physics; Coding; Coding theory; Eigenvalues; Energy; Energy consumption; Energy efficiency; Entropy; Free energy; Ideal gas; Neural networks; Observations and Techniques; Oil and gas laws; Permutations; Phase transitions; Physics; Physics and Astronomy; Recurrent neural networks; Temperature; Theoretical physics; Tradeoffs; Upper bounds
• ISSN: 1674-7348; 1869-1927
• DOI: 10.1007/s11433-024-2341-2

Lateral predictive coding is a recurrent neural network that creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here, we analytically investigate the trade-off between information robustness and energy in a linear model of lateral predictive coding and numerically minimize a free energy quantity. We observed several phase transitions in the synaptic weight matrix, particularly a continuous transition that breaks reciprocity and permutation symmetry and builds cyclic dominance and a discontinuous transition with the associated sudden emergence of tight balance between excitatory and inhibitory interactions. The optimal network follows an ideal gas law over an extended temperature range and saturates the efficiency upper bound of energy use. These results provide theoretical insights into the emergence and evolution of complex internal models in predictive processing systems.