The Entropy Economy and the Kolmogorov Learning Cycle: Leveraging the intersection of Machine Learning and Algorithmic Information Theory to jointly optimize energy and learning

• Published in: Physica. D 2024-05, Vol.461 (C), p.134051, Article 134051
• Main Authors: Evans, Scott C., Shah, Tapan, Huang, Hao, Ekanayake, Sachini Piyoni
• Format: Article
• Language: English
• Subjects: Algorithmic Information Theory; Entropy Economy; Kolmogorov complexity; Kolmogorov structure function; Machine learning
• ISSN: 0167-2789; 1872-8022
• DOI: 10.1016/j.physd.2024.134051

•Joint op(miza(on of energy and learning has poten(al to reduce carbon 30 %.•Entropy economy op(mized where, when, and how AI/ML models are created.•Less-Energy-Usage network (LEAN) minimizes deep network energy costs.•Kolmogorov structure func(on to achieve energy efficient, generalizable AI/ML models.•Kolmogorov learning cycle to explain the thermodynamic of machine learning. We augment the Kolmogorov Structure Function with energy cost and drive the concept of “Additive AI” where Machine Learning Models are created by traversing the Kolmogorov Structure function from low model complexity to high while seeking models achieving the Kolmogorov Minimum Sufficient Statistic with least energy cost. In this way, the intersection of Algorithmic Information Theory (AIT) with Machine Learning (ML) can enable optimization of the “Entropy Economy,” where the precious resource of entropy flow is managed to jointly optimize computation, energy, and learning. In this paper we lay out the Kolmogorov Learning Cycle as a framework for this joint optimization and demonstrate the energy efficient machine learning algorithm Least Energy Usage Network (LEAN) as an example of how restraining complexity can reduce learning energy cost while maintaining performance. We motivate further directions for optimizing how AI models can be optimally learned and discuss additional opportunities to optimize where and when AI and machine learning models can be created to maximize learning while minimizing energy (and subsequently carbon costs) through the intersection of AIT and ML.