Another Approach to Build Lyapunov Functions for the First Order Methods in the Quadratic Case

Lyapunov functions play a fundamental role in analyzing the stability and convergence properties of optimization methods. In this paper, we propose a novel and straightforward approach for constructing Lyapunov functions for first-order methods applied to quadratic functions. Our approach involves b...

Full description

Saved in:
Bibliographic Details
Published in:Computational mathematics and mathematical physics 2024, Vol.64 (4), p.788-805
Main Authors: Merkulov, D. M., Oseledets, I. V.
Format: Article
Language:English
Subjects:
Computational Mathematics and Numerical Analysis
Convergence
Convex analysis
Convexity
Decomposition
Iterative methods
Liapunov functions
Mathematics
Mathematics and Statistics
Optimal Control
Optimization
Quadratic equations
Quadratic programming
Smoothness
State vectors
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Lyapunov functions play a fundamental role in analyzing the stability and convergence properties of optimization methods. In this paper, we propose a novel and straightforward approach for constructing Lyapunov functions for first-order methods applied to quadratic functions. Our approach involves bringing the iteration matrix to an upper triangular form using Schur decomposition, then examining the value of the last coordinate of the state vector. This value is multiplied by a magnitude smaller than one at each iteration. Consequently, this value should decrease at each iteration, provided that the method converges. We rigorously prove the suitability of this Lyapunov function for all first-order methods and derive the necessary conditions for the proposed function to decrease monotonically. Experiments conducted with general convex functions are also presented, alongside a study on the limitations of the proposed approach. Remarkably, the newly discovered L-yapunov function is straightforward and does not explicitly depend on the exact method formulation or function characteristics like strong convexity or smoothness constants. In essence, a single expression serves as a Lyapunov function for several methods, including Heavy Ball, Nesterov Accelerated Gradient, and Triple Momentum, among others. To the best of our knowledge, this approach has not been previously reported in the literature.
ISSN:0965-5425
1555-6662
DOI:10.1134/S0965542524700131