Using Euler's Method to Prove the Convergence of Neural Networks

It was shown in the literature that, for a fully connected neural network (NN), the gradient descent algorithm converges to zero. Motivated by that work, we provide here general conditions under which we can derive the convergence of the gradient descent algorithm from the convergence of the gradien...

Full description

Saved in:
Bibliographic Details
Published in:IEEE control systems letters 2022-01, Vol.6, p.3224-3228
Main Authors: Jerray, Jawher, Saoud, Adnane, Fribourg, Laurent
Format: Article
Language:English
Subjects:
Artificial neural networks
Automatic
Convergence
Eigenvalues and eigenfunctions
Engineering Sciences
Euler’s method
gradient descent
Heuristic algorithms
Neural networks
Systematics
Training data
Writing
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It was shown in the literature that, for a fully connected neural network (NN), the gradient descent algorithm converges to zero. Motivated by that work, we provide here general conditions under which we can derive the convergence of the gradient descent algorithm from the convergence of the gradient flow, in the case of NNs, in a systematic way. Our approach is based on an analysis of the error in Euler's method in the case of NNs, and relies on the concept of local strong convexity. Unlike existing approaches in the literature, our approach allows to provide convergence guarantees without making any assumptions on the number of hidden nodes of the NN or the number of training data points. A numerical example is proposed, showing the merits of our approach.
ISSN:2475-1456
2475-1456
DOI:10.1109/LCSYS.2022.3184040