Preconditioned Stochastic Gradient Descent

• Published in: IEEE transaction on neural networks and learning systems 2018-05, Vol.29 (5), p.1454-1466
• Main Author: Li, Xi-Lin
• Format: Article
• Language: English
• Subjects: Acceleration; Approximation; Convergence; Eigenvalues and eigenfunctions; Moisture content; Neural network; Neural networks; Newton method; Newton methods; nonconvex optimization; Optimization; preconditioner; Recurrent neural networks; stochastic gradient descent (SGD); Stochasticity; Training
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2017.2672978

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in significantly more complicated methods than SGD. This paper proposes a new method to adaptively estimate a preconditioner, such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to Newton method for deterministic optimization. Unlike the preconditioners based on secant equation fitting as done in deterministic quasi-Newton methods, which assume positive definite Hessian and approximate its inverse, the new preconditioner works equally well for both convex and nonconvex optimizations with exact or noisy gradients. When stochastic gradient is used, it can naturally damp the gradient noise to stabilize SGD. Efficient preconditioner estimation methods are developed, and with reasonable simplifications, they are applicable to large-scale problems. Experimental results demonstrate that equipped with the new preconditioner, without any tuning effort, preconditioned SGD can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long-term memories.