Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches

• Published in: Journal of global optimization 2021, Vol.79 (1), p.111-152
• Main Authors: Kafka, Dominic, Wilke, Daniel N.
• Format: Article
• Language: English
• Subjects: Computer Science; Convergence; Iterative methods; Learning; Mathematics; Mathematics and Statistics; Neural networks; Operations Research/Decision Theory; Optimization; Real Functions; Searching; Training
• ISSN: 0925-5001; 1573-2916
• DOI: 10.1007/s10898-020-00921-z

Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. Attempts to resolve learning rates adaptively, using line searches, have proven computationally demanding. Reducing the computational cost by considering mini-batch sub-sampling (MBSS) introduces challenges due to significant variance in information between batches that may present as discontinuities in the loss function, depending on the MBSS approach. This study proposes a robust approach to adaptively resolve learning rates in dynamic MBSS loss functions. This is achieved by finding sign changes from negative to positive along directional derivatives, which ultimately converge to a stochastic non-negative associated gradient projection point. Through a number of investigative studies, we demonstrate that gradient-only line searches (GOLS) resolve learning rates adaptively, improving convergence performance over minimization line searches, ignoring certain local minima and eliminating an otherwise expensive hyperparameter. We also show that poor search directions may benefit computationally from overstepping optima along a descent direction, which can be resolved by considering improved search directions. Having shown that GOLS is a reliable line search allows for comparative investigations between static and dynamic MBSS.