An advanced active set L-BFGS algorithm for training weight-constrained neural networks

• Published in: Neural computing & applications 2020-06, Vol.32 (11), p.6669-6684
• Main Author: Livieris, Ioannis E.
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Convergence; Data Mining and Knowledge Discovery; Error functions; Image Processing and Computer Vision; Neural networks; Probability and Statistics in Computer Science; S.I. : Brain inspired Computing&Machine Learning Applied Research-BISMLARE; Training; Weight
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-019-04689-6

In this work, a new advanced active set limited memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is proposed for efficiently training weight-constrained neural networks, called AA-L-BFGS. The proposed algorithm possesses the significant property of approximating the curvature of the error function with high-order accuracy by utilizing the theoretically advanced secant condition proposed by Livieris and Pintelas (Appl Math Comput 221:491–502, 2013). Moreover, the global convergence of the proposed algorithm is established provided that the line search satisfies the modified Armijo condition. The presented numerical experiments illustrate the efficiency of the proposed AA-L-BFGS, providing empirical evidence that it significantly accelerates the convergence of the training process.