A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications

Ensuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressi...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2020-11, Vol.31 (11), p.4776-4790
Main Authors: Chen, Huiming, Wu, Ho-Chun, Chan, Shing-Chow, Lam, Wong-Hing
Format: Article
Language:English
Subjects:
Bayesian analysis
Computer applications
Convergence
Damped parameter
Datasets
Ill-conditioned problems (mathematics)
Learning algorithms
limited memory BFGS (LBFGS)
Linear programming
Logistics
Machine learning
Machine learning algorithms
nonconjugate exponential models
nonconvex optimization
Optimization
Quasi Newton methods
Robustness (mathematics)
Stochastic processes
stochastic quasi-Newton (SQN) method
Stochasticity
variational inference
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Ensuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressing the above problems. While the proposed regularized strategy helps to prevent the BFGS matrix from being close to singularity, the new damped parameter further ensures the positivity of the product of correction pairs. To alleviate the computational cost of the stochastic limited memory BFGS (LBFGS) updates and to improve its robustness, the curvature information is updated using the averaged iterate at spaced intervals. The effectiveness of the proposed method is evaluated through the logistic regression and Bayesian logistic regression problems in machine learning. Numerical experiments are conducted by using both synthetic data set and several real data sets. The results show that the proposed method generally outperforms the stochastic damped LBFGS (SdLBFGS) method. In particular, for problems with small sample sizes, our method has shown superior performance and is capable of mitigating ill-conditioned problems. Furthermore, our method is more robust to the variations of the batch size and memory size than the SdLBFGS method.
ISSN:2162-237X
2162-2388
DOI:10.1109/TNNLS.2019.2957843