Manifold Proximal Point Algorithms for Dual Principal Component Pursuit and Orthogonal Dictionary Learning

• Published in: IEEE transactions on signal processing 2021, Vol.69, p.4759-4773
• Main Authors: Chen, Shixiang, Deng, Zengde, Ma, Shiqian, So, Anthony Man-Cho
• Format: Article
• Language: English
• Subjects: Algorithms; Convergence; Dictionaries; dictionary learning; Heuristic methods; Iterative methods; Machine learning; Manifolds; Manifolds (mathematics); Optimization; proximal point algorithm; Riemannian optimization; Robustness (mathematics); Signal processing algorithms; Smoothing methods; subspace recovery
• ISSN: 1053-587X; 1941-0476
• DOI: 10.1109/TSP.2021.3099643

We consider the problem of minimizing the \ell _1 norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms with provable guarantees for tackling this problem. Specifically, our contribution is fourfold. First, we present a manifold proximal point algorithm (ManPPA) for the problem and show that it converges at a global sublinear rate. Furthermore, we show that ManPPA can achieve a local quadratic convergence rate when applied to sharp instances of the problem. Second, we develop a semismooth Newton-based inexact augmented Lagrangian method for computing the search direction in each iteration of ManPPA and show that it has an asymptotic superlinear convergence rate. Third, we propose a stochastic variant of ManPPA called StManPPA, which is well suited for large-scale computation, and establish its sublinear convergence rate. Both ManPPA and StManPPA have provably faster convergence rates than existing subgradient-type methods. Fourth, using ManPPA as a building block, we propose a new heuristic method for solving a matrix analog of the problem, in which the sphere is replaced by the Stiefel manifold. The results from our extensive numerical experiments on the ODL and RSR problems demonstrate the efficiency and efficacy of our proposed methods.