Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

• Published in: IEEE transactions on information forensics and security 2022, Vol.17, p.265-279
• Main Authors: Gratton, Cristiano, Venkategowda, Naveen K. D., Arablouei, Reza, Werner, Stefan
• Format: Article
• Language: English
• Subjects: Accuracy; Algorithms; Alternating direction method of multipliers; Convergence; Convex functions; Cost analysis; Differential privacy; Distributed databases; distributed optimization; Empirical analysis; Lagrangian function; Linear programming; Machine learning; Multiagent systems; Optimization; Perturbation; Privacy; Signal processing algorithms; zeroth-order optimization methods
• ISSN: 1556-6013; 1556-6021; 1556-6021
• DOI: 10.1109/TIFS.2021.3139267

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, by analyzing the inherent randomness due to the use of a zeroth-order method, we show that D-ZOA is intrinsically endowed with (\epsilon,\delta)- differential privacy. In addition, we employ the moments accountant method to show that the total privacy leakage of D-ZOA grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially-private approaches in terms of accuracy while yielding similar privacy guarantee. We prove that D-ZOA reaches a neighborhood of the optimal solution whose size depends on the privacy parameter. The convergence analysis also reveals a practically important trade-off between privacy and accuracy. Simulation results verify the desirable privacy-preserving properties of D-ZOA and its superiority over the state-of-the-art algorithms as well as its network-wide convergence.