Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm

Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods...

Full description

Saved in:
Bibliographic Details
Main Authors: Elbakary, Ahmed, Issaid, Chaouki Ben, Shehab, Mohammad, Seddik, Karim, ElBatt, Tamer, Bennis, Mehdi
Format: Conference Proceeding
Language:English
Subjects:
communication efficiency
deep neural networks
Distributed optimization
Estimation
Federated learning
Numerical models
Robustness
Scalability
second-order methods
Servers
Task analysis
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines.
ISSN:1938-1883
DOI:10.1109/ICC51166.2024.10622862