FedSynth: Gradient Compression via Synthetic Data in Federated Learning

Model compression is important in federated learning (FL) with large models to reduce communication cost. Prior works have been focusing on sparsification based compression that could desparately affect the global model accuracy. In this work, we propose a new scheme for upstream communication where...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-04
Main Authors: Hu, Shengyuan, Goetz, Jack, Malik, Kshitiz, Zhan, Hongyuan, Liu, Zhe, Liu, Yue
Format: Article
Language:English
Subjects:
Algorithms
Datasets
Federated learning
Model accuracy
Synthetic data
Weight reduction
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Model compression is important in federated learning (FL) with large models to reduce communication cost. Prior works have been focusing on sparsification based compression that could desparately affect the global model accuracy. In this work, we propose a new scheme for upstream communication where instead of transmitting the model update, each client learns and transmits a light-weight synthetic dataset such that using it as the training data, the model performs similarly well on the real training data. The server will recover the local model update via the synthetic data and apply standard aggregation. We then provide a new algorithm FedSynth to learn the synthetic data locally. Empirically, we find our method is comparable/better than random masking baselines in all three common federated learning benchmark datasets.
ISSN:2331-8422