Think Locally, Act Globally: Federated Learning with Local and Global Representations

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated lear...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-07
Main Authors: Paul Pu Liang, Liu, Terrance, Liu Ziyin, Allen, Nicholas B, Auerbach, Randy P, Brent, David, Salakhutdinov, Ruslan, Louis-Philippe Morency
Format: Article
Language:English
Subjects:
Communication
Electronic devices
Empirical analysis
Federated learning
Mathematical models
Parameters
Representations
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.
ISSN:2331-8422