Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices

Federated Learning (FL) is a technique to train models on distributed edge devices with local data samples. Differential Privacy (DP) can be applied with FL to provide a formal privacy guarantee for sensitive data on device. Our goal is to train a large neural network language model (NNLM) on comput...

Full description

Saved in:
Bibliographic Details
Main Authors: Xu, Mingbin, Song, Congzheng, Tian, Ye, Agrawal, Neha, Granqvist, Filip, van Dalen, Rogier, Zhang, Xiao, Argueta, Arturo, Han, Shiyi, Deng, Yaqiao, Liu, Leo, Walia, Anmol, Jin, Alex
Format: Conference Proceeding
Language:English
Subjects:
Adaptation models
Computational modeling
Differential privacy
Federated learning
Privacy
Training
Vocabulary
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Federated Learning (FL) is a technique to train models on distributed edge devices with local data samples. Differential Privacy (DP) can be applied with FL to provide a formal privacy guarantee for sensitive data on device. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the noise required to guarantee differential privacy increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to reduce the impact of DP-noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. We demonstrate in simulation and with real devices that this combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.
ISSN:2379-190X
DOI:10.1109/ICASSP49357.2023.10096570