Overcoming Client Data Deficiency in Federated Learning by Exploiting Unlabeled Data on the Server

Federated Learning (FL) is a distributed machine learning paradigm involving multiple clients to train a server model. In practice, clients often possess limited data and are not always available for simultaneous participation in FL, which can lead to data deficiency. This data deficiency degrades t...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2024, Vol.12, p.130007-130021
Main Authors: Park, Jae-Min, Jang, Won-Jun, Oh, Tae-Hyun, Lee, Si-Hyeon
Format: Article
Language:English
Subjects:
Accuracy
Clients
Data models
ensemble distillation
Entropy
Federated Learning
knowledge distillation
Knowledge transfer
Labels
Machine learning
Self-supervised learning
Servers
Uncertainty
Weighting methods
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Federated Learning (FL) is a distributed machine learning paradigm involving multiple clients to train a server model. In practice, clients often possess limited data and are not always available for simultaneous participation in FL, which can lead to data deficiency. This data deficiency degrades the entire learning process. To address this, we propose Federated learning with entropy-weighted ensemble Distillation and Self-supervised learning (FedDS). FedDS effectively handles situations with limited data per client and few clients. The key idea is to exploit the unlabeled data available on the server in the aggregating step of client models into a server model. We distill the multiple client models to a server model in an ensemble way. To robustly weigh the quality of source pseudo-labels from the client models, we propose an entropy weighting method and show a favorable tendency that our method assigns higher weights to more accurate predictions. Furthermore, we jointly leverage a separate self-supervised loss for improving generalization of the server model. We demonstrate the effectiveness of our FedDS both empirically and theoretically. For CIFAR-10, our method shows an improvement over FedAVG of 12.54% in the data deficient regime, and of 17.16% and 23.56% in the more challenging scenarios of noisy label or Byzantine client cases, respectively. For CIFAR-100 and ImageNet-100, our method shows an improvement over FedAVG of 18.68% and 15.06% in the data deficient regime, respectively.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2024.3458911