Fed2PKD: Bridging Model Diversity in Federated Learning via Two-Pronged Knowledge Distillation

• Main Authors: Xie, Zaipeng, Xu, Han, Gao, Xing, Jiang, Junchen, Han, Ruiqian
• Format: Conference Proceeding
• Language: English
• Subjects: Adaptation models; Cloud computing; Computational modeling; data heterogeneity; Data privacy; Federated learning; Heterogeneous federated learning; Knowledge engineering; model diver-sity; Prototypes; two-pronged knowledge distillation
• ISSN: 2159-6190
• DOI: 10.1109/CLOUD62652.2024.00011

Heterogeneous federated learning (HFL) enables collaborative learning across clients with diverse model architectures and data distributions while preserving privacy. However, existing HFL approaches often struggle to effectively address the challenges posed by model diversity, leading to suboptimal performance and limited generalization ability. This paper pro-poses Fed2PKD, a novel HFL framework that tackles these challenges through a two-pronged knowledge distillation approach. Fed2PKD combines prototypical contrastive knowledge distillation to align client embeddings with global class prototypes and semi-supervised global knowledge distillation to capture global data characteristics. Experimental results on three benchmarks (MNIST, CIFAR10, and CIFAR100) demonstrate that Fed2PKD significantly outperforms existing state-of-the-art HFL methods, achieving average improvements of up to 30.53%, 13.89%, and 5.80 % in global model accuracy, respectively. Furthermore, Fed2PKD enables personalized models for each client, adapting to their specific data distributions and model architectures while benefiting from global knowledge sharing. Theoretical analysis provides convergence guarantees for Fed2PKD under realistic assumptions. Fed2PKD represents a significant step forward in HFL, unlocking the potential for privacy-preserving collaborative learning in real-world scenarios with model and data diversity.