Take Your Pick: Enabling Effective Distributed Learning Within Low-Dimensional Feature Space

• Published in: IEEE transaction on neural networks and learning systems 2024-11, p.1-15
• Main Authors: Zhu, Guogang, Liu, Xuefeng, Tang, Shaojie, Niu, Jianwei, Wu, Xinghao, Shen, Jiaxing, Lin, Wanyu
• Format: Article
• Language: English
• Subjects: Accuracy; Adaptation models; Computer aided instruction; Data models; Distance learning; Distributed databases; Distributed learning; Feature extraction; feature selection; Load modeling; low-dimensional feature space; personalized federated learning (PFL); Solid modeling; Training
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2024.3498460

Personalized federated learning (PFL) is a popular distributed learning framework that allows clients to have different models and has many applications where clients' data are in different domains, including autonomous driving, traffic surveillance, and medical diagnosis. The typical model of a client in PFL features a global encoder trained by all clients to extract universal features from the raw data and personalized layers (e.g., a classifier) trained using the client's local data. Nonetheless, due to the differences between the data distributions of different clients (also known as, domain gaps), the universal features produced by the global encoder largely encompass numerous components irrelevant to a certain client's local task. Some recent PFL methods address the above problem by personalizing specific parameters within the encoder. However, these methods encounter substantial challenges attributed to the high dimensionality and nonlinearity of neural network parameter space. In contrast, the feature space exhibits a lower dimensionality, providing greater intuitiveness and interpretability as compared to the parameter space. To this end, we propose a novel PFL framework named FedPick. FedPick achieves PFL within the low-dimensional feature space by adaptively selecting task-relevant features for each client from the features generated by the global encoder based on its local data distribution. It presents a more accessible and interpretable implementation of PFL compared to those methods working in the parameter space. Extensive experimental results on multiple cross-domain datasets show that FedPick can effectively select task-relevant features for each client and improve model performance in cross-domain FL.