FedSR: A Semi-Decentralized Federated Learning Framework for Non-IID Data Based on Incremental Subgradient Optimization

In the era of computational social systems (CSS), where AI is increasingly embedded in critical domains such as public services, healthcare, and social governance, concerns about system vulnerabilities have become paramount. Federated learning (FL) offers a promising solution by enabling privacy-preserving and secure AI model training in real-world applications. However, traditional FL approaches often assume that client data is either imbalanced or non-independent and identically distributed (non-IID), whereas in practice, these issues frequently coexist. In such complex scenarios, minority classes may appear only on a small subset of clients, significantly degrading global model performance and convergence. In this article, we propose a novel FL framework designed to tackle the intertwined challenges of data imbalance and non-IID data by a principled, decoupled approach. Unlike existing methods that address these issues in isolation, our approach introduces three synergistic innovations, each operating at a different level of the FL system. At the data level, a global rebalancing mechanism employs a privacy-preserving modified Z-score to rectify class imbalance. At the client level, a distribution-aware clustering strategy groups participants based on data subspace similarity. At the feature level, a novel contrastive loss function, leveraging the effective number of samples, simultaneously mitigates local imbalance and model drift. This cohesive framework ensures that interventions at each level reinforce each other, leading to robust performance. Extensive experiments show that FedScFc outperforms state-of-the-art baselines.

Federated learning (FL) is a new machine learning paradigm to collaboratively learn an intelligent model across many clients without uploading local data to the server. Non-IID data across clients is a major challenge for the FL system because its inherited distributed machine learning framework is designed for the scenario of IID data across clients. Clustered FL is a type of FL method to solve non-IID challenges using a client clustering method in the FL context. However, existing clustered FL methods suffer the challenge of processing client-wise outliers which could be produced by minority clients with abnormal behaviour patterns or be derived from malicious clients. This paper is to propose a novel Federated learning framework with Robust Clustering (FedRoC) to tackle client-wise outliers in the FL system. Specifically, we will develop a robust federated aggregation operator using a bootstrap median-of-means mechanism that can produce a higher breakdown point to tolerate a larger proportion of outliers. We formulate the proposed FL framework into a bi-level optimization problem, and then a stochastic expectation-maximization method is adopted to solve the optimization problem in an alternative updating manner by considering EM steps and distributed computing simultaneously. The experiments on three benchmark datasets have demonstrated the effectiveness of the proposed method that outperforms other baseline methods in terms of evaluation criteria.KeywordsFederated learningRobust clusteringBootstrap median-of-means

The rapid advancement in technology has simplified human life and provides convenience. However, this convenience has led to many lifestyle diseases like diabetes and obesity. The 2021 reports of the International Diabetes Federation (IDF) show 537 million diabetics, three-fourths of whom are from developing countries. About 28.5% of these diabetics over 40 suffer from Diabetic Retinopathy (DR), 4.4% face vision-threatening DR, and 3.8% suffer from Diabetic Macular Edema (DME). These conditions can lead to complete vision loss affecting health and quality of life. Early detection of DR and DME is crucial to prevent harmful effects. The proposed work employs a collaborative, privacy-preserving Federated Deep Learning (FDL) framework with lightweight MobileNetV2 architecture for early detection of DR and DME. The proposed FDL framework uses both Independent and Identically Distributed (IID) and non-IID data for 2 and 3-client architectures. In a 2-client scenario, the FDL implementation with FedAvg aggregation achieved 98.69% accuracy on IID data and 87.09% on non-IID data, while FedProx aggregation scored 98.03% on IID data and 98.25% on non-IID data. In a 3-client scenario, FedAvg aggregation achieved 97.62% accuracy on IID data and 96.28% on non-IID data, whereas FedProx aggregation achieved 98.69% on IID data and 97.77% on non-IID data. The results demonstrate that the FedProx aggregation is more stable and converges earlier than FedAvg aggregation in the non-IID settings of an FDL framework. The proposed FDL framework with its collaborative training feature, preserves privacy and maintains high prediction accuracy.

Federated Learning (FL) aims to protect data privacy by enabling clients to collectively train machine learning models without sharing their raw data. However, recent studies demonstrate that information exchanged during FL is subject to Gradient Inversion Attacks (GIA) and, consequently, a variety of privacy-preserving methods have been integrated into FL to thwart such attacks, such as Secure Multi-party Computing (SMC), Homomorphic Encryption (HE), and Differential Privacy (DP). Despite their ability to protect data privacy, these approaches inherently involve substantial privacy-utility trade-offs. By revisiting the key to privacy exposure in FL under GIA, which lies in the frequent sharing of model gradients that contain private data, we take a new perspective by designing a novel privacy preserve FL framework that effectively ``breaks the direct connection'' between the shared parameters and the local private data to defend against GIA. Specifically, we propose a Hypernetwork Federated Learning (HyperFL) framework that utilizes hypernetworks to generate the parameters of the local model and only the hypernetwork parameters are uploaded to the server for aggregation. Theoretical analyses demonstrate the convergence rate of the proposed HyperFL, while extensive experimental results show the privacy-preserving capability and comparable performance of HyperFL.

A large number of mobile devices, smart wearable devices, and medical and health sensors continue to generate massive amounts of data, making edge devices’ data explode and making it possible to implement data-driven artificial intelligence. However, the “data silos” and other issues still exist and need to be solved. Fortunately, federated learning (FL) can deal with “data silos” in the medical field, facilitating collaborative learning across multiple institutions without sharing local data and avoiding user concerns about data privacy. However, it encounters two main challenges in the medical field. One is statistical heterogeneity, also known as non-IID (non-independent and identically distributed) data, i.e., data being non-IID between clients, which leads to model drift. The second is limited labeling because labels are hard to obtain due to the high cost and expertise requirement. Most existing federated learning algorithms only allow for supervised training settings. In this work, we proposed a novel federated learning framework, MixFedGAN, to tackle the above issues in federated networks with dynamic aggregation and knowledge distillation. A dynamic aggregation scheme was designed to reduce the impact of current low-performing clients and improve stability. Knowledge distillation was introduced into the local generator model with a new distillation regularization loss function to prevent essential parameters of the global generator model from significantly changing. In addition, we considered two scenarios under this framework: complete annotated data and limited labeled data. An experimental analysis on four heterogeneous COVID-19 infection segmentation datasets and three heterogeneous prostate MRI segmentation datasets verified the effectiveness of the proposed federated learning method.

The trade-off between privacy and accuracy presents a challenge for current federated learning (FL) frameworks, hindering their progress from theory to application. The main issues with existing FL frameworks stem from a lack of interpretability and targeted privacy protections. To cope with these, we proposed Disentangled Federated Learning for Privacy (DFLP) which employes disentanglement, one of interpretability techniques, in private FL frameworks. Since sensitive properties are client-specific in nature, our main idea is to turn this feature into a tool that strikes the balance between data privacy and FL model performance, enabling the sensitive attributes to be private. DFLP disentangles the client-specific and class-invariant attributes to mask the sensitive attributes precisely. To our knowledge, this is the first work that successfully integrates disentanglement and the nature of sensitive attributes to achieve privacy protection while ensuring high FL model performance. Extensive experiments validate that disentanglement is an effective method for accuracy-aware privacy protection in FL frameworks.

Federated learning (FL) is widely used in privacy sensitive applications for isolated data islands, with the aim of achieving distributed model training, privacy enhancement and model sharing. Electromyographic (EMG) signals are a type of data collected from wearable sensors of subjects which are distributed on multiple devices, highly personalized and play an important role in several applications including prosthetic hand control, sign languages, grasp recognition, etc. This paper utilizes the FL method to detect single and combined finger movements based on EMG signals. The existing research on FL for wearable healthcare faces challenges of variable probability distributions of data, the need for prerequisite knowledge of server model and computational burdens in parameter transmission. To address these problems, this paper proposes a communication efficient FL framework in which each device only needs to transmit the weight matrices of local models to the server for model aggregation. To further reduce the FL transmission delay, a joint learning and resource allocation problem is formulated via optimizing transmit power of each device, time allocation, and user selection. To solve the delay minimization problem, the objective function is first converted to a tractable expression and then the difference of two convex functions programming is adopted. Simulation results using real EMG signals show that the proposed FL framework with personalized training process successfully detects single and combined finger movements for distributed users. Two public EMG datasets with 10 and 15 different finger movements are employed. Over 98% overall test accuracy is achieved in both datasets which surpasses the conventional learning framework by 1.6% and 0.5% on average. Different scenarios with regard to access points and users are investigated and the convexity of the proposed model is discussed.

Within the federated learning (FL) framework, the client collaboratively trains the model in coordination with a central server, while the training data can be kept locally on the client. Thus, the FL framework mitigates the privacy disclosure and costs related to conventional centralized machine learning. Nevertheless, current surveys indicate that FL still has problems in terms of communication efficiency and privacy risks. In this paper, to solve these problems, we develop an FL framework with communication-efficient and privacy-preserving (FLCP). To realize the FLCP, we design a novel compression algorithm with efficient communication, namely, adaptive weight compression FedAvg (AWC-FedAvg). On the basis of the non-independent and identically distributed (non-IID) and unbalanced data distribution in FL, a specific compression rate is provided for each client, and homomorphic encryption (HE) and differential privacy (DP) are integrated to provide demonstrable privacy protection and maintain the desirability of the model. Therefore, our proposed FLCP smoothly balances communication efficiency and privacy risks, and we prove its security against “honest-but-curious” servers and extreme collusion under the defined threat model. We evaluate the scheme by comparing it with state-of-the-art results on the MNIST and CIFAR-10 datasets. The results show that the FLCP performs better in terms of training efficiency and model accuracy than the baseline method.

This paper proposes a novel scheme for cell-free massive multiple-input multiple-output (CFmMIMO) networks to support any federated learning (FL) framework. This scheme allows each instead of all the iterations of the FL framework to happen in a large-scale coherence time to guarantee a stable operation of an FL process. To show how to optimize the FL performance using this proposed scheme, we consider an existing FL framework as an example and target FL training time minimization for this framework. An optimization problem is then formulated to jointly optimize the local accuracy, transmit power, data rate, and users' processing frequency. This mixed-timescale stochastic nonconvex problem captures the complex interactions among the training time, and transmission and computation of training updates of one FL process. By employing the online successive convex approximation approach, we develop a new algorithm to solve the formulated problem with proven convergence to the neighbourhood of its stationary points. Our numerical results confirm that the presented joint design reduces the training time by up to 55% over baseline approaches. They also show that CFmMIMO here requires the lowest training time for FL processes compared with cell-free time-division multiple access massive MIMO and collocated massive MIMO.

Federated learning (FL) empowers a cohort of participating devices to contribute collaboratively to a global neural network model, ensuring that their training data remains private and stored locally. Despite its advantages in computational efficiency and privacy preservation, FL grapples with the challenge of non-IID (not independent and identically distributed) data from diverse clients, leading to discrepancies between local and global models and potential performance degradation. In this article, we propose FedGK, an innovative communication-efficient Group-Guided FL framework designed for heterogeneous data distributions. FedGK employs a localized-guided framework that enables the client to effectively assimilate key knowledge from teachers and peers while minimizing extraneous peer information in FL scenarios. We conduct an in-depth analysis of the dynamic similarities among clients over successive communication rounds and develop a novel clustering approach that accurately groups clients with diverse heterogeneities. We implement FedGK on public datasets with an innovative data transformation pattern called “cluster-shift non-IID”, which mirrors the more prevalent data distributions in real-world settings and could be grouped into clusters with similar data distributions. Extensive experimental results on public datasets demonstrate that the proposed approach FedGK improves accuracy by up to 32.89% and saves up to 53.33% communication cost over state-of-the-art baselines.

Federated learning (FL) enables privacy-preserving distributed learning but faces significant communication bottle-necks, especially for resource-constrained edge devices. Existing approaches often incur additional computational cost or require auxiliary data. To address these problems, we propose FedSNN-QC, a lightweight FL framework leveraging Spiking Neural Networks (SNNs) for their efficiency and robustness to Non-IID and noisy data. The framework incorporates a gradient quantization scheme exploiting SNN sparsity to minimize transmission and joint gradient clipping techniques to enhance robustness and convergence. Experimental results demonstrate that the FedSNN-QC maintains competitive accuracy while substantially reducing both communication and computational energy costs compared to conventional methods.

Federated Learning (FL) offers an efficient framework for local devices to train a global model collaboratively, while authorizing users to control their data locally. In this research work, an efficient Federated Deep Learning (FDL) framework is proposed for estimating users’ sentiments toward a specific topic, product, or service. More precisely, we employed Deep Learning (DL) models to train decentralized devices using FL that maintains data locally without transferring the raw data during training. Nevertheless, the non-IID genre of the data when distributed among multiple clients renders FL a challenging task. To address this, we employed an efficient Quantity-level Dirichlet Allocation strategy for data distribution among multiple clients. Two DL models have been trained, namely, LSTM and BERT, using the proposed FDL framework on local datasets. For model aggregation, a robust aggregation strategy called FedMedian is used, which efficiently handles non-IID data. The efficiency of the proposed framework is estimated using two distinct sentiment datasets: (i) Sentiment140 dataset and (ii) Yelp reviews dataset. The experimental outcomes illustrate that the proposed model surpass centralized learning on both coarse (two-class) and fine-grained (five-class) sentiment classification tasks. Further, the models are evaluated on a secure aggregation environment integrated with the proposed FL framework. The study reports that the model’s performance is consistent with secured aggregation, ensuring data privacy.

In recent years, to surmount the challenges of data sensitivity and data silos, federated learning (FL), a collaborative decentralized privacy preservation machine learning framework was proposed and rapidly became a current research hotspot. In this paper, we present an overview of FL workflow, as well as the communication overhead and privacy leakage problems associated with FL. In order to alleviate these two pain points, several corresponding solutions are presented in this paper along with their advantages and disadvantages. To reduce the communication overhead, this paper suggests three reliable approaches by reviewing related research including increasing the amount of computing on the edge side, model compression, and improving parallelism. To address the privacy concerns, this paper introduces three prevailing privacy preservation techniques: differential privacy, homomorphic encryption, and secure multiparty computation (SMC), and shows how to efficiently incorporate these privacy-preserving techniques into a federated learning framework by summarizing relevant literature. On the whole, this paper contributes to explaining the overall workflow of a FL framework and helps advance the understanding of FL in improving communication efficiency and enhancing privacy security.

Review on application progress of federated learning model and security hazard protection

With the development of the Internet of things (IoT), federated learning (FL) has received increasing attention as a distributed machine learning (ML) framework that does not require data exchange. However, current FL frameworks follow an idealized setup in which the task size is fixed and the storage space is unlimited, which is impossible in the real world. In fact, new classes of these participating clients always emerge over time, and some samples are overwritten or discarded due to storage limitations. We urgently need a new framework to adapt to the dynamic task sequences and strict storage constraints in the real world. Continuous learning or incremental learning is the ultimate goal of deep learning, and we introduce incremental learning into FL to describe a new federated learning framework. New generation federated learning (NGFL) is probably the most desirable framework for FL, in which, in addition to the basic task of training the server, each client needs to learn its private tasks, which arrive continuously independent of communication with the server. We give a rigorous mathematical representation of this framework, detail several major challenges faced under this framework, and address the main challenges of combining incremental learning with federated learning (aggregation of heterogeneous output layers and the task transformation mutual knowledge problem), and show the lower and upper baselines of the framework.