Just-in-Time Aggregation for Federated Learning

The federated learning (FL) paradigm is well-suited for the field of medical image analysis, as it can effectively cope with machine learning on isolated multi-center data while protecting the privacy of participating parties. However, current research on optimization algorithms in FL often focuses on limited datasets and scenarios, primarily centered around natural images, with insufficient comparative experiments in medical contexts. In this work, we conduct a comprehensive evaluation of several state-of-the-art FL algorithms in the context of medical imaging. We conduct a fair comparison of classification models trained using various FL algorithms across multiple medical imaging datasets. Additionally, we evaluate system performance metrics, such as communication cost and computational efficiency, while considering different FL architectures. Our findings show that medical imaging datasets pose substantial challenges for current FL optimization algorithms. No single algorithm consistently delivers optimal performance across all medical FL scenarios, and many optimization algorithms may under-perform when applied to these datasets. Our experiments provide a benchmark and guidance for future research and application of FL in medical imaging contexts. Furthermore, we propose an efficient and robust method that combines generative techniques using denoising diffusion probabilistic models with label smoothing to augment datasets, widely enhancing the performance of FL on classification tasks across various medical imaging datasets.

In this paper, the convergence time of federated learning (FL), when deployed over a realistic wireless network, is studied. In particular, a wireless network is considered in which wireless users transmit their local FL models (trained using their locally collected data) to a base station (BS). The BS, acting as a central controller, generates a global FL model using the received local FL models and broadcasts it back to all users. Due to the limited number of resource blocks (RBs) in a wireless network, only a subset of users can be selected to transmit their local FL model parameters to the BS at each learning step. Moreover, since each user has unique training data samples, the BS prefers to include all local user FL models to generate a converged global FL model. Hence, the FL training loss and convergence time will be significantly affected by the user selection scheme. Therefore, it is necessary to design an appropriate user selection scheme that can select the users who can contribute toward improving the FL convergence speed more frequently. This joint learning, wireless resource allocation, and user selection problem is formulated as an optimization problem whose goal is to minimize the FL convergence time and the FL training loss. To solve this problem, a probabilistic user selection scheme is proposed such that the BS is connected to the users whose local FL models have significant effects on the global FL model with high probabilities. Given the user selection policy, the uplink RB allocation can be determined. To further reduce the FL convergence time, artificial neural networks (ANNs) are used to estimate the local FL models of the users that are not allocated any RBs for local FL model transmission at each given learning step, which enables the BS to improve the global model, the FL convergence speed, and the training loss. Simulation results show that the proposed approach can reduce the FL convergence time by up to 56% and improve the accuracy of identifying handwritten digits by up to 3%, compared to a standard FL algorithm.

Federated learning (FL), which enables multiple distributed devices (clients) to collaboratively train a global model without transmitting their private data, has attracted much attention in the Internet of Things (IoT) domain. Compared with centralized learning, FL has obvious privacy advantages because it can protect the clients’ raw data from direct access by adversaries. Furthermore, to prevent the adversaries from inferring private information from the transmitted parameters, several FL algorithms based on differential privacy (DP) have been proposed, where the clients add artificial noise to their local parameters for privacy protection. Nevertheless, the added noise would disrupt the learning process and degrade the performance of the trained model. Considering this, in this article, we develop a performance-enhanced DP-based FL (PEDPFL) algorithm, where a classifier-perturbation regularization method is proposed to improve the robustness of the trained model against DP-injected noise. We derive the theoretical privacy and convergence analysis of the proposed algorithm, and also demonstrate the influence of some hyperparameters on the convergence performance. Simulation results on real-world data sets show that the proposed algorithm has better classification performance than the existing DP-based FL algorithms at the same level of privacy protection, and thus, it is more applicable to IoT applications.

Edge servers and mobile devices are often assigned a large number of computing tasks. However, the data involved in computing tasks is often sensitive in terms of privacy. Our initial proposal is a federated edge learning strategy based on real-world scenarios, which combines blurred data or pseudo shared data. Federated learning is used to train device models with the aim of protecting privacy while enabling mobile devices to more effectively utilize data for decision-making. In the case of limited energy on mobile devices, we propose a federated edge learning algorithm with blurred data sharing. This algorithm can generate more accurate models by uploading partially blurred data. In order to further improve model accuracy and protect privacy of mobile devices, we propose a federated edge learning algorithm with pseudo data sharing based on dataset distillation and generative adversarial networks (GANs) in scenarios with relatively sufficient energy. The experimental results on several traditional datasets show that our proposed algorithms outperform traditional algorithms in terms of accuracy and energy consumption.

In this paper, the optimization of network performance to support the deployment of federated learning (FL) is investigated. In particular, in the considered model, each user owns a machine learning (ML) model by training through its own dataset, and then transmits its ML parameters to a base station (BS) which aggregates the ML parameters to obtain a global ML model and transmits it to each user. Due to limited radio frequency (RF) resources, the number of users that participate in FL is restricted. Meanwhile, each user uploading and downloading the FL parameters may increase communication costs thus reducing the number of participating users. To this end, we propose to introduce visible light communication (VLC) as a supplement to RF and use compression methods to reduce the resources needed to transmit FL parameters over wireless links so as to further improve the communication efficiency and simultaneously optimize wireless network through user selection and resource allocation. This user selection and bandwidth allocation problem is formulated as an optimization problem whose goal is to minimize the training loss of FL. We first use a model compression method to reduce the size of FL model parameters that are transmitted over wireless links. Then, the optimization problem is separated into two subproblems. The first subproblem is a user selection problem with a given bandwidth allocation, which is solved by a traversal algorithm. The second subproblem is a bandwidth allocation problem with a given user selection, which is solved by a numerical method. The ultimate user selection and bandwidth allocation are obtained by iteratively compressing the model and solving these two subproblems. Simulation results show that the proposed FL algorithm can improve the accuracy of object recognition by up to 16.7% and improve the number of selected users by up to 68.7%, compared to a conventional FL algorithm using only RF.

Traditional federated learning algorithms suffer from considerable performance reduction with non-identically and independently distributed datasets. This paper proposes a federated learning algorithm based on parallel-ensemble learning, which improves performance for image classification on these datasets. The training process of this algorithm includes basic federation learning and meta federation learning. First, several basic models are trained through cross-validation of federated learning, and then the meta-model is trained using the prediction results of the validation sets. In the training process, the training of different basic models is parallel. In prediction, meta-model is used to aggregate the output of the basic models to get the final prediction results. Our algorithm can achieve higher accuracy than traditional federated learning when using non-independent identically distributed datasets for image classification. Our algorithm aggregates different models through federated learning based on parallel-ensemble method, and improves the image classification performance of federated learning on non-independent identically distributed datasets.

Federated learning (FL) has gained attention as a distributed learning paradigm for its data privacy benefits and accelerated convergence through parallel computation. Traditional FL relies on a server-client (SC) architecture, where a central server coordinates multiple clients to train a global model, but this approach faces scalability challenges due to server communication bottlenecks. To overcome this, the ring-all-reduce (RAR) architecture has been introduced, eliminating the central server and achieving bandwidth optimality. However, the tightly coupled nature of RAR's ring topology exposes it to unique Byzantine attack risks not present in SC-based FL. Despite its potential, designing Byzantine-robust RAR-based FL algorithms remains an open problem. To address this gap, we propose BRACE (<u>B</u>yzantine-<u>r</u>obust ring-<u>a</u>ll-redu<u>ce</u>), the first RAR-based FL algorithm to achieve both Byzantine robustness and communication efficiency. We provide theoretical guarantees for the convergence of BRACE under Byzantine attacks, demonstrate its bandwidth efficiency, and validate its practical effectiveness through experiments. Our work offers a foundational understanding of Byzantine-robust RAR-based FL design.

With the growth of participating clients, the centralized parameter server (PS) will seriously limit the scale and efficiency of Federated Learning (FL). A straightforward approach to scale up the FL system is to construct a Parallel FL (PFL) system with multiple parallel PSes. However, it is unclear whether PFL can really accelerate FL or reduce the training time of FL. Even if the answer is yes, it is non-trivial to design a highly efficient parameter average algorithm for a PFL system. In this paper, we propose a completely parallelizable FL algorithm called P-FedAvg under the PFL architecture. P-FedAvg extends the well-known FedAvg algorithm by allowing multiple PSes to cooperate and train a learning model together. In P-FedAvg, each PS is only responsible for a fraction of total clients, but PSes can mix model parameters in a dedicatedly designed way so that the FL model can well converge. Different from heuristic-based algorithms, P-FedAvg is with theoretical guarantees. To be rigorous, we theoretically analyze the convergence rate of P-FedAvg in terms of the number of conducted iterations, the communication cost of each global iteration and the optimal weights for each PS to mix parameters with its neighbors. Based on theoretical analysis, we conduct a case study on five typical overlay topolgoies formed by PSes to further examine the communication efficiency under different topologies, and investigate how the overlay topology affects the convergence rate, communication cost and robustness of a PFL system. Lastly, we perform extensive experiments with real datasets to verify our analysis and demonstrate that P-FedAvg can significantly speed up FL than traditional FedAvg and other competitive baselines. We believe that our work can help to lay a theoretical foundation for building more efficient PFL systems.

Federated learning (FL) is a distributed model training paradigm that preserves clients’ data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose, an automatic FL hyper-parameter tuning algorithm tailored to applications’ diverse system requirements in FL training. iteratively adjusts FL hyper-parameters during FL training and can be easily integrated into existing FL systems. Through extensive evaluations of for diverse applications and FL aggregation algorithms, we show that is lightweight and effective, achieving 8.48%-26.75% system overhead reduction compared to using fixed FL hyper-parameters. This paper assists FL practitioners in designing high-performance FL training solutions. The source code of is available at.

Federated Learning (FL) has emerged as a promising approach to collaborative machine learning without the need to share raw data. It enables decentralized model updates while preserving the privacy of each device and reducing the communication overhead. This experiment evaluates the effectiveness of the personalized FL algorithms, namely FedAvg, APPLE, FedBABU and FedProto, in a decentralized setting, with a particular focus on the Fashion MNIST dataset, which is characterized by a non-ideal data distribution. The objective is to identify which algorithm performs optimally in image classification tasks. The experimental results show that both FedProto and APPLE have nearly equivalent and better performance compared to FedBABU and FedAvg. Interestingly, increasing the number of uploads in FedBABU leads to similar results to APPLE and FedProto. However, under limited upload conditions, FedBABU performs similarly to FedAvg. These results provide valuable insights into the differential performance of personalized FL algorithms in non-id data scenarios and provide guidance for their application in distributed environments, especially in sensitive domains such as medical, military and confidential image analysis tasks where privacy and communication efficiency are paramount concerns.

Federated Learning (FL) has recently attracted great interest in sensor-based human activity recognition (HAR) tasks. However, in real-world environment, sensor data on devices is non-independently and identically distributed (Non-IID), e.g., activity data recorded by most devices is sparse, and sensor data distribution for each client may be inconsistent. As a result, the traditional FL methods in the heterogeneous environment may incur a drifted global model that causes slow convergence and a heavy communication burden. Although some FL methods are gradually being applied to HAR, they are designed for overly ideal scenarios and do not address such Non-IID problem in the real-world setting. It is still a question whether they can be applied to cross-device FL. To tackle this challenge, we propose ProtoHAR, a prototype-guided FL framework for HAR, which aims to decouple the representation and classifier in the heterogeneous FL setting efficiently. It leverages the global prototype to correct the activity feature representation to make the prototype knowledge flow among clients without leaking privacy while solving a better classifier to avoid excessive drift of the local model in personalized training. Extensive experiments are conducted on four publicly available datasets: USC-HAD, UNIMIB-SHAR, PAMAP2, and HARBOX, which are collected in both controlled environments and real-world scenarios. The results show that compared with the state-of-the-art FL algorithms, ProtoHAR achieves the best performance and faster convergence speed in HAR datasets.

In modern devices, such as smartphones and IoT, AI hardware implements different ML models to train massive amounts of data for various applications. However, based on the sensitivity of this data, privacy and security concerns, or both, may restrict users from accessing the data storage to conduct the ML training using conventional methods. Federated learning (FL) is consequently emerged to maintain training data distribution among smart mobile devices while aggregating locally processed updates. In addition to improving the model training performance as the number of clients rises, FL also creates a privacy-preserved shared data model. FL execution, however, can be time-consuming. This study proposes a parallelized approach to enhance the performance and privacy of the FL algorithm (FedAvg). In this regard, the FedAvg algorithm is expanded to our proposed model, distributed FedAvg (D-FedAvg), which enables several clients to collaborate concurrently and train a single learning model. To evaluate the performance of the proposed model, we investigated the impact of various numbers of clients and training rounds on the result. To validate our findings, we conducted extensive experiments using MNIST datasets trained using two federated learning models, MLP and CNN. The results show our D-FedAvg can maintain data privacy and dramatically enhance the execution time of FL compared to traditional FedAvg. According to the study, the FL framework's time complexity increases as the number of clients and rounds increases, but parallelization allows it to operate 2–3 times faster than usual.

Secure and efficient federated learning via novel multi-party computation and compressed sensing

Federated learning (FL) has the potential to boost deep learning in neuroimaging but is rarely deployed in real-world scenarios, where its true potential lies. We propose FLightcase, a new FL toolbox tailored for brain research, and evaluate it on a real-world FL network to predict the cognitive status in patients with multiple sclerosis (MS) from brain magnetic resonance imaging (MRI). We first trained a DenseNet neural network to predict age from T1-weighted brain MRI on three open-source datasets: IXI (586 images), SALD (491 images), and CamCAN (653 images). These were distributed across the three centres in our FL network: Brussels (BE), Greifswald (DE), and Prague (CZ). We benchmarked this federated model with a centralised version. The best-performing brain age model was then fine-tuned to predict performance on the symbol digit modalities test (SDMT) of patients with MS (Brussels: 96 images, Greifswald: 756 images, Prague: 2,424 images). Shallow transfer learning (TL) was compared with deep transfer learning, in which weights were updated either in the last layer or across the entire network, respectively. Federated training outperformed centralised training, predicting age with a mean absolute error (MAE) of 6.08 versus 7.02. Federated training yielded Pearson correlations (all p < .001) between true and predicted age of0.88 (IXI, Brussels), 0.91 (SALD, Greifswald), and 0.93 (CamCAN, Prague). Fine-tuning of the centralised model to SDMT was most successful with a deep TL paradigm (MAE = 9.19) compared to shallow TL (MAE = 11.05). Across Brussels, Greifswald, and Prague, deep TL predicted SDMT with MAEs of 10.71, 9.67, and 8.98, respectively, and yielded Pearson correlations between true and predicted SDMT of.25 (p = 0.282), 0.40 (p < 0.001), and 0.50 (p < 0.001). Real-world federated learning using FLightcase is feasible for neuroimaging research in MS, enabling access to large MS imaging databases without sharing data. The federated SDMT-decoding model is promising and could be improved in the future by adopting FL algorithms that address the non-IID data issue and consider other imaging modalities. We hope our detailed real-world experiments and open-source distribution of FLightcase will prompt researchers to move beyond simulated FL environments.

In recent years, Federated Learning (FL) has gained traction as a privacy-centric approach in medical imaging. This study explores the challenges posed by data heterogeneity on FL algorithms, using the COVIDx CXR-3 dataset as a case study. We contrast the performance of the Federated Averaging (FedAvg) algorithm on non-identically and independently distributed (non-IID) data against identically and independently distributed (IID) data. Our findings reveal a notable performance decline with increased data heterogeneity, emphasizing the need for innovative strategies to enhance FL in diverse environments. This research contributes to the practical implementation of FL, extending beyond theoretical concepts and addressing the nuances in medical imaging applications. This research uncovers the inherent challenges in FL due to data diversity. It sets the stage for future advancements in FL strategies to effectively manage data heterogeneity, especially in sensitive fields like healthcare.