M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortion

With the growth of participating clients, the centralized parameter server (PS) will seriously limit the scale and efficiency of Federated Learning (FL). A straightforward approach to scale up the FL system is to construct a Parallel FL (PFL) system with multiple parallel PSes. However, it is unclear whether PFL can really accelerate FL or reduce the training time of FL. Even if the answer is yes, it is non-trivial to design a highly efficient parameter average algorithm for a PFL system. In this paper, we propose a completely parallelizable FL algorithm called P-FedAvg under the PFL architecture. P-FedAvg extends the well-known FedAvg algorithm by allowing multiple PSes to cooperate and train a learning model together. In P-FedAvg, each PS is only responsible for a fraction of total clients, but PSes can mix model parameters in a dedicatedly designed way so that the FL model can well converge. Different from heuristic-based algorithms, P-FedAvg is with theoretical guarantees. To be rigorous, we theoretically analyze the convergence rate of P-FedAvg in terms of the number of conducted iterations, the communication cost of each global iteration and the optimal weights for each PS to mix parameters with its neighbors. Based on theoretical analysis, we conduct a case study on five typical overlay topolgoies formed by PSes to further examine the communication efficiency under different topologies, and investigate how the overlay topology affects the convergence rate, communication cost and robustness of a PFL system. Lastly, we perform extensive experiments with real datasets to verify our analysis and demonstrate that P-FedAvg can significantly speed up FL than traditional FedAvg and other competitive baselines. We believe that our work can help to lay a theoretical foundation for building more efficient PFL systems.

Federated Learning (FL) is a recently proposed machine learning scheme for decentralized training across distributed devices with enhanced data privacy. FL is known to provide a solution in cloud environments to overcome the privacy concerns arising out of centralized data collection. In this study, we investigate Federated Learning in cloud-based system, in particular, cognitive regarding its capability to secure data privacy, scalability and impact on model training by AI. Then, results were obtained in experiments to evaluate how FL performs, in terms of model accuracy and communication overhead, and in how scalable it is using publicly available datasets. Results of time to completion and accuracy across Federated Learning and centralized learning systems indicate that while there is a loss in accuracy from non IID data distribution, Federated Learning also exhibits advantages regarding scale (scaling order) and privacy. Communication costs increased due to need for frequent updates across distributed devices, but gradient compression was found to mitigate this challenge. Focusing on the trade-offs between Federated and centralized learning systems, this research provides important hints for future studies on privacy preserving AI in cloud environments.

In this paper, we present a communication-efficient federated learning framework inspired by quantized compressed sensing. The presented framework consists of gradient compression for wireless devices and gradient reconstruction for a parameter server (PS). Our strategy for gradient compression is to sequentially perform block sparsification, dimensional reduction, and quantization. By leveraging both dimension reduction and quantization, our strategy can achieve a higher compression ratio than one-bit gradient compression. For accurate aggregation of local gradients from the compressed signals, we put forth an approximate minimum mean square error (MMSE) approach for gradient reconstruction using the expectation-maximization generalized-approximate-message-passing (EM-GAMP) algorithm. Assuming Bernoulli Gaussian-mixture prior, this algorithm iteratively updates the posterior mean and variance of local gradients from the compressed signals. We also present a low-complexity approach for the gradient reconstruction. In this approach, we use the Bussgang theorem to aggregate local gradients from the compressed signals, then compute an approximate MMSE estimate of the aggregated gradient using the EM-GAMP algorithm. We also provide a convergence rate analysis of the presented framework. Using the MNIST dataset, we demonstrate that the presented framework achieves almost identical performance with the case that performs no compression, while significantly reducing communication overhead for federated learning.

Federated learning (FL) aims to learn a model with privacy protection through a distributed scheme over many clients. In FL, an important problem is to reduce the transmission quantity between clients and parameter server during gradient uploading. Because FL environment is not stable and requires enough client responses to be collected within a certain period of time, traditional model compression practices are not entirely suitable for FL setting. For instance, both design of the low-rank filter and the algorithm used to pursue sparse neural network generally need to perform more training rounds locally to ensure that the accuracy of model is not excessively lost. To breakthrough transmission bottleneck, we propose low rank communication Fedlr to compress whole neural network in clients reporting phase. Our innovation is to propose the concept of optimal compression rate. In addition, two measures are introduced to make up accuracy loss caused by truncation: training low rank parameter matrix and using iterative averaging. The algorithm is verified by experimental evaluation on public datasets. In particular, CNN model parameters training on the MNIST dataset can be compressed 32 times and lose only 2% of accuracy.

We study federated edge learning, where a global model is trained collaboratively using privacy-sensitive data at the edge of a wireless network. A parameter server (PS) keeps track of the global model and shares it with the wireless edge devices for training using their private local data. The devices then transmit their local model updates, which are used to update the global model, to the PS. The algorithm, which involves transmission over PS-to-device and device-to-PS links, continues until the convergence of the global model or lack of any participating devices. In this study, we consider device selection based on downlink channels over which the PS shares the global model with the devices. Performing digital downlink transmission, we design a partial device participation framework where a subset of the devices is selected for training at each iteration. Therefore, the participating devices can have a better estimate of the global model compared to the full device participation case which is due to the shared nature of the broadcast channel with the price of updating the global model with respect to a smaller set of data. At each iteration, the PS broadcasts different quantized global model updates to different participating devices based on the last global model estimates available at the devices. We investigate the best number of participating devices through experimental results for image classification using the MNIST dataset with biased distribution.

EC-DNN: A new method for parallel training of deep neural networks

In order to meet the extremely heterogeneous requirements of the next generation wireless communication networks, research community is increasingly dependent on using machine‐learning solutions for real‐time decision‐making and radio resource management. Traditional machine learning employs fully centralized architecture in which the entire training data is collected at one node for example, cloud server, that significantly increases the communication overheads and also raises severe privacy concerns. Toward this end, a distributed machine‐learning paradigm termed as federated learning (FL) has been proposed recently. In FL, each participating edge device trains its local model by using its own training data. Then, via the wireless channels the weights or parameters of the locally trained models are sent to the central parameter server (PS), that aggregates them and updates the global model. On one hand, FL plays an important role for optimizing the resources of wireless communication networks, on the other hand, wireless communications is crucial for FL. Thus, a “bidirectional” relationship exists between FL and wireless communications. Although FL is an emerging concept, many publications have already been published in the domain of FL and its applications for next generation wireless networks. Nevertheless, we noticed that none of the works have highlighted the bidirectional relationship between FL and wireless communications. Therefore, the purpose of this survey article is to bridge this gap in literature by providing a timely and comprehensive discussion on the interdependency between FL and wireless communications.

Federated Learning (FL) systems orchestrate the cooperative training of a shared machine learning (ML) model across connected devices. Recently, decentralized FL architectures driven by consensus have been proposed to enable the devices to share and aggregate the ML parameters via direct sidelink communications. The approach has the advantage of promoting the federation among the agents even in the absence of a server, but may require an intensive use of communication resources compared to vanilla FL methods. This paper proposes a communication-efficient design of consensus-driven FL optimized for training of deep neural networks (DNNs). Devices independently select fragments of the DNN to be shared with neighbors on each training round. Selection is based on a local optimizer that trades model quality improvement with sidelink communication resource savings. The proposed technique is validated on a vehicular cooperative sensing use case characterized by challenging real-world datasets and complex DNNs typically employed in autonomous driving with up to 40 trainable layers. The impact of layer selection is analyzed under different distributed coordination configurations. The results show that it is better to prioritize the DNN layers possessing few parameters, while the selection policy should optimally balance gradient sorting and randomization. Latency, accuracy and communication tradeoffs are analyzed in detail targeting sustainable federation policies.

Federated learning (FL) is a machine learning framework where multiple distributed edge internet-of-things (IoT) devices collaboratively train a model under the orchestration of a central server while keeping the training data distributed on the IoT devices. FL can mitigate the privacy risks and costs from data collection in traditional centralized machine learning. However, the deployment of standard FL is hindered by the expense of the communication of the gradients from the devices to the server. Hence, many gradient compression methods have been proposed to reduce the communication cost. However, the existing methods ignore the structural correlations of the gradients, and therefore lead to a large compression loss which will decelerate the training convergence. Moreover, many of the existing compression schemes do not enable over-the-air aggregation, and hence require huge communication resources. In this work, we propose a gradient compression scheme, named FedOComp, which leverages the correlations of the stochastic gradients in FL systems for efficient compression of the high-dimension gradients with over-the-air aggregation. The proposed design can achieve a smaller deceleration of the training convergence compared to other gradient compression methods since the compression kernel exploits the structural correlations of the gradients. It also directly enables over-the-air aggregation to save communication resources. The derived convergence analysis and simulation results further illustrate that under the same power cost, the proposed scheme has a much faster convergence rate and higher test accuracy compared to existing baselines.

This artilce aims to develop a differential private federated learning (FL) scheme with the least artificial noises added while minimizing the energy consumption of participating mobile devices. By observing that some communication efficient FL approaches and even the nature of wireless communications contribute to the differential privacy (DP) preservation of training data on mobile devices, in this paper, we propose to jointly leverage gradient compression techniques (i.e., gradient quantization and sparsification) and additive white Gaussian noises (AWGN) in wireless channels to develop a piggyback DP approach for FL over mobile devices. Even with the piggyback DP approach, information distortion caused by gradient compression and noise perturbation may slow down FL convergence, which in turn consumes more energy of mobile devices for local computing and model update communications. Thus, we theoretically analyze FL convergence and formulate an energy efficient FL optimization under piggyback DP, transmission power, and FL convergence constraints. Furthermore, we propose an efficient iterative algorithm where closed-form solutions for artificial DP noise and power control are derived. Extensive simulation and experimental results demonstrate the effectiveness of the proposed scheme in terms of energy efficiency and privacy preservation.

Federated learning (FL) enables decentralized training of deep neural networks (DNNs) for object detection over a distributed population of clients. It allows edge clients to keep their data local and only share parameter updates with a federated server. However, the distributed nature of FL also opens doors to new threats. In this paper, we present targeted perception poisoning attacks against federated object detection learning in which a subset of malicious clients seeks to poison the federated training of a global object detection model by sharing perception-poisoned local model parameters. We first introduce three targeted perception poisoning attacks, which have severe adverse effects only on the objects under attack. We then analyze the attack feasibility, the impact of malicious client availability, and attack timing. To safeguard FL systems against such contagious threats, we introduce spatial signature analysis as a defense to separate benign local model parameters from poisoned local model contributions, identify malicious clients, and eliminate their impact on the federated training. Extensive experiments on object detection benchmark datasets validate that the defense-empowered federated object detection learning can improve the robustness against all three types of perception poisoning attacks. The source code is available at https://github.com/git-disl/Perception-Poisoning.

Federated Learning (FL) is considered the key, enabling approach for privacy-preserving, distributed machine learning (ML) systems. FL requires the periodic transmission of ML models from users to the server. Therefore, communication via resource-constrained networks is currently a fundamental bottleneck in FL, which is restricting the ML model complexity and user participation. One of the notable trends to reduce the communication cost of FL systems is gradient compression, in which techniques in the form of sparsification are utilized. However, these methods utilize a single compression rate for all users and do not consider communication heterogeneity in a real-world FL system. Therefore, these methods are bottlenecked by the worst communication capacity across users. Further, sparsification methods are non-adaptive and do not utilize the redundant, similar information across users' ML models for compression. In this paper, we introduce a novel Dynamic Sparsification for Federated Learning (DSFL) approach that enables users to compress their local models based on their communication capacity at each iteration by using two novel sparsification methods: layer-wise similarity sparsification (LSS) and extended top- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> sparsification. LSS enables DSFL to utilize the global redundant information in users' models by using the Centralized Kernel Alignment (CKA) similarity for sparsification. The extended top- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> model sparsification method empowers DSFL to accommodate the heterogeneous communication capacity of user devices by allowing different values of sparsification rate <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$K$</tex> for each user at each iteration. Our extensive experimental results <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> All code and experiments are publicly available at: https://github.com/mahdibeit/DSFL. on three datasets show that DSFL has a faster convergence rate than fixed sparsification, and as the communication heterogeneity increases, this gap increases. Further, our thorough experimental investigations uncover the similarities of user models across the FL system.

To build intelligent model learning in conventional architecture, the local data are required to be transmitted toward the cloud server, which causes heavy backhaul congestion, leakage of personalization, and insufficient use of network resources. To address these issues, federated learning (FL) is introduced by offering a systematical framework that converges the distributed modeling process between local participants and the parameter server. However, the challenging issues of insufficient participant scheduling, aggregation policies, model offloading, and resource management still remain within conventional FL architecture. In this survey article, the state-of-the-art solutions for optimizing the orchestration in FL communications are presented, primarily querying the deep reinforcement learning (DRL)-based autonomy approaches. The correlations between the DRL and FL mechanisms are described within the optimized system architectures of selected literature approaches. The observable states, configurable actions, and target rewards are inquired into to illustrate the applicability of DRL-assisted control toward self-organizing FL systems. Various deployment strategies for Internet of Things applications are discussed. Furthermore, this article offers a review of the challenges and future research perspectives for advancing practical performances. Advanced solutions in these aspects will drive the applicability of converged DRL and FL for future autonomous communication-efficient and privacy-aware learning.

Federated learning (FL) is an attractive scheme for training machine learning models across distributed edge devices without requiring raw data to be shared. However, deploying FL at the edge of the net- work raises two interrelated issues: (1) the significant communication overhead due to transmitting massive gradient vectors from resource- constrained devices and (2) the risk of privacy leakage via attacks that can invert gradients or infer training samples. Current methods treat gradient compression and differential privacy (DP) as separate concerns, leading to compounded accuracy loss. We propose AGC-DP—Adaptive Gradient Compression (AGC) with Differential Privacy (DP)—a uni- fied framework that achieves efficient communication and strong pri- vacy guarantees across diverse edge FL environments. AGC-DP also introduces a resource-aware client selection algorithm to balance con- tributions from battery level, available memory, and uplink bandwidth; an adaptive compression ratio for each client based on the optimization problem; and a privacy amplification construction that exploits sub- sampling to improve DP accounting. We provide rigorous theoretical analysis, including convergence guarantees under non-i.i.d. data dis- tributions and formal (ϵ, δ)- DP proofs. Comprehensive experiments across the CIFAR-10, HAR, and FEMNIST datasets with up to 100 edge clients show that AGC-DP reduces communication overhead by up to 91.1% compared to vanilla FedAvg, while maintaining accuracy within 1.5% of the vanilla baseline. Our results show that privacy, com- munication efficiency, and model utility can be achieved simultaneously in resource-constrained edge FL.

Federated learning is a framework for machine learning that is dedicated to data privacy protection. In federated learning, system cannot fully control the behavior of clients which can be faulty. These behaviors include sharing arbitrary faulty gradients and delaying the process of sharing due to Byzantine attacks or clients’ own software and hardware failures. In federated learning, the parameter server may also be faulty during gradient collection and aggregation, mainly including gradient-based training data inference and model parameter faulty update. The above problems may lead to reduced accuracy of federated learning model training, leakage of client privacy, etc. Existing research enhances the robustness of federated learning by exploiting the decentralization and immutability of Blockchain. For untrusted clients, most research is based on Byzantine fault tolerance to defend against clients indiscriminately, and may cause model accuracy reduction. In addition, most of the research focus on unencrypted gradients, and there is insufficient research on dealing with client anomalies in the case of gradient encryption. For untrusted parameter servers, existing research has problems in energy overhead and scalability. Aiming at the problems above, this paper studies the robustness of federated learning, and proposes a blockchain-based federated learning parameter update architecture PUS-FL. Through experiments simulating distributed machine learning on neural networks, we demonstrate that the anomaly detection algorithm of PUS-FL outperforms conventional gradient filters including geometric median, Multi-Krum and trimmed mean. In addition, our experiments also verify that the scalability-enhanced parameter aggregation consensus algorithm proposed in this paper(SE-PBFT) improves consensus scalability by reducing communication complexity.