Towards Fair and Efficient Congestion Control through Multi-Agent Deep Reinforcement Learning

With the development of the Internet of Things (IoT) and 5G, there are ubiquitous smart devices and network functions providing emerging network services efficiently and optimally through building many network connections based on WiFi, LTE/5G, Ethernet, and etc. The Multipath TCP (MPTCP) protocol that enables these devices to establish multiple paths for simultaneous data transmission, has been a widely used extension of standard TCP in smart devices and network functions. On the other hand, more heavy and time-varying traffic loads are generated in an MPTCP network, so that an efficient congestion control mechanism that schedules the traffic between multiple subflows and avoids congestion is highly required. In this paper, we propose a decentralized learning approach, DeepCC, to adapt to the volatile environments and realize the efficient congestion control. The Multi-Agent Deep Reinforcement Learning (MADRL) is used to learn a policy of congestion control for each subflow according to the real-time network states. To deal with the problem of the fixed state space and slow convergence, we adopt two self-attention mechanisms to receive the states and train the policy, respectively. Due to the asynchronous design of DeepCC, the learning process will not introduce extra delay and overhead on the decision-making process. Experiment results show that DeepCC consistently outperforms the well-known heuristic method and DRL-based MPTCP congestion control method in terms of goodput and jitter. Besides, DeepCC with the attention mechanism reduces convergence time by about 50% and increase goodput by about 80% compared with the commonly used structures of neural networks.

In recent years, hybrid congestion control (CC) algorithms that combine rule-based CC and learning-based CC have gained significant attention. They incorporate the fast adaption ability of learning-based CC and the stability of rule-based CC, tending to select the better-performing rate based on the network feedback. However, the practical implementation of such algorithms has revealed primary issues. Specifically, they require both CCs to run alternately, which results in a poorly performing CC continuing to run in the network. Moreover, hybrid CCs cannot converge to the optimal rate when both CCs perform poorly. This paper proposes Anole to address these issues. Anole has three main algorithmic contributions: 1) Anole always selects the better-performing CC, 2) Anole temporarily deprecates the consistently underperforming CC, 3) when both CCs pervform poorly, Anole infers the optimal sending rate based on the network feedback. We carry out comprehensive experiments in both emulated and real-world wired networks, as well as in real-world WiFi networks, to assess the performance of Anole. The experiment results demonstrate that Anole achieves approximately 6% higher throughput in real-world links and 34% lower delay in the 48Mbps link compared to the state-of-the-art CC. Anole also exhibits superior performance in adaptability and fair convergence.

Dynamic holding control to avoid bus bunching: A multi-agent deep reinforcement learning framework

This paper introduces FedMediX, an adaptive federated multi agent deep reinforcement learning (MARL) framework for radiological data analysis. FedMediX accurately predicts multiple diseases from different types of diagnostic scans by involving a collaborative deep learning model trained across hospitals without sharing patient data. It is also able to automatically identify the type of medical image and explain the prediction using heatmaps. To address privacy constraints among multiple hospitals and non-stationary growth of medical data, federated learning is incorporated with a multi-agent reinforcement learning controller. To handle the non-IID data and enhance robustness, the federated learning involves a hybrid aggregation strategy incorporating attention mechanisms, trust scoring, and robust filtering. With a shared backbone, it automatically classifies the imaging modalities before disease-specific inference. This framework enables dynamic client selection and includes a Grad-CAM-based explainable AI providing visual justification for model predictions. Evaluation results on datasets confirm improved accuracy, faster convergence, and enhanced robustness of the proposed system.

Dear Editor, This letter presents a new transfer learning framework for the deep multi-agent reinforcement learning (DMARL) to reduce the convergence difficulty and training time when applying DMARL to a new scenario [1], [2]. The proposed transfer learning framework includes the design of neural network architecture, curriculum transfer learning (CTL) and strategy distillation. Experimental results demonstrate that our framework enables DMARL models to converge faster while improving the final performance.

Congestion control in Cluster Wireless Sensor Networks (CWSNs) has drawn widespread attention and research interests. The increasing number of nodes and scale of networks cause more complex congestion control and management. Active Queue Management (AQM) is one of the major congestion control approaches in CWSNs, and Random Early Detection (RED) algorithm is commonly used to achieve high utilization in AQM. However, traditional RED algorithm depends exclusively on source-side control, which is insufficient to maintain efficiency and state stability. Specifically, when congestion occurs, deficiency of feedback will hinder the instability of the system. In this paper, we adopt the Additive-Increase Multiplicative-Decrease (AIMD) adjustment scheme and propose an improved RED algorithm by using neighbor feedback and scheduling scheme. The congestion control model is presented, which is a linear system with a non-linear feedback, and modeled by Lur’e type system. In the context of delayed Lur’e dynamical network, we adopt the concept of cluster synchronization and show that the congestion controlled system is able to achieve cluster synchronization. Sufficient conditions are derived by applying Lyapunov-Krasovskii functionals. Numerical examples are investigated to validate the effectiveness of the congestion control algorithm and the stability of the network.

A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning

In recent years, reinforcement learning (RL) methods have shown powerful learning capabilities in single-vehicle autonomous driving. However, few studies have focused on multi-vehicle cooperative driving based on RL, particularly in the dynamically changing traffic environments of highway ramp merge zones. In this paper, a multi-agent deep reinforcement learning (MARL) framework for multi-vehicle cooperative decision-making is proposed based on actor–critic, which categorizes vehicles into two groups according to their origins in the merging area. At the same time, the complexity of the network is reduced and the training process of the model is accelerated by utilizing mechanisms such as partial parameter sharing and experience playback. Additionally, a combination of global and individual rewards is adopted to promote cooperation in connected autonomous vehicles (CAVs) and balance individual and group interests. The training performance of the model is compared under three traffic densities, and our method is also compared with state-of-the-art benchmark methods. The simulation results show that the proposed MARL framework can have stronger policy learning capability and stability under various traffic flow conditions. Moreover, it can also effectively improve the speed of vehicles in the merging zone and reduce traffic conflicts.

Efficient congestion control in communications using novel weighted ensemble deep reinforcement learning

Transmission Control Protocol (TCP) used multipath for achievement of data instantaneously to improve its performance.Anyhow, the preceding TCP protocol in the Internet of Things (IoT) network experienced a struggle to transmit a superior number of subflows.The Bottleneck Bandwidth and Round-trip propagation time (BBR) generate a network path model by assessing the available bottleneck bandwidth and the minimal Round-Trip Time (RTT) to achieve the desired delivery rate, consequently reducing the latency.However, some research studies have indicated the presence of significant fairness issues related to RTT in the BBR algorithm.To overcome this issue, we proposed a New Enhanced Congestion Control (CC) mechanism referred to as Bottleneck Bandwidth and Round-trip propagation time (BBR-NEnh) which is cross-layer-based efficient congestion control in a 5G IoT heterogeneous network.The proposed work encloses three sequential processes including grid-based network construction, improved two-factor clustering, and Hybrid TCP congestion control.Initially, we perform network construction by grid-based to improve the information transfer rate and increase connectivity among devices.After that, IoT devices are clustered based on two factors congestion and buffer space using the Improved K-Medoids algorithm (IK-Med) by the edge server.Further, to enhance the efficiency of TCP, the proposed research adopts a BBR-NEnh by optimizing the congestion control parameters using the Adaptive Tunicate Swarm Optimization (ATSO) algorithm.Finally, the dynamic handover and adjustment of windows are performed based on the Fast Adaptation Technique (FAdT) which can send the data quickly without congestion.The implementation of the proposed research is carried out using NS-3.26 and the performance of the proposed BBR-NEnh model is evaluated using a variety of performance metrics, including goodput, delay, packet loss, queue length, transmission rate, and throughput.The performance of the proposed work is compared with IMPRTT (Improved RTT) and CW-IoT (Congestion Window Algorithm for the Internet of Things).The average throughput, goodput, and packet loss of the proposed work for 1500 packets size are 4, 0.9, and 10.3, and the average delay, transmission rate, and queue length over 150 seconds are 0.004, 2500, and 36.1, demonstrating that the proposed work outperforms earlier work like IMPRTT and CW-IoT.

Traditional beam scheduling schemes typically divide the satellite coverage area into fixed cells and perform beam hopping among these predefined regions. However, such static partitioning lacks responsiveness to the dynamic distribution of users, thereby limiting the optimality of scheduling performance. To overcome this limitation, this paper proposes User-Centric Deep Beam Coverage Optimization (UC-DBCO), a demanddriven beam coverage optimization strategy. First, UC-DBCO abandons static area partitioning and dynamically adjusts the coverage region of each beam-including its center position and beamwidth-based on real-time user demands. This enables more flexible spatial resource allocation and finer-grained service matching. Next, to enhance collaborative decision-making among multiple beams, UC-DBCO incorporates an attention mechanism into the mixing network to model inter-beam interference and cooperation, allowing the system to focus on critical interaction information and improve policy learning efficiency. In addition, the system integrates Orthogonal Frequency Division Multiple Access (OFDMA) for spectrum resource allocation. Specifically, while the deep reinforcement learning framework optimizes beam coverage, a simulated annealing (SA) optimization method is employed to determine the subcarrier assignments, jointly responding to real-time user distribution and enhancing overall spectral efficiency.

The rapid growth of connected medical devices generates massive volumes of heterogeneous health data that must be processed and transmitted in real time. In such environments, minimizing latency and energy consumption remains a critical challenge for next-generation health monitoring systems. Existing reinforcement learning and optimization methods for intelligent communication networks face several challenges, including slow convergence, high computational overhead, and inefficiency in handling task prioritization. To resolve these issues, this work develops a chaotic dung beetle optimization-boosted multi-agent deep reinforcement learning that jointly optimizes communication reliability, computational efficiency, and task prioritization. A reward function is designed to jointly minimize delay, energy usage, and system cost while preserving information freshness. Specifically, the dung beetle optimization process is combined with a piecewise linear chaotic map to enhance population diversity, which significantly improves search space exploration and leads to faster convergence and higher solution quality. The proposed algorithm enhances the exploration capability of multi-agent deep reinforcement learning through the integration of chaotic dung beetle optimization, enabling more accurate and reliable decision-making in real-world applications. Extensive experiments demonstrate that the proposed chaotic dung beetle optimization-boosted multi-agent deep reinforcement learning model achieves superior performance compared to baseline algorithms. Specifically, it reaches an accuracy of over 97.00% with rapid convergence, reduces system cost under varying health data sizes and Medical Internet of Things devices, and maintains robust scalability across diverse workloads. Moreover, the model achieves significant reductions in communication latency and energy consumption as central processing unit cycles and bandwidth increase, while effectively prioritizing high-criticality tasks.

Multicast is attracting a resurgence of interest because it has a potential to address the explosively growing need for efficient streaming of large-volume Internet content. However, to realize the potential, large-scale multicast distribution requires effective congestion control. In this paper, we design SIM, a protocol that combines three distinct mechanisms (Selective participation, Intra-group transmission adjustment, and Menu adaptation) to provide a general solution for efficient fair scalable privacy-preserving multicast congestion control with heterogeneous receivers. Whereas the individual mechanisms have appeared in earlier multicast protocols, our main contribution lies in the cohesive integration of the techniques. SIM achieves such integration by operating the three mechanisms at different timescales and distributing the responsibility for control to different entities in the network. Besides, to instantiate and integrate the three control mechanisms, SIM employs several novel techniques of independent interest. Our extensive evaluation confirms the salient properties of SIM in diverse settings where receiving capabilities are highly heterogeneous, bottleneck capacities fluctuate, bottlenecks migrate, and session membership is dynamic.

Highly dynamic Internet of Vehicles spectrum sharing can share spectrum owned by vehicle-to-infrastructure links through multiple workshop links to achieve efficient resource allocation. Aiming at the problem that the rapid variations in channel states in highly dynamic vehicular environments can make it challenging for base stations to gather and manage information about instantaneous channel states, we present a multi-agent deep reinforcement learning-based V2X spectrum access algorithm. The algorithm is designed to optimize the throughput of V2I user under V2V user delay and reliability constraints, and uses the experience gained from interacting with the communication environment to update the Q network to improve spectrum and power allocation strategies. Implicit collaborative agents are trained through an improved DQN model combined with dueling network architecture and long short-term memory network layers and public rewards. With lagged Q-learning and concurrent experience replay trajectories, the training process was stabilized and the non-stationarity problem caused by concurrent learning of multiple agents was resolved. Simulation results demonstrate that our presented algorithm achieves a mean successful payload delivery rate of 95.89%, which is 16.48% greater than that of the randomized baseline algorithm. Our algorithm obtains approximately the optimal value and shows performance close to the centralized brute force algorithm, which provides a better strategy for further minimizing the signaling overhead of the Internet of Vehicles communication system.

The power control is a significant technique for suppressing co-channel interference that severely limits the capacity and connectivity of multi-cell communication systems. In this paper, we propose a novel and efficient multi-agent deep reinforcement learning (MADRL)-based uplink power control method for multi-cell multi-user communication systems. We first formulate the multi-user uplink transmission power optimization problem to maximize the sum throughput of multiple cells. Then, the optimization problem is transformed into a Markov decision process. Since the multi-user power control needs to consider the cooperation of strategies between users, the MADRL technique can be adopted. In our MADRL model, each agent outputs the uplink transmission power of the corresponding user by leveraging the value decomposition network. We also design a pruning algorithm to accelerate the training process of the MADRL model. The experimental results indicate that the proposed MADRL-based uplink power control method is superior to the baseline methods in terms of system throughput and quality of service. The designed pruning algorithm can effectively accelerate model training and also further improve the throughput performance of the proposed method.