A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

Dynamic holding control to avoid bus bunching: A multi-agent deep reinforcement learning framework

Modern power systems are more vulnerable to cascading failure due to the increasing complexity and renewable integration in modern power systems. Cooperative, adaptive protection schemes are considered fundamental to improving grid resilience, but as a matter of fact, the implementation of such schemes is inherently limited by the data privacy and ownership rules on multi-entity grids. The current centralized and multi-agent deep reinforcement learning (DRL) solutions require consolidation of sensitive operational information, which forms critical single points of failures and privacy violations. To address this gap, this paper will present a new Federated Deep Reinforcement Learning (FDRL) framework. The methodology develops relay coordination as a Partially Observable Markov Decision Process (POMDP) and uses a Federated Deep Deterministic Policy Gradient (F-DDPG) algorithm such that distributed relay agents learn local models with privately available data and just provides encrypted model parameters to a central aggregator to securely train the federating process. Simulation outcomes on the IEEE 39-bus system show that the proposed scheme decreases the cascade size by 52.5% and load shedding by 54.7% relative to traditional protection, and has fault discrimination accuracy equal to 95.8, and can operate at 13% of the speed of a privacy-violating centralized DRL benchmark. The framework manages to accomplish intelligent and collaborative protection without data confidentiality.

In recent years, reinforcement learning (RL) methods have shown powerful learning capabilities in single-vehicle autonomous driving. However, few studies have focused on multi-vehicle cooperative driving based on RL, particularly in the dynamically changing traffic environments of highway ramp merge zones. In this paper, a multi-agent deep reinforcement learning (MARL) framework for multi-vehicle cooperative decision-making is proposed based on actor–critic, which categorizes vehicles into two groups according to their origins in the merging area. At the same time, the complexity of the network is reduced and the training process of the model is accelerated by utilizing mechanisms such as partial parameter sharing and experience playback. Additionally, a combination of global and individual rewards is adopted to promote cooperation in connected autonomous vehicles (CAVs) and balance individual and group interests. The training performance of the model is compared under three traffic densities, and our method is also compared with state-of-the-art benchmark methods. The simulation results show that the proposed MARL framework can have stronger policy learning capability and stability under various traffic flow conditions. Moreover, it can also effectively improve the speed of vehicles in the merging zone and reduce traffic conflicts.

Deploying low-altitude unmanned aerial vehicles (UAVs) as aerial base stations (ABSs) is a promising approach to enhance cellular network coverage in dynamic and infrastructure-constrained environments. However, real-world UAV deployment involves complex challenges, including managing partially observable and mobile users, maintaining multi-hop backhaul connectivity to a fixed ground base station, mitigating co-channel interference via minimum UAV separation, and optimizing coverage efficiency, fairness, and energy consumption. This paper proposes a multi-agent deep reinforcement learning (MARL) framework that formulates the dynamic UAV deployment task as a centralized multi-input multi-output policy. Our method employs sequential action masking to guarantee multi-hop connectivity and enforce minimum distance constraints during UAV placement. The policy network integrates graph neural networks, residual convolutional layers, and feature-wise linear modulation to effectively capture spatial relations, inter-UAV interactions, and global and local network states. We validate our approach through extensive simulations across various user density scenarios, demonstrating significant improvements in coverage efficiency, fairness of coverage distribution, and flight energy reduction compared to strong baseline algorithms. Our proposed framework provides a comprehensive solution that simultaneously addresses dynamic user mobility, multi-hop connectivity, interference mitigation, and energy efficiency in UAV-assisted cellular networks.

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

With the increasing real-time demand in the internet era, especially dynamic requests for last-mile delivery, route planning is becoming more computationally expensive than ever before. Many supply chains (SCs) choose the joint distribution of multiple depots to reduce transportation costs and delivery times. However, providing real-time and high-quality solutions for such complex routing problems remains challenging. Current solution methods like mathematical programming and heuristics suffer from scalability issues and long computation times. In contrast, artificial intelligence, especially Deep Reinforcement Learning (DRL), provides a general-purpose framework for sequential decision-making that has produced good results for many challenging real-life problems. However, applying DRL to route multiple vehicles is nontrivial, as the joint distribution requires an effective method that facilitates collaboration and communication among all agents while they carry out the delivery mission. In this research, a collaborative Multi-Agent Deep Reinforcement Learning (MADRL) approach is proposed for routing multiple vehicles in the SC. The proposed MADRL model leverages the power of two frameworks, deep learning and reinforcement learning, to generate routing policies for all agents in real time. Experimental results show the ability of the proposed learning model to obtain fast and quality solutions for complex delivery problems. Furthermore, the generalization ability of MADRL is also validated by testing the well-trained model on different scale problems.

Different types of construction equipment engage in complex cooperative interactions on construction sites, significantly influencing project progress. This study addresses the problem of cooperative excavation involving multiple heterogeneous machines—such as excavators and bulldozers—within a multi-agent Deep Reinforcement Learning (DRL) framework. A parameter-sharing Proximal Policy Optimization (PPO) architecture is introduced in a grid-world simulation environment, where all agents share a unified policy network and learn coordinated strategies through end-to-end training. To promote effective collaboration and address exploration challenges, a dense reward structure is designed, incorporating incentives for task completion, spoil delivery, excavation chain formation, and depth progression. Experimental results demonstrate that the proposed method achieves full cooperation within 2 minutes in small environments. As the scale increases, runtime grows approximately linearly and completion drops to 0.85.

This paper introduces FedMediX, an adaptive federated multi agent deep reinforcement learning (MARL) framework for radiological data analysis. FedMediX accurately predicts multiple diseases from different types of diagnostic scans by involving a collaborative deep learning model trained across hospitals without sharing patient data. It is also able to automatically identify the type of medical image and explain the prediction using heatmaps. To address privacy constraints among multiple hospitals and non-stationary growth of medical data, federated learning is incorporated with a multi-agent reinforcement learning controller. To handle the non-IID data and enhance robustness, the federated learning involves a hybrid aggregation strategy incorporating attention mechanisms, trust scoring, and robust filtering. With a shared backbone, it automatically classifies the imaging modalities before disease-specific inference. This framework enables dynamic client selection and includes a Grad-CAM-based explainable AI providing visual justification for model predictions. Evaluation results on datasets confirm improved accuracy, faster convergence, and enhanced robustness of the proposed system.

Objective: To design a deep transfer learning framework for modeling fluence map predictions for stereotactic body radiation therapy (SBRT) of adrenal cancer and similar sites that usually have a small number of cases. Approach: We developed a transfer learning framework for adrenal SBRT planning that leverages knowledge in a pancreas SBRT planning model. Treatment plans from the two sites had different dose prescriptions and beam settings but both prioritized gastrointestinal sparing. A base framework was first trained with 100 pancreas cases. This framework consists of two convolutional neural networks (CNN), which predict individual beam doses (BD-CNN) and fluence maps (FM-CNN) sequentially for 9-beam intensity-modulated radiation therapy (IMRT) plans. Forty-five adrenal plans were split into training/validation/test sets with the ratio of 20/10/15. The base BD-CNN was re-trained with transfer learning using 5/10/15/20 adrenal training cases to produce multiple candidate adrenal BD-CNN models. The base FM-CNN was directly used for adrenal cases. The deep learning (DL) plans were evaluated by several clinically relevant dosimetric endpoints, producing a percentage score relative to the clinical plans. Main results: Transfer learning significantly reduced the number of training cases and training time needed to train such a DL framework. The adrenal transfer learning model trained with 5/10/15/20 cases achieved validation plan scores of 85.4/91.2/90.7/89.4, suggesting that model performance saturated with 10 training cases. Meanwhile, a model using all 20 adrenal training cases without transfer learning only scored 80.5. For the final test set, the 5/10/15/20-case models achieved scores of 73.5/75.3/78.9/83.3. Significance: It is feasible to use deep transfer learning to train an IMRT fluence prediction framework. This technique could adapt to different dose prescriptions and beam configurations. This framework potentially enables DL modeling for clinical sites that have a limited dataset, either due to few cases or due to rapid technology evolution.

Vehicle Edge Computing (VEC) leverages compact cloud computing at the mobile network edge to meet the processing and latency needs of vehicles. By bringing computation closer to the vehicles, VEC reduces data transmission, minimizes latency, and boosts performance for compute-intensive applications. However, during peak hours of urban road traffic, the scarce computational resources available at edge servers could pose challenges in fulfilling the processing needs of vehicles. Introducing Unmanned Aerial Vehicles (UAVs) as supplementary edge computing nodes could significantly mitigate the aforementioned issue. In this paper, we propose a flexible edge computing framework in which a fleet of UAVs function as mobile computational service providers, offering computation offloading services to multiple vehicles. We design and optimize a computation offloading model for the UAV-enabled vehicle edge computing environment. The proposed model tackles the task offloading challenge, aiming to optimize UAV revenue and task processing efficiency while considering the constraints of UAVs’ restricted computational power and energy resources. Towards this end, our model jointly considers two key factors: task partitioning and computational resource allocation. To tackle the challenges posed by the aforementioned non-convex optimization problem, we construct a Markov Decision Process (MDP) model for the multi-UAV-enabled mobile edge computing system and introduce an innovative Multi-Agent Deep Reinforcement Learning (MADRL) framework addressing the decision-making challenge represented by MDP model. Comprehensive simulation outcomes illustrate that our devised task offloading technique outperforms other optimization methods.

The sheer increase in the number of connected devices of the 5G and beyond wireless networks has compounded the pressure on smart and dynamic spectrum sharing. Conventional spectrum allocation algorithms such as static assignment, game-theoretic optimization, and single-agent reinforcement learning struggle with the high-dimensional, diverse, and time-varying characteristics of cognitive radio environments. To overcome these difficulties, the proposed study proposes a Multi-Agent Deep Reinforcement Learning (MADRL) framework that can be used to facilitate cooperative and decentralized spectrum management between multiple cognitive radio users. The agents set policies in a collaborative manner, and each can learn the best policies using Deep Q-Networks (DQN) with Actor-Critic system aggregation around each agent independent of the other agent, observing local environmental states, which include channel occupancy, signal-to-interference-plusnoise ratio (SINR), and transmission power levels of the agent. The suggested framework exploits the idea of interagent communication and experience sharing and reduces spectral collisions, increases throughput, and maximizes energy efficiency. The proposed MADRL-based model under dynamic traffic and user mobility conditions has a 32 % improvement in spectrum utilization efficiency, 25% decrease in interference, and 20 % energy saving as compared to conventional Q-learning and single-agent DRL frameworks. Combining multi-agent coordination and deep reinforcement learning offers scalability, adaptability and robustness needed in next-generation cognitive 6 G networks to provide intelligent, selforganizing, and sustainable spectrum management. This research is expected to result in a very adaptive spectrum allocation scheme that will allow improving the performance, reliability, and spectral fairness of a network in heterogeneous and high-density communication scenarios.

Attempts to address the production scheduling problem thus far rely on simplifying assumptions, such as static environment and inflexible size of the problem, which compromises the schedule performance in practice due to many unpredictable disruptions to the system. Thus, the study of scheduling in the presence of real-time events, termed dynamic scheduling, continues to attract attention given the agility, flexibility, and timeliness modern production systems must deliver. Additionally, the changing nature of the manufacturing system also raises new challenges to existing scheduling strategies. At the front-end, the development of advanced data creation and exchange frameworks such as the Internet of things and cyber-physical system and their applications to the industrial environment have created an abundance of industrial data, while at the backend, edge and cloud computing technologies greatly enhance the capacity to process that data. Industrial data must be mined and analyzed so that the investment in infrastructure is not wasted, and the production system managed more effectively and in real-time. Many data-driven technologies have been adopted in scheduling research, a promising candidate among them being reinforcement learning (RL) which is able to build a direct mapping from observation of environment to actions that improve its performance. In this thesis, a deep multi-agent reinforcement learning (deep MARL) architecture is proposed to solve the dynamic scheduling problem (DSP). The deep reinforcement learning (DRL) algorithm is used to train the decentralized scheduling agents, to capture the relationship between information on the factory floor and scheduling objectives, with the aim of making real-time decisions for a manufacturing system with frequent unexpected events. Two major aspects of deep MARL application to DSP are addressed in this work, namely the conversion from traditional static scheduling problem (SSP) to dynamic scheduling in a practical context, and the adaptation of existing deep MARL algorithms to solve the scheduling problem in such an environment. Some impractical constraints of traditional studies are removed to create a research context that is closer to actual practice, result in a scheduling problem of variable size and scope. Specialized state and action representations that can handle the ever-changing specification of problem are developed; the criteria of feature selection in dynamic environment are also discussed. Recent progressions in DRL and MARL research are integrated into the proposed approach after selection and adaptation. In addition, various improvements to common deep MARL architecture are proposed, including the lightweight multilayer perceptron (MLP) encoder that is efficient in handling unstructured industrial data, a training scheme under the multi-agent architecture to improve the stability of training and overall performance, and knowledge-based reward-shaping techniques to decompose the joint reward signal into individual utilities to speed up the learning and encourage cooperative behavior between agents. Simulation studies are then conducted for the ablation study and validation. In the first stage, the performance of the proposed approach, either as individual components or as an integrated model, are tested in iterative simulation runs within which a unique instance of production is created. Meanwhile, a set of DRL-based approaches from recent publications are run in parallel. Results suggest that the contribution of each improvement is significant; the integrated architecture also delivers stronger performance than peer DRL-based approaches. For the validation, a set of priority rules that have strong performance in specified context and are widely applied in actual production scheduling are used as the benchmark. Proposed approach also provides performance gain compared to the strongest rule, with a minor increase in computation cost and negligible latency in decision-making.

In this paper, we address the difficulty of solving large-scale multi-dimensional knapsack instances (MKP), presenting a novel deep reinforcement learning (DRL) framework. In this DRL framework, we train different agents compatible with a discrete action space for sequential decision-making while still satisfying any resource constraint of the MKP. This novel framework incorporates the decision variable values in the 2D DRL where the agent is responsible for assigning a value of 1 or 0 to each of the variables. To the best of our knowledge, this is the first DRL model of its kind in which a 2D environment is formulated, and an element of the DRL solution matrix represents an item of the MKP. Our framework is configured to solve MKP instances of different dimensions and distributions. We propose a K-means approach to obtain an initial feasible solution that is used to train the DRL agent. We train four different agents in our framework and present the results comparing each of them with the CPLEX commercial solver. The results show that our agents can learn and generalize over instances with different sizes and distributions. Our DRL framework shows that it can solve medium-sized instances at least 45 times faster in CPU solution time and at least 10 times faster for large instances, with a maximum solution gap of 0.28% compared to the performance of CPLEX. Furthermore, at least 95% of the items are predicted in line with the CPLEX solution. Computations with DRL also provide a better optimality gap with respect to state-of-the-art approaches.

Reconfigurable intelligent surfaces (RISs) play a vital role in future wireless systems with the capability of enhancing propagation environments by intelligently reflecting the signals toward the target receivers. However, optimal tuning of the phase shifters at the RIS is challenging due to the passive nature of reflective elements and the high complexity of acquiring channel state information (CSI). Furthermore, the joint active beamforming and RIS reflection beam design is a tedious task due to the high computational complexity and the dynamic nature of the wireless environment. Today’s cellular networks establish data transmission by relying on pre-defined generic beamforming codebooks, which are neither site-specific nor adaptive to the changes in the wireless environment. Moreover, identifying the best beam is typically performed using an exhaustive search approach that prohibits the use of large codebook sizes due to the resulting high beam training overhead. Depending merely on the binary received signal strength, this work develops a multi-agent deep reinforcement learning (MA-DRL) framework that jointly designs the active and the passive reflection beam codebooks for the BS and the RIS, reflectively. To accelerate learning convergence and reduce the search space, the proposed model divides the RIS into multiple partitions and associates beam patterns to the surrounding environments with low computational complexity. Moreover, a hierarchical beam training solution is proposed to further reduce the beam training overhead of the single-beam training approach. Simulation results show that the proposed MA-DRL approach can provide a 97% beam training overhead reduction over the discrete Fourier transform (DFT) codebook.

Deep reinforcement learning (DRL) has superior autonomous decision-making capabilities, combining deep learning and reinforcement learning (RL). Unlike DRL employs deep neural networks (DNNs), broad RL (BRL) adopts the broad learning system (BLS) that is established with flat networks to generate the strategy. This article proposes the multiagent adaptive broad-DRL (ABDRL) approach for traffic light control (TLC), which combines the broad network with the deep network structure. Specifically, the structure of ABDRL first expands in the form of flatted broad networks. Then, the feature representation module that contains DNNs is employed to extract the critical traffic information. In addition, experiences sampled randomly by the experience replay mechanism cannot reflect the current training status of the agent effectively. In order to alleviate the impacts caused by random sampling, the forgetful experience mechanism (FEM) is incorporated into ABDRL. The FEM enables the agent to discriminate the importance of experiences stored in the experience reply buffer to improve robustness and adaptability. We validate the effectiveness of ABDRL in TLC, and the results illustrate the optimality and robustness of ABDRL over the state-of-the-art multiagent DRL (MADRL) algorithms.