Multiagent Deep-Reinforcement-Learning-Based Resource Allocation for Heterogeneous QoS Guarantees for Vehicular Networks

Vehicular networks face significant challenges in resource allocation due to heterogeneous Quality-of-Service (QoS) requirements and dynamic channel conditions. This paper proposes an efficient resource allocation framework for heterogeneous vehicular networks to address diverse QoS requirements in dynamic environments. We introduce a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm enhanced by Graph Attention Networks (GATs), named MADDPG-GAT, to optimize sub-bands allocation and power control for Vehicle-to-Vehicle (V2V) links, while minimizing interference to Vehicle-to-Infrastructure (V2I) links. By constructing a hierarchical relationship graph to capture intra-group and inter-group interactions among vehicles, the proposed MADDPG-GAT leverages local and global embeddings for cooperative decision-making. The framework employs centralized offline training for efficient policy learning and decentralized online execution for real-time decision-making with low overhead, with normalized reward function to enhance convergence and training stability. Simulation results demonstrate that our approach outperforms baseline methods in terms of system performance and robustness.

The Dual Active Bridge converter (DABC), known for its bidirectional power transfer capability and high efficiency, plays a crucial role in various applications, particularly in electric vehicles (EVs), where it facilitates energy storage, battery charging, and grid integration. The Dual Active Bridge Converter (DABC), when paired with a high-performance CLLC filter, is well-regarded for its ability to transfer power bidirectionally with high efficiency, making it valuable across a range of energy applications. While these features make the DABC highly efficient, they also complicate controller design due to nonlinear behavior, fast switching, and sensitivity to component variations. We have used a Fractional-order PID (FOPID) controller to benefit from the simple structure of classical PID controllers with lower complexity and improved flexibility because of additional filtering gains adopted in this method. However, for a FOPID controller to operate effectively under real-time conditions, its parameters must adapt continuously to changes in the system. To achieve this adaptability, a Multi-Agent Reinforcement Learning (MARL) approach is adopted, where each gain of the controller is tuned individually using the Deep Deterministic Policy Gradient (DDPG) algorithm. This structure enhances the controller’s ability to respond to external disturbances with greater robustness and adaptability. Meanwhile, finding the best initial gains in the RL structure can decrease the overall efficiency and tracking performance of the controller. To overcome this issue, Grey Wolf Optimization (GWO) algorithm is proposed to identify the most suitable initial gains for each agent, providing faster adaptation and consistent performance during the training process. The complete approach is tested using a Hardware-in-the-Loop (HIL) platform, where results confirm accurate voltage control and resilient dynamic behavior under practical conditions. In addition, the controller’s performance was validated under a battery management scenario where the DAB converter interacts with a nonlinear lithium-ion battery. The controller successfully regulated the State of Charge (SOC) through automated charging and discharging transitions, demonstrating its real-time adaptability for BMS-integrated EV systems. Consequently, the proposed MARL-FOPID controller reported better disturbance-rejection performance in different working cases compared to other conventional methods.

The convergence of mobile edge computing (MEC) and blockchain is transforming the current computing services in mobile networks, by offering task offloading solutions with security enhancement empowered by blockchain mining. Nevertheless, these important enabling technologies have been studied separately in most existing works. This article proposes a novel cooperative task offloading and block mining (TOBM) scheme for a blockchain-based MEC system where each edge device not only handles data tasks but also deals with block mining for improving the system utility. To address the latency issues caused by the blockchain operation in MEC, we develop a new Proof-of-Reputation consensus mechanism based on a lightweight block verification strategy. A multi-objective function is then formulated to maximize the system utility of the blockchain-based MEC system, by jointly optimizing offloading decision, channel selection, transmit power allocation, and computational resource allocation. We propose a novel distributed deep reinforcement learning-based approach by using a multi-agent deep deterministic policy gradient algorithm. We then develop a game-theoretic solution to model the offloading and mining competition among edge devices as a potential game, and prove the existence of a pure Nash equilibrium. Simulation results demonstrate the significant system utility improvements of our proposed scheme over baseline approaches.

In the fifth generation (5G) wireless communication network, to better offer service for different vertical industry applications, network slicing is needed to meet the differential requirements. In this paper, we study the joint problem of radio resource allocation and power resource allocation in the radio access network (RAN) to improve the radio resource utilization with multi-user demands in smart grid scenario. Different with conventional single-agent algorithm, we propose a non-orthogonal multiple access (NOMA) based multi-agent deep deterministic policy gradient (MADDPG) algorithm to solve this dynamic optimization problem. The scheme determines the remote radio unit (RRU) and physical resource block (PRB) allocation for each user, and updates the neural network to dynamically adjust the resource allocation strategy to maximize the long-term rate of network slices. The reliability and latency requirements of network slicing are guaranteed as well. Simulation results show the convergence and efficiency of the proposed algorithm.

With the development of the IoT (Internet of Things), sensors networks can bring a large amount of valuable data. In addition to be utilized in the local IoT applications, the data can also be traded in the connected edge servers. As an efficient resource allocation mechanism, the double auction has been widely used in the stock and futures markets and can be also applied in the data resource allocation in sensor networks. Currently, there usually exist multiple edge servers running double auctions competing with each other to attract data users (buyers) and producers (sellers). Therefore, the double auction market run on each edge server needs efficient mechanism to improve the allocation efficiency. Specifically, the pricing strategy of the double auction plays an important role on affecting traders’ profit, and thus, will affect the traders’ market choices and bidding strategies, which in turn affect the competition result of double auction markets. In addition, the traders’ trading strategies will also affect the market’s pricing strategy. Therefore, we need to analyze the double auction markets’ pricing strategy and traders’ trading strategies. Specifically, we use a deep reinforcement learning algorithm combined with mean field theory to solve this problem with a huge state and action space. For trading strategies, we use the Independent Parametrized Deep Q‐Network (I‐PDQN) algorithm combined with mean field theory to compute the Nash equilibrium strategies. We then compare it with the fictitious play (FP) algorithm. The experimental results show that the computation speed of I‐PDQN algorithm is significantly faster than that of FP algorithm. For pricing strategies, the double auction markets will dynamically adjust the pricing strategy according to traders’ trading strategies. This is a sequential decision‐making process involving multiple agents. Therefore, we model it as a Markov game. We adopt Multiagent Deep Deterministic Policy Gradient (MADDPG) algorithm to analyze the Nash equilibrium pricing strategies. The experimental results show that the MADDPG algorithm solves the problem faster than the FP algorithm.

• A grid-aware multi-agent reinforcement learning was developed for EV charging. • Public charging data worked with domestic demand to build a real environment. • A dual-head actor optimised the acceptance and cross-site redirection of EV charging. • A centralised attention-based critic and delayed replay buffer ensured stable learning. • The RL policy reduced peak load variability by 40 % and threshold violations by 37 %. As electric vehicles (EV) become more widespread, managing charging demand is critical for grid stability and efficient resource allocation. However, prior work often optimises single sites or treats demand as exogenous, without explicitly modelling grid capacity constraints or the aggregate effects of redirection, and it rarely offers an integrated capacity-aware view that links domestic and public charging under system-level coordination. This study, therefore, develops a multi-agent reinforcement learning framework with cross-site redirection to optimise EV charging operations. The research area focuses on the Perth and Kinross region of Scotland in the UK, using real public charging records from the council database as well as domestic charging statistics from the UK Department for Transport. Charging sites are modelled as cooperative agents using the Multi-Agent Deep Deterministic Policy Gradient algorithm, trained in a spatiotemporal environment that integrates public charging data and simulated domestic demand. We contribute by unifying public and home charging demand in one learning environment, introducing a cross-site actor trained under centralised training and decentralised execution, and incorporating a delayed replay buffer with short-horizon forecasts so redirection aligns with future congestion. The learned policy reduces peak-hour load standard deviation by up to 40 % and lowers cumulative threshold violations by 37 % compared to the baseline. Distinct weekday and weekend strategies emerge, enabling adaptive coordination under varying demand patterns. The study provides interpretable control for EV networks, balancing peak demand and service quality across sites while addressing the system-level coordination.

Industrial edge computing in time-sensitive, heterogeneous environments faces significant challenges in delivering deterministic, low-latency, and fair task offloading under mixed-criticality traffic. Existing solutions struggle with unpredictable delays, resource contention, and the joint optimization of offloading and scheduling. In this work, we propose a task queue mapping mechanism-based architecture that incorporates formal mixed-deterministic traffic modeling under time-sensitive networking, enabling unified computation and network resource management. The joint task offloading and traffic scheduling problem is rigorously formulated as a two-stage Markov decision process with embedded fairness constraints. To address the strong coupling and computational complexity, we develop a fairness-aware deterministic offloading and resource scheduling (FA-DORS) framework, a hierarchical multiagent deep deterministic policy gradient algorithm, enhanced by a contribution-based local reward mechanism to ensure adaptive resource allocation and fairness. Extensive experimental results demonstrate that in the proposed architecture, FA-DORS achieves lower task latency, higher completion rates, improved resource utilization, and enhanced transmission determinism while ensuring fairness in multitask offloading compared to baseline methods. In particular, FA-DORS achieves an 8.89%–39.15% reduction in average end-to-end delay and up to 39.73% reduction in average jitter under high load, compared with the baseline method.

Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader–follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.

Non-orthogonal multiple access (NOMA) and reconfigurable intelligent surfaces (RISs) are recognized as key technologies for beyond 5G and 6G wireless communications. To address the high computational complexity and non-convex optimization challenges, this letter proposes an optimization framework based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. The proposed framework jointly makes use of sensor grouping, power allocation, an RIS computation strategy, and phase shifts to minimize the remote state estimation (RSE) error. Simulation results demonstrate that the MADDPG algorithm, when applied in an RIS-assisted NOMA system, significantly reduces the RSE error.

To address the challenges of real-time decision-making and resource optimization in multi-agent cooperative interception tasks within dynamic environments, this paper proposes a hierarchical framework for reinforcement learning-based interception algorithm (HFRL-IA). By constructing a hierarchical Markov decision process (MDP) model based on dynamic game equilibrium theory, the complex interception task is decomposed into two hierarchically optimized stages: dynamic task allocation and distributed path planning. At the high level, a sequence-to-sequence reinforcement learning approach is employed to achieve dynamic bipartite graph matching, leveraging a graph neural network encoder–decoder architecture to handle dynamically expanding threat targets. At the low level, an improved prioritized experience replay multi-agent deep deterministic policy gradient algorithm (PER-MADDPG) is designed, integrating curriculum learning and prioritized experience replay mechanisms to effectively enhance the interception success rate against complex maneuvering targets. Extensive simulations in diverse scenarios and comparisons with conventional task assignment strategies demonstrate the superiority of the proposed algorithm. Taking a typical scenario of 10 agents intercepting as an example, the HFRL-IA algorithm achieves a 22.51% increase in training rewards compared to the traditional end-to-end MADDPG algorithm, and the interception success rate is improved by 26.37%. This study provides a new methodological framework for distributed cooperative decision-making in dynamic adversarial environments, with significant application potential in areas such as maritime multi-agent security defense and marine environment monitoring.

The large-scale integration of distributed renewable generation (DRG) reduces carbon emissions in power systems, but its intermittency and volatility introduce new challenges for voltage and frequency control. This paper proposes a multi-time-scale reactive power optimization approach that incorporates demand-side response. Firstly, based on the interaction characteristics between sources and loads in the distribution network, a demand-side response mechanism using dynamic time-of-use pricing is designed to explore the source-load matching and tracking capabilities. Secondly, a multi-scale day-ahead reactive power optimization control model is constructed, which comprehensively considers the dynamic response mechanism on the demand side. The objective function of this model aims to minimize network losses and voltage deviations while accounting for the day-ahead price response characteristics of loads. The multi-scale reactive power optimization model is then transformed into a Markov decision process and solved via a locally-observed multi-agent deep deterministic policy gradient algorithm to obtain real-time intraday reactive power and voltage control strategies. Finally, the effectiveness of the proposed method is verified through experiments on the IEEE 33-bus system and an actual power grid. Experimental results show that this method can effectively coordinate various reactive power compensation devices and flexible resources, and improve the efficiency and real-time performance of reactive power and voltage control.

Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management

PASCAL: PopulAtion-Specific Curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms

One of the toughest challenges in the multi-agent deep reinforcement learning (MADRL) is that when the opponents' policies change rapidly, the collaborative agents can't learn well to respond to the opponents' policies effectively. This may lead to a local optimum w.r.t. the learned policy of the collaborative agents may be only locally optimal to the opponents' current policies. To address this problem, we propose a novel algorithm termed Friend-or-Foe Deep Deterministic Policy Gradient (FD2PG), in which the cooperative agents can be trained more robust and have stronger cooperation ability in continuous action space. These collaborative agents can generalize easily and respond correctly, even if their opponents' policies alter. Inspired by the classic Friend-or-Foe Q-learning algorithm (FFQ), we introduce the idea of minimizing the foes and maximizing the friends into the centralized training distributed execution framework, multi-agent deep deterministic policy gradient algorithm (MADDPG), to enhance collaborative agents' robustness and cooperativity. Besides, we introduce a Minimax Multi-Agent Learning (MMAL) method to explore two special equilibriums (the adversarial equilibrium and the coordination equilibrium), which can guarantee the convergence of FD2PG and improve optimization. Extensive fine-grained experiments, including four representative scenario experiments and two scale-performance correlation experiments, were conducted to demonstrate the superior performance of FD2PG comparing with existing baselines.

Multi-access edge computing provides localized resources within mobile networks to address the requirements of emerging latency-sensitive and computing-intensive applications. At the edge, dynamic requests necessitate sophisticated resource management for adaptive network slicing. This involves optimizing resource allocations, scaling functions, and load balancing to utilize only essential resources under constrained network scenarios. However, existing solutions largely assume static slice counts, ignoring the re-optimization overhead associated with management algorithms when slices fluctuate. Moreover, many approaches rely on simplified energy models that overlook intertemporal resource scheduling and are predominantly evaluated through simulations, neglecting critical practical considerations. This paper presents an incremental cooperative Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm for resource management in dynamic edge slicing. The proposed approach optimizes long-term slicing benefits by reducing delay and energy consumption while minimizing retraining overhead in response to slice variations. Furthermore, we implement an urban-wide edge computing testbed based on OpenStack and Kubernetes to validate the algorithm’s performance. Experimental results demonstrate that our incremental MADDPG method outperforms benchmark strategies in aggregated slicing utility and reduces training energy consumption by up to 50% compared to the re-optimization approach.