A Dynamically Adaptive Approach to Reducing Strategic Interference for Multiagent Systems

In the future, mixed traffic flow will include two types of vehicles: connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs). CAVs emerge as new solutions to disrupt the traditional transportation system. This new solution shares real-time data with each other and the roadside units (RSU) for network management. Reinforcement learning (RL) is a promising approach for traffic signal management in complex urban areas by leveraging information gathered from CAVs. In particular, coordinating signal management at many intersections is a critical challenge in multi-agent reinforcement learning (MARL). According to this vision, we propose an approach that combines an actor–critic network–based multi-agent deep deterministic policy gradient (MADDPG) model and a rerouting technique (RT) to increase traffic performance in vehicular networks. This algorithm overcomes the inherent non-stationary of Q-learning and the high variance of policy gradient (PG) algorithms. Based on centralized learning with decentralized execution, the MADDPG model employs one actor and one critic for each agent. The actor network uses local information to execute actions, while the critic network is trained with extra information, including the states and actions of other agents. Through a centralized learning process, agents can coordinate with each other, diminishing the influence of an unstable environment. Unlike previous studies, we not only manage traffic light systems but also consider the effect of platooning vehicles on increasing throughput. Experimental results show that our model outperforms other models in terms of traffic performance in different scenarios.

With increasing travel demand, highway congestion and accidents have become more frequent. As an essential component of intelligent transportation systems (ITS), Hard Shoulder Running (HSR) provides a dynamic strategy to mitigate congestion by temporarily opening shoulder lanes, yet traditional methods often fail to adapt to real-time traffic changes and overlook the shoulder’s critical role in ensuring emergency vehicle access. This study proposes a novel HSR optimization framework based on the Long Short-Term Memory (LSTM) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm, integrated with an improved A* algorithm for EV lane clearing. LSTM is used to extract temporal features from traffic data to support intelligent decision-making in MADDPG, enhanced with prioritized experience replay and importance sampling. The EV lane-clearing task is formulated using graph theory, and the improved A* algorithm determines the optimal path for clearing. A simulation case was developed using Simulation of Urban MObility (SUMO) for a section of the Jinan City Ring Highway, China, evaluating four levels of traffic service. Results show the proposed method reduces total travel time by 14.6%, Time Integrated Time-to-Collision (TIT) by 45.2%, and CO2 emissions by 11.9%. Additionally, with EV intervention, braking times are reduced by up to 376.3% and travel time by 18.1% using the improved A* strategy. These findings demonstrate that the integrated LSTM-MADDPG and A* approach effectively enhances highway traffic efficiency, safety, and sustainability under complex real-world conditions.

Financial investigation decision is a complex affair that highly relies on expert knowledge. In the era of big data, it remains practical to make intelligent financial investigation decisions via computing technology. As a consequence, this work introduces reinforcement learning to search for effective decision schemes that are adaptive to the diverse financial environment. Therefore, an intelligent financial investment decision model based on multi-agent reinforcement learning is proposed in this paper. First, the long short-term memory (LSTM) model is utilized to strengthen financial investment data. Then, the hidden Markov model (HMM) is utilized to make a preliminary judgment on financial investment strategy. Next, a multi-agent deep deterministic policy gradient (MADDPG) is employed to set the goal of agent reinforcement and determine decision indicators based on the financial market environment. Multiple agents (investors) form a cooperative strategy through learning and interaction to achieve the optimal investment decision in the financial market. The experimental results on realistic data indicate that the proposal can play an important role in investment decision-making.

Multi-agent combat is a combat scenario in multiagent reinforcement learning (MARL). In this combat, agents use reinforcement learning methods to learn optimal policies. Actually, policy may be changed, which leads to a non-stationary environment. In this case, it is difficult to predict opponents' policies. Many reinforcement learning methods try to solve nonstationary problems. Most of the previous works put all agents into a frame and model their policies to deal with non-stationarity of environments. But, in a combat environment, opponents can not be in the same frame as our agents. We group opponents and our agents into two frames, only considering opponents as a part of the environment. In this paper, we focus on the problem of modelling opponents' policies in non-stationary environments. To solve this problem, we propose an algorithm called Additional Opponent Characteristics Multi-agent Deep Deterministic Policy Gradient (AOC-MADDPG) with the following contributions: (1) We propose a new actor-critic framework to deal with nonstationarity of environments in MARL, so that agents can adapt to more complex environments. (2) A model for opponents' policies is built by introducing observations and actions of the opponents into the critic network as additional characteristics. We evaluate our AOC-MADDPG algorithm in two multi-agent combat environments. As a result, our approach significantly outperforms the baseline. Agents trained by our method can get higher rewards in non-stationary environments.

This paper investigates the problem of age of information (AoI) aware radio resource management for a platooning system. Multiple autonomous platoons exploit the cellular wireless vehicle-to-everything (C-V2X) communication technology to disseminate the cooperative awareness messages (CAMs) to their followers while ensuring timely delivery of safety-critical messages to the Road-Side Unit (RSU). To lower the computational load at the RSU and cope with the challenges of dynamic channel conditions, we exploit a distributed resource allocation framework based on multi-agent reinforcement learning (MARL), where each platoon leader (PL) acts as an agent and interacts with the environment to learn its optimal policy. Motivated by the existing literature in RL, we propose two novel MARL frameworks based on the multi-agent deep deterministic policy gradient (MADDPG), named Modified MADDPG, and Modified MADDPG with task decomposition. Both algorithms train two critics with the following goals: A global critic which estimates the global expected reward and motivates the agents toward a cooperating behavior and an exclusive local critic for each agent that estimates the local individual reward. Furthermore, based on the tasks each agent has to accomplish, in the second algorithm, the holistic individual reward of each agent is decomposed into multiple sub-reward functions where task-wise value functions are learned separately. Numerical results indicate our proposed algorithms' effectiveness compared with other contemporary RL frameworks, e.g., federated reinforcement learning (FRL) in terms of AoI performance and CAM message transmission probability.

The Dual Active Bridge converter (DABC), known for its bidirectional power transfer capability and high efficiency, plays a crucial role in various applications, particularly in electric vehicles (EVs), where it facilitates energy storage, battery charging, and grid integration. The Dual Active Bridge Converter (DABC), when paired with a high-performance CLLC filter, is well-regarded for its ability to transfer power bidirectionally with high efficiency, making it valuable across a range of energy applications. While these features make the DABC highly efficient, they also complicate controller design due to nonlinear behavior, fast switching, and sensitivity to component variations. We have used a Fractional-order PID (FOPID) controller to benefit from the simple structure of classical PID controllers with lower complexity and improved flexibility because of additional filtering gains adopted in this method. However, for a FOPID controller to operate effectively under real-time conditions, its parameters must adapt continuously to changes in the system. To achieve this adaptability, a Multi-Agent Reinforcement Learning (MARL) approach is adopted, where each gain of the controller is tuned individually using the Deep Deterministic Policy Gradient (DDPG) algorithm. This structure enhances the controller’s ability to respond to external disturbances with greater robustness and adaptability. Meanwhile, finding the best initial gains in the RL structure can decrease the overall efficiency and tracking performance of the controller. To overcome this issue, Grey Wolf Optimization (GWO) algorithm is proposed to identify the most suitable initial gains for each agent, providing faster adaptation and consistent performance during the training process. The complete approach is tested using a Hardware-in-the-Loop (HIL) platform, where results confirm accurate voltage control and resilient dynamic behavior under practical conditions. In addition, the controller’s performance was validated under a battery management scenario where the DAB converter interacts with a nonlinear lithium-ion battery. The controller successfully regulated the State of Charge (SOC) through automated charging and discharging transitions, demonstrating its real-time adaptability for BMS-integrated EV systems. Consequently, the proposed MARL-FOPID controller reported better disturbance-rejection performance in different working cases compared to other conventional methods.

In this article, we explore how to optimize task allocation for robot swarms in dynamic environments, emphasizing the necessity of formulating robust, flexible, and scalable strategies for robot cooperation. We introduce a novel framework using a decentralized partially observable Markov decision process (Dec-POMDP), specifically designed for distributed robot swarm networks. At the core of our methodology is the local information aggregation multiagent deep deterministic policy gradient (LIA-MADDPG) algorithm, which merges centralized training with distributed execution. During the centralized training phase, a local information aggregation (LIA) module is meticulously designed to gather critical data from neighboring robots, enhancing decision-making efficiency. In the distributed execution phase, a strategy improvement method is proposed to dynamically adjust task allocation based on changing and partially observable environmental conditions. Our empirical evaluations show that the LIA module can be seamlessly integrated into various centralized training and decentralized execution (CTDE)-based multiagent reinforcement learning (MARL) methods, significantly enhancing their performance. Additionally, by comparing LIA-MADDPG with six conventional reinforcement learning algorithms and a heuristic algorithm, we demonstrate its superior scalability, rapid adaptation to environmental changes, and ability to maintain both stability and convergence speed. These results underscore LIA-MADDPG's outstanding performance and its potential to significantly improve dynamic task allocation in robot swarms through enhanced local collaboration and adaptive strategy execution.

• A grid-aware multi-agent reinforcement learning was developed for EV charging. • Public charging data worked with domestic demand to build a real environment. • A dual-head actor optimised the acceptance and cross-site redirection of EV charging. • A centralised attention-based critic and delayed replay buffer ensured stable learning. • The RL policy reduced peak load variability by 40 % and threshold violations by 37 %. As electric vehicles (EV) become more widespread, managing charging demand is critical for grid stability and efficient resource allocation. However, prior work often optimises single sites or treats demand as exogenous, without explicitly modelling grid capacity constraints or the aggregate effects of redirection, and it rarely offers an integrated capacity-aware view that links domestic and public charging under system-level coordination. This study, therefore, develops a multi-agent reinforcement learning framework with cross-site redirection to optimise EV charging operations. The research area focuses on the Perth and Kinross region of Scotland in the UK, using real public charging records from the council database as well as domestic charging statistics from the UK Department for Transport. Charging sites are modelled as cooperative agents using the Multi-Agent Deep Deterministic Policy Gradient algorithm, trained in a spatiotemporal environment that integrates public charging data and simulated domestic demand. We contribute by unifying public and home charging demand in one learning environment, introducing a cross-site actor trained under centralised training and decentralised execution, and incorporating a delayed replay buffer with short-horizon forecasts so redirection aligns with future congestion. The learned policy reduces peak-hour load standard deviation by up to 40 % and lowers cumulative threshold violations by 37 % compared to the baseline. Distinct weekday and weekend strategies emerge, enabling adaptive coordination under varying demand patterns. The study provides interpretable control for EV networks, balancing peak demand and service quality across sites while addressing the system-level coordination.

Introduction To address the urgent need for enhanced grid flexibility in high-penetration distributed photovoltaic (PV) systems, this paper proposes a novel two-stage framework integrating multi-objective cluster partitioning and multi-agent reinforcement learning (MARL) for cooperative peak shaving. Methods First, a comprehensive multi-objective optimization model defines PV cluster partitioning using three domain-specific metrics: available peak shaving margin (APSM), peak shaving synergy (PSS), and node correlation degree (NCD). This approach replaces conventional clustering algorithms, ensuring partitions balance electrical compactness, regulation capacity, and operational coordination. Subsequently, each partitioned cluster acts as an autonomous agent in a hierarchical MARL framework. Leveraging the multi-agent deep deterministic policy gradient (MADDPG) algorithm, agents collaboratively optimize active/reactive power allocation among clusters while coordinating unit-level control within clusters. Results Validated on the IEEE-33 node system, the strategy achieves a 29.39% reduction in peak-to-valley difference and 54.44% improvement in power balance deviation, significantly smoothing net load curves and enhancing voltage stability. Discussion This work demonstrates how MARL-driven coordination unlocks the inherent flexibility of distributed PV clusters, providing a scalable solution for grid-edge resource participation in system-level peak shaving.

Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader–follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.

The distributed blocking flowshop scheduling problem (DBFSP) with new job insertions is studied. Rescheduling all remaining jobs after a dynamic event like a new job insertion is unreasonable to an actual distributed blocking flowshop production process. A deep reinforcement learning (DRL) algorithm is proposed to optimise the job selection model, and local modifications are made on the basis of the original scheduling plan when new jobs arrive. The objective is to minimise the total completion time deviation of all products so that all jobs can be finished on time to reduce the cost of storage. First, according to the definitions of the dynamic DBFSP problem, a DRL framework based on multi‐agent deep deterministic policy gradient (MADDPG) is proposed. In this framework, a full schedule is generated by the variable neighbourhood descent algorithm before a dynamic event occurs. Meanwhile, all newly added jobs are reordered before the agents make decisions to select the one that needs to be scheduled most urgently. This study defines the observations, actions and reward calculation methods and applies centralised training and distributed execution in MADDPG. Finally, a comprehensive computational experiment is carried out to compare the proposed method with the closely related and well‐performing methods. The results indicate that the proposed method can solve the dynamic DBFSP effectively and efficiently.

Due to the scarcity of spectrum resources in Non-orthogonal Multiple Access (NOMA) systems and insufficient satellite-ground integration in satellite Internet of Things (IoT), this paper investigates its issue in spectrum resource management. We propose a resource allocation method based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) for NOMA enabled satellite IoT. We formulate the spectrum allocation problem of the satellite-ground integrated network as a distributed optimization problem. Then we decouple the problem into two sub-problems. Firstly, a user grouping method based on matching coefficients is defined, and a Linear Programming (LP) method is utilized for obtaining solution. Secondly, the power allocation problem is transformed into a multi-agent problem, where MADDPG is employed to allocate the power. Through this approach, the system is capable of real-time user association and spectrum resource allocation optimization, achieving optimal user grouping while maximizing system transmission rate. Based on the simulation results, the MADDPG-based method demonstrates fast convergence within 100 training iterations. The proposed MADDPG-based resource management method also achieves increased system transmission rate with more effective matching outcomes over Deep Deterministic Policy Gradient (DDPG), Orthogonal Multiple Access (OMA), and random allocation baselines.

Many cooperative multi-agent problems require agents to learn individual tasks while contributing to the collective success of the group. This is a challenging task for current state-of-the-art multi-agent reinforcement algorithms that are designed to either maximize the global reward of the team or the individual local rewards. The problem is exacerbated when either of the rewards is sparse leading to unstable learning. To address this problem, we present Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG): a novel cooperative multi-agent reinforcement learning framework that simultaneously learns to maximize the global and local rewards. We evaluate our solution on the challenging defensive escort team problem and show that our solution achieves a significantly better and more stable performance than the direct adaptation of the MADDPG algorithm.

With the development of unmanned combat air vehicles (UCAVs) and artificial intelligence (AI), within visual range (WVR) air combat confrontations utilizing intelligent UCAVs are expected to be widely used in future air combats. As controlling highly dynamic and uncertain WVR air combats from the ground stations of the UCAV is not feasible, it is necessary to develop an algorithm that can generate highly intelligent air combat strategies in order to enable UCAV to independently complete air combat missions. In this paper, a 1-vs.-1 WVR air combat strategy generation algorithm is proposed using the multi-agent deep deterministic policy gradient (MADDPG). A 1-vs.-1 WVR air combat is modeled as a two-player zero-sum Markov game (ZSMG). A method for predicting the position of the target is introduced into the model in order to enable the UCAV to predict the target’s actions and position. Moreover, to ensure that the UCAV is not limited by the constraints of the basic fighter maneuver (BFM) library, the action space is considered to be a continuous one. At the same time, a potential-based reward shaping method is proposed in order to improve the efficiency of the air combat strategy generation algorithm. Finally, the efficiency of the air combat strategy generation algorithm and the intelligence level of the resulting strategy is verified through simulation experiments. The results show that an air combat strategy using target position prediction is superior to the one that does not use target position prediction.

Time-triggered Ethernet combines time-triggered and event-triggered communication, and is suitable for fields with high real-time requirements. Aiming at the problem that the traditional scheduling algorithm is not effective in scheduling event-triggered messages, a message scheduling algorithm based on multi-agent reinforcement learning (MADDPG, Multi-Agent Deep Deterministic Policy Gradient) and a hybrid algorithm combining SMT (Satisfiability Modulo Theories) solver and MADDPG are proposed. This method aims to optimize the scheduling of event-triggered messages while maintaining the uniformity of time-triggered message scheduling, providing more time slots for event-triggered messages, and reducing their waiting time and end-to-end delay. Through the designed scheduling software, in the experiment, compared with the SMT-based algorithm and the traditional DQN (Deep Q-Network) algorithm, the new method shows better load balance and lower message jitter, and it is verified in the OPNET simulation environment that it can effectively reduce the delay of event-triggered messages.