State Representation Learning for Multi-agent Deep Deterministic Policy Gradient

In the future, mixed traffic flow will include two types of vehicles: connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs). CAVs emerge as new solutions to disrupt the traditional transportation system. This new solution shares real-time data with each other and the roadside units (RSU) for network management. Reinforcement learning (RL) is a promising approach for traffic signal management in complex urban areas by leveraging information gathered from CAVs. In particular, coordinating signal management at many intersections is a critical challenge in multi-agent reinforcement learning (MARL). According to this vision, we propose an approach that combines an actor–critic network–based multi-agent deep deterministic policy gradient (MADDPG) model and a rerouting technique (RT) to increase traffic performance in vehicular networks. This algorithm overcomes the inherent non-stationary of Q-learning and the high variance of policy gradient (PG) algorithms. Based on centralized learning with decentralized execution, the MADDPG model employs one actor and one critic for each agent. The actor network uses local information to execute actions, while the critic network is trained with extra information, including the states and actions of other agents. Through a centralized learning process, agents can coordinate with each other, diminishing the influence of an unstable environment. Unlike previous studies, we not only manage traffic light systems but also consider the effect of platooning vehicles on increasing throughput. Experimental results show that our model outperforms other models in terms of traffic performance in different scenarios.

Under the background of intelligent ship development, a multi-ship cooperative collision avoidance model based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is proposed to address the multi-ship collision avoidance problem. The proposed model incorporates an innovative Route Exchange (RE) mechanism that enables real-time sharing of ship route information, thereby facilitating coordinated navigation decisions. A novel reward function architecture is systematically designed to integrate RE, enabling dynamic collision risk assessment while optimizing individual ship route adjustments through collaborative policy learning. The proposed approach adopts the "centralized training with decentralized execution (CTDE)" framework of MADDPG, where each ship is treated as an independent intelligent agent equipped with its own Actor and Critic networks. During the training phase, the Critic network integrates global information to evaluate inter-ship interactions. In the execution phase, the Actor network leverages local observations to formulate route adjustment strategies, optimizing cooperative multi-ship collision avoidance policies. Simulation results demonstrate that the proposed model effectively improves joint decision-making strategies, enhances route safety and overall navigational efficiency, and reduces collision risks. This research provides valuable insights into the intelligent development of autonomous ship navigation systems.

Multiagent reinforcement learning (RL) is widely used and can successfully solve many problems in the real world. In the multiagent RL system, a global critic network is used to guide each agent’s strategy to update to learn the most beneficial strategy for the collective. However, the global critic network also makes the current agent’s learning be affected by other agents’ strategies, which leads to unstable learning. To solve this problem, we propose dynamic decomposed multiagent deep deterministic policy gradient (DD-MADDPG): a new network that considers both global and local evaluations and adaptively adjusts the agent’s attention to the two evaluations. Besides, the use of the experience replay buffer by multiagent deep deterministic policy gradient (MADDPG) produces outdated experience, and the outdated strategies of other agents further affect the learning of the current agent. To reduce the influence of other agents’ outdated experience, we propose TD-Error and Time-based experience sampling (T2-PER) based on DD-MADDPG. We evaluate the proposed algorithm’s performance according to the learning stability and the average return obtained by the agents. We have conducted experiments in the MPE environment. The results show that the proposed method has better stability and higher learning efficiency than MADDPG and has a certain generalization ability.

The lateral control of the vehicle is significant for reducing the rollover risk of high-speed cars and improving the stability of the following vehicle. However, the existing car-following (CF) models rarely consider lateral control. Therefore, a CF model with combined longitudinal and lateral control is constructed based on the three degrees of freedom vehicle dynamics model and reinforcement learning method. First, 100 CF segments were selected from the OpenACC database, including 50 straight and 50 curved road trajectories. Afterward, the deep deterministic policy gradient (DDPG) car-following model and multi-agent deep deterministic policy gradient (MADDPG) car-following model were constructed based on the deterministic policy gradient theory. Finally, the models are trained with the extracted trajectory data and verified by comparison with the observed data. The results indicate that the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model are both safer and more comfortable than the human-driven vehicle (HDV) on straight roads and curved roads. Under the premise of safety, the vehicle under the control of the MADDPG model has the highest road traffic flow efficiency. The maximum lateral offset of the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model in straight road conditions is respectively reduced by 80.86% and 71.92%, compared with the HDV, and the maximum lateral offset in the curved road conditions is lessened by 83.67% and 78.95%. The proposed car following model can provide a reference for developing an adaptive cruise control system considering lateral stability.

Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader–follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.

In this paper, we consider unmanned aerial vehicles (UAVs) equipped with a visible light communication (VLC) access point and coordinated multipoint (CoMP) capability that allows users to connect to more than one UAV. UAVs can move in 3-dimensional (3D) at a constant acceleration, where a central server is responsible for synchronization and cooperation among UAVs. The effect of accelerated movement in UAV is necessary to be considered. Unlike most existing works, we examine the effects of variable speed on kinetics and radio resource allocations. For the proposed system model, we define two different time scales. In the frame, the acceleration of each UAV is specified, and in each slot, radio resources are allocated. Our goal is to formulate a multi-objective optimization problem where the total data rate is maximized, and the total communication power consumption is minimized simultaneously. To handle this multi-objective optimization, we first apply the scalarization method and then apply multi-agent deep deterministic policy gradient (MADDPG). We improve this solution method by adding two critic networks together with two-stage resource allocation.

Multi-agent combat is a combat scenario in multiagent reinforcement learning (MARL). In this combat, agents use reinforcement learning methods to learn optimal policies. Actually, policy may be changed, which leads to a non-stationary environment. In this case, it is difficult to predict opponents' policies. Many reinforcement learning methods try to solve nonstationary problems. Most of the previous works put all agents into a frame and model their policies to deal with non-stationarity of environments. But, in a combat environment, opponents can not be in the same frame as our agents. We group opponents and our agents into two frames, only considering opponents as a part of the environment. In this paper, we focus on the problem of modelling opponents' policies in non-stationary environments. To solve this problem, we propose an algorithm called Additional Opponent Characteristics Multi-agent Deep Deterministic Policy Gradient (AOC-MADDPG) with the following contributions: (1) We propose a new actor-critic framework to deal with nonstationarity of environments in MARL, so that agents can adapt to more complex environments. (2) A model for opponents' policies is built by introducing observations and actions of the opponents into the critic network as additional characteristics. We evaluate our AOC-MADDPG algorithm in two multi-agent combat environments. As a result, our approach significantly outperforms the baseline. Agents trained by our method can get higher rewards in non-stationary environments.

Electric vehicles (EVs) are a sustainable transportation solution with environmental benefits and energy efficiency. However, their popularity has raised challenges in locating appropriate charging stations, especially in cities with limited infrastructure and dynamic charging demands. To address this, we propose a multi-agent deep deterministic policy gradient (MADDPG) method for optimal EV charging station recommendations, considering real-time traffic conditions. Our approach aims to minimize total travel time in a stochastic environment for efficient smart transportation management. We adopt a centralized learning and decentralized execution strategy, treating each region of charging stations as an individual agent. Agents cooperate to recommend optimal charging stations based on various incentive functions and competitive contexts. The problem is modeled as a Markov game, suitable for analyzing multi-agent decisions in stochastic environments. Intelligent transportation systems provide us with traffic information, and each charging station feeds relevant data to the agents. Our MADDPG method is challenged with a substantial number of EV requests, enabling efficient handling of dynamic charging demands. Simulation experiments compare our method with DDPG and deterministic approaches, considering different distributions and EV numbers. The results highlight MADDPG’s superiority, emphasizing its value for sustainable urban mobility and efficient EV charging station scheduling.

A multi-agent deep reinforcement learning (MADRL) is a promising approach to challenging problems in wireless environments involving multiple decision-makers (or actors) with high-dimensional continuous action space. In this paper, we present a MADRL-based approach that can jointly optimize precoders to achieve the outer-boundary, called pareto-boundary, of the achievable rate region for a multiple-input single-output (MISO) interference channel (IFC). In order to address two main challenges, namely, multiple actors (or agents) with partial observability and multi-dimensional continuous action space in MISO IFC setup, we adopt a multi-agent deep deterministic policy gradient (MA-DDPG) framework in which decentralized actors with partial observability can learn a multi-dimensional continuous policy in a centralized manner with the aid of shared critic with global information. Meanwhile, we will also address a phase ambiguity issue with the conventional complex baseband representation of signals widely used in radio communications. In order to mitigate the impact of phase ambiguity on training performance, we propose a training method, called phase ambiguity elimination (PAE), that leads to faster learning and better performance of MA-DDPG in wireless communication systems. The simulation results exhibit that MA-DDPG is capable of learning a near-optimal precoding strategy in a MISO IFC environment.

The distributed blocking flowshop scheduling problem (DBFSP) with new job insertions is studied. Rescheduling all remaining jobs after a dynamic event like a new job insertion is unreasonable to an actual distributed blocking flowshop production process. A deep reinforcement learning (DRL) algorithm is proposed to optimise the job selection model, and local modifications are made on the basis of the original scheduling plan when new jobs arrive. The objective is to minimise the total completion time deviation of all products so that all jobs can be finished on time to reduce the cost of storage. First, according to the definitions of the dynamic DBFSP problem, a DRL framework based on multi‐agent deep deterministic policy gradient (MADDPG) is proposed. In this framework, a full schedule is generated by the variable neighbourhood descent algorithm before a dynamic event occurs. Meanwhile, all newly added jobs are reordered before the agents make decisions to select the one that needs to be scheduled most urgently. This study defines the observations, actions and reward calculation methods and applies centralised training and distributed execution in MADDPG. Finally, a comprehensive computational experiment is carried out to compare the proposed method with the closely related and well‐performing methods. The results indicate that the proposed method can solve the dynamic DBFSP effectively and efficiently.

As a critical and prevalent service in future mobile networks, virtual reality (VR) is latency-sensitive and power-hungry, bringing out the optimization problem of trade-off among power saving, delay, and resource utilization. Content caching and render offloading are deemed as promising solutions to meet the stringent requirements of VR on data transmission speed and end-to-end latency. In this article, we propose a novel distributed computing framework based on multi-agent deep deterministic policy gradient (MADDPG) for joint optimizing terminal-cooperative caching and offloading for VR tasks. Since the individual VR user can hardly reach the optimal actions based on its limited local observed states and samples, MADDPG with centralized training and distributed execution is exploited to solve the above challenge. In addition, the generative adversarial network (GAN) is introduced to obtain experience-enhanced agents in the offline training phase and to achieve an optimal allocation to minimize energy consumption in the online inferring phase. The Nash equilibrium is proven in the case that the distribution of finite real VR data samples is well imitated and complemented by GAN. Numerical results demonstrate that our algorithm has significant superiorities in terms of convergence performance and energy consumption over other benchmarks.

Non-orthogonal multiple access (NOMA) and reconfigurable intelligent surfaces (RISs) are recognized as key technologies for beyond 5G and 6G wireless communications. To address the high computational complexity and non-convex optimization challenges, this letter proposes an optimization framework based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. The proposed framework jointly makes use of sensor grouping, power allocation, an RIS computation strategy, and phase shifts to minimize the remote state estimation (RSE) error. Simulation results demonstrate that the MADDPG algorithm, when applied in an RIS-assisted NOMA system, significantly reduces the RSE error.

With increasing travel demand, highway congestion and accidents have become more frequent. As an essential component of intelligent transportation systems (ITS), Hard Shoulder Running (HSR) provides a dynamic strategy to mitigate congestion by temporarily opening shoulder lanes, yet traditional methods often fail to adapt to real-time traffic changes and overlook the shoulder’s critical role in ensuring emergency vehicle access. This study proposes a novel HSR optimization framework based on the Long Short-Term Memory (LSTM) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm, integrated with an improved A* algorithm for EV lane clearing. LSTM is used to extract temporal features from traffic data to support intelligent decision-making in MADDPG, enhanced with prioritized experience replay and importance sampling. The EV lane-clearing task is formulated using graph theory, and the improved A* algorithm determines the optimal path for clearing. A simulation case was developed using Simulation of Urban MObility (SUMO) for a section of the Jinan City Ring Highway, China, evaluating four levels of traffic service. Results show the proposed method reduces total travel time by 14.6%, Time Integrated Time-to-Collision (TIT) by 45.2%, and CO2 emissions by 11.9%. Additionally, with EV intervention, braking times are reduced by up to 376.3% and travel time by 18.1% using the improved A* strategy. These findings demonstrate that the integrated LSTM-MADDPG and A* approach effectively enhances highway traffic efficiency, safety, and sustainability under complex real-world conditions.

In the future mobile edge networks, the Internet of things (IoT) applications will be latency-sensitive and computationally intensive. Given the resource limitation of IoT devices, mobile edge computing (MEC) servers are critical to support the efficient processing of IoT tasks. Since MEC servers attached to the ground base stations are generally deployed in fixed locations and vulnerable to physical damage, the unmanned aerial vehicle (UAV) and satellite-assisted MEC framework has been proposed to leverage the flexibility of UAVs and the broad coverage of satellites. However, efficient utilization of the UAV/satellite resources is challenging for the static ground IoT devices because of the dynamic in terms of aerial and space network topology and IoT task arrival rates. To adapt to the changing environment and utilize the interaction among multiple UAVs, we propose a multi-agent deep deterministic policy gradient (MADDPG) framework to jointly optimize the traveling routes of multi-UAVs and the offloading decision of IoT devices. To minimize the processing cost in terms of task processing latency and energy consumption of IoT devices, cooperative UAVs can help find the optimal task offloading location for each IoT device. Simulation results show the proposed algorithm based on MADDPG can averagely decrease 20% of the above processing cost compared with the benchmark approach.

Due to the scarcity of spectrum resources in Non-orthogonal Multiple Access (NOMA) systems and insufficient satellite-ground integration in satellite Internet of Things (IoT), this paper investigates its issue in spectrum resource management. We propose a resource allocation method based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) for NOMA enabled satellite IoT. We formulate the spectrum allocation problem of the satellite-ground integrated network as a distributed optimization problem. Then we decouple the problem into two sub-problems. Firstly, a user grouping method based on matching coefficients is defined, and a Linear Programming (LP) method is utilized for obtaining solution. Secondly, the power allocation problem is transformed into a multi-agent problem, where MADDPG is employed to allocate the power. Through this approach, the system is capable of real-time user association and spectrum resource allocation optimization, achieving optimal user grouping while maximizing system transmission rate. Based on the simulation results, the MADDPG-based method demonstrates fast convergence within 100 training iterations. The proposed MADDPG-based resource management method also achieves increased system transmission rate with more effective matching outcomes over Deep Deterministic Policy Gradient (DDPG), Orthogonal Multiple Access (OMA), and random allocation baselines.