Friend-or-Foe Deep Deterministic Policy Gradient

A multi-agent deep reinforcement learning (MADRL) is a promising approach to challenging problems in wireless environments involving multiple decision-makers (or actors) with high-dimensional continuous action space. In this paper, we present a MADRL-based approach that can jointly optimize precoders to achieve the outer-boundary, called pareto-boundary, of the achievable rate region for a multiple-input single-output (MISO) interference channel (IFC). In order to address two main challenges, namely, multiple actors (or agents) with partial observability and multi-dimensional continuous action space in MISO IFC setup, we adopt a multi-agent deep deterministic policy gradient (MA-DDPG) framework in which decentralized actors with partial observability can learn a multi-dimensional continuous policy in a centralized manner with the aid of shared critic with global information. Meanwhile, we will also address a phase ambiguity issue with the conventional complex baseband representation of signals widely used in radio communications. In order to mitigate the impact of phase ambiguity on training performance, we propose a training method, called phase ambiguity elimination (PAE), that leads to faster learning and better performance of MA-DDPG in wireless communication systems. The simulation results exhibit that MA-DDPG is capable of learning a near-optimal precoding strategy in a MISO IFC environment.

Aerial base stations (AeBSs), as crucial components of air-ground integrated networks, are widely employed in cloud computing, disaster relief, and various applications. How to quickly and efficiently deploy multi-AeBSs for higher capacity gain has become a key research issue. In this paper, we address the 3D deployment optimization problem of multi-AeBSs with the objective of maximizing system capacity. To overcome communication overhead and privacy challenges in multi-agent deep reinforcement learning (MADRL), we propose a federated deep deterministic policy gradient (Fed-DDPG) algorithm for the multi-AeBS deployment decision. Specifically, a high-altitude platform (HAP)-assisted multi-AeBS deployment architecture is designed, in which low-altitude AeBS act as the local nodes to train its own deployment decision model, while the HAP acts as the global node to aggregate the weights of local models. In this architecture, AeBSs do not exchange raw data, addressing data privacy concerns and reducing communication overhead. Simulation results show that the proposed algorithm outperforms fully distributed MADRL algorithms and closely approximates the performance of multi-agent deep deterministic policy gradient (MADDPG), which requires global information during training, but with less training time.

Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management

This paper presents a parameter selection method of notch filters for suppressing mechanical resonances in industrial servo systems using the deep deterministic policy gradient (DDPG) algorithm. Several methods for tuning the notch filter parameters were studied such as the fast-Fourier-transform-based methods, extended-Kalman-filter-based methods and adaptive notch filter methods. However, these methods do not find the Q parameters of notch filters which play an important role in determining the system stability, and do not consider the cases in which multiple notch filters are required. Deep-Q-network-based method was developed to solve these problems, but the notch filter parameter tuning is limited to discrete action spaces. This paper develops a new parameter selection method of notch filters, using the DDPG algorithm. DDPG algorithm, which is a model-free and actor-critic algorithm using deep neural networks, is utilized for its capability to operate over continuous action spaces. Experiments are performed using an actual industrial servo system to demonstrate that the developed parameter selection method successfully finds the notch filter parameters to suppress the resonances of the system.

The large-scale integration of distributed renewable generation (DRG) reduces carbon emissions in power systems, but its intermittency and volatility introduce new challenges for voltage and frequency control. This paper proposes a multi-time-scale reactive power optimization approach that incorporates demand-side response. Firstly, based on the interaction characteristics between sources and loads in the distribution network, a demand-side response mechanism using dynamic time-of-use pricing is designed to explore the source-load matching and tracking capabilities. Secondly, a multi-scale day-ahead reactive power optimization control model is constructed, which comprehensively considers the dynamic response mechanism on the demand side. The objective function of this model aims to minimize network losses and voltage deviations while accounting for the day-ahead price response characteristics of loads. The multi-scale reactive power optimization model is then transformed into a Markov decision process and solved via a locally-observed multi-agent deep deterministic policy gradient algorithm to obtain real-time intraday reactive power and voltage control strategies. Finally, the effectiveness of the proposed method is verified through experiments on the IEEE 33-bus system and an actual power grid. Experimental results show that this method can effectively coordinate various reactive power compensation devices and flexible resources, and improve the efficiency and real-time performance of reactive power and voltage control.

PASCAL: PopulAtion-Specific Curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms

The Dual Active Bridge converter (DABC), known for its bidirectional power transfer capability and high efficiency, plays a crucial role in various applications, particularly in electric vehicles (EVs), where it facilitates energy storage, battery charging, and grid integration. The Dual Active Bridge Converter (DABC), when paired with a high-performance CLLC filter, is well-regarded for its ability to transfer power bidirectionally with high efficiency, making it valuable across a range of energy applications. While these features make the DABC highly efficient, they also complicate controller design due to nonlinear behavior, fast switching, and sensitivity to component variations. We have used a Fractional-order PID (FOPID) controller to benefit from the simple structure of classical PID controllers with lower complexity and improved flexibility because of additional filtering gains adopted in this method. However, for a FOPID controller to operate effectively under real-time conditions, its parameters must adapt continuously to changes in the system. To achieve this adaptability, a Multi-Agent Reinforcement Learning (MARL) approach is adopted, where each gain of the controller is tuned individually using the Deep Deterministic Policy Gradient (DDPG) algorithm. This structure enhances the controller’s ability to respond to external disturbances with greater robustness and adaptability. Meanwhile, finding the best initial gains in the RL structure can decrease the overall efficiency and tracking performance of the controller. To overcome this issue, Grey Wolf Optimization (GWO) algorithm is proposed to identify the most suitable initial gains for each agent, providing faster adaptation and consistent performance during the training process. The complete approach is tested using a Hardware-in-the-Loop (HIL) platform, where results confirm accurate voltage control and resilient dynamic behavior under practical conditions. In addition, the controller’s performance was validated under a battery management scenario where the DAB converter interacts with a nonlinear lithium-ion battery. The controller successfully regulated the State of Charge (SOC) through automated charging and discharging transitions, demonstrating its real-time adaptability for BMS-integrated EV systems. Consequently, the proposed MARL-FOPID controller reported better disturbance-rejection performance in different working cases compared to other conventional methods.

The convergence of mobile edge computing (MEC) and blockchain is transforming the current computing services in mobile networks, by offering task offloading solutions with security enhancement empowered by blockchain mining. Nevertheless, these important enabling technologies have been studied separately in most existing works. This article proposes a novel cooperative task offloading and block mining (TOBM) scheme for a blockchain-based MEC system where each edge device not only handles data tasks but also deals with block mining for improving the system utility. To address the latency issues caused by the blockchain operation in MEC, we develop a new Proof-of-Reputation consensus mechanism based on a lightweight block verification strategy. A multi-objective function is then formulated to maximize the system utility of the blockchain-based MEC system, by jointly optimizing offloading decision, channel selection, transmit power allocation, and computational resource allocation. We propose a novel distributed deep reinforcement learning-based approach by using a multi-agent deep deterministic policy gradient algorithm. We then develop a game-theoretic solution to model the offloading and mining competition among edge devices as a potential game, and prove the existence of a pure Nash equilibrium. Simulation results demonstrate the significant system utility improvements of our proposed scheme over baseline approaches.

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

In order to perform the operational advantages of manned aerial vehicle (MAV) /unmanned aerial vehicle (UAV) cooperative system, a method of MAV/UAV intelligent decision-making in air combat based on deep deterministic policy gradient (DDPG) algorithm is proposed. Based on the continuous action space, four typical intentions of MAV were introduced, and the optimization models under different intentions were established. The simulation results show that the multiple UAVs can realize the cooperative operations under the guidance of MAV with a favorable effect. In addition, it makes up the gap for the 1VS1 air combat.KeywordsMAV/UAVIntention guidingDecision-makingDeep deterministic policy gradientMultiple UAVs

A data-driven output voltage control of solid oxide fuel cell using multi-agent deep reinforcement learning

• A grid-aware multi-agent reinforcement learning was developed for EV charging. • Public charging data worked with domestic demand to build a real environment. • A dual-head actor optimised the acceptance and cross-site redirection of EV charging. • A centralised attention-based critic and delayed replay buffer ensured stable learning. • The RL policy reduced peak load variability by 40 % and threshold violations by 37 %. As electric vehicles (EV) become more widespread, managing charging demand is critical for grid stability and efficient resource allocation. However, prior work often optimises single sites or treats demand as exogenous, without explicitly modelling grid capacity constraints or the aggregate effects of redirection, and it rarely offers an integrated capacity-aware view that links domestic and public charging under system-level coordination. This study, therefore, develops a multi-agent reinforcement learning framework with cross-site redirection to optimise EV charging operations. The research area focuses on the Perth and Kinross region of Scotland in the UK, using real public charging records from the council database as well as domestic charging statistics from the UK Department for Transport. Charging sites are modelled as cooperative agents using the Multi-Agent Deep Deterministic Policy Gradient algorithm, trained in a spatiotemporal environment that integrates public charging data and simulated domestic demand. We contribute by unifying public and home charging demand in one learning environment, introducing a cross-site actor trained under centralised training and decentralised execution, and incorporating a delayed replay buffer with short-horizon forecasts so redirection aligns with future congestion. The learned policy reduces peak-hour load standard deviation by up to 40 % and lowers cumulative threshold violations by 37 % compared to the baseline. Distinct weekday and weekend strategies emerge, enabling adaptive coordination under varying demand patterns. The study provides interpretable control for EV networks, balancing peak demand and service quality across sites while addressing the system-level coordination.

Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader–follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.

Multi-agent deep reinforcement learning (MADRL) has been used for disaster rescue and other robotic applications. In most MADRL cases, agents are trained only to deal with specific tasks, so once there are unpredictable changes in target tasks, the earlier trained-models might no longer work, and agents have to get trained again from scratch in limited time, which means traditional MADRL is unsuited to unpredictable post-disaster environments. In order to promote the scalability and flexibility of the multi-agent system, meta-learning, which has achieved few shot learning successfully in supervised deep learning field, could reuse the earlier models as the prior knowledge to guide and speed up the new task learning process. In this work, we propose a framework to apply meta learning methods to multi-agent deep deterministic policy gradient (MADDPG), and achieve efficient adaption to new tasks with less time and fewer samples. Some common experiences in the previous tasks learning will be refined as the meta knowledge. Once the tasks get changed with new scenarios, agents will get retrained with good initial network parameters based on this meta knowledge. Although the new scenarios might be different from the previous scenarios, agents could quickly adjust their policy and get better performance within only few episodes. In two experiments, our method performs better learning efficiency than others, getting higher rewards in the first several episodes of new tasks learning.