Real-time optimal scheduling for active distribution networks: A graph reinforcement learning method

The quantity of wind and photovoltaic power-based distributed generators (DGs) is continually rising within the distribution network, presenting obstacles to its safe, steady, and cost-effective functioning. Active distribution network dynamic reconfiguration (ADNDR) improves the consumption rate of renewable energy, reduces line losses, and optimizes voltage quality by optimizing the distribution network structure. Despite being formulated as a highly dimensional and combinatorial nonconvex stochastic programming task, conventional model-based solvers often suffer from computational inefficiency and approximation errors, whereas population-based search methods frequently exhibit premature convergence to suboptimal solutions. Moreover, when dealing with high-dimensional ADNDR problems, these algorithms often face modeling difficulties due to their large scale. Deep reinforcement learning algorithms can effectively solve the problems above. Therefore, by combining the graph attention network (GAT) with the deep deterministic policy gradient (DDPG) algorithm, a method based on the graph attention network deep deterministic policy gradient (GATDDPG) algorithm is proposed to online solve the ADNDR problem with the uncertain outputs of DGs and loads. Firstly, considering the uncertainty in distributed power generation outputs and loads, a nonlinear stochastic optimization mathematical model for ADNDR is constructed. Secondly, to mitigate the dimensionality of the decision space in ADNDR, a cyclic topology encoding mechanism is implemented, which leverages graph-theoretic principles to reformulate the grid infrastructure as an adaptive structural mapping characterized by time-varying node–edge interactions Furthermore, the GATDDPG method proposed in this paper is used to solve the ADNDR problem. The GAT is employed to extract characteristics pertaining to the distribution network state, while the DDPG serves the purpose of enhancing the process of reconfiguration decision-making. This collaboration aims to ensure the safe, stable, and cost-effective operation of the distribution network. Finally, we verified the effectiveness of our method using an enhanced IEEE 33-bus power system model. The outcomes of the simulations demonstrate its capacity to significantly enhance the economic performance and stability of the distribution network, thereby affirming the proposed method’s effectiveness in this study.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

Anti-rolling devices are widely used in modern shipboard components. In particular, ship anti-rolling control systems are developed to achieve a wide range of ship speeds and efficient anti-rolling capabilities. However, factors that are challenging to solve accurately, such as strong nonlinearities, a complex working environment, and hydrodynamic system parameters, limit the investigation of the rolling motion of ships at sea. Moreover, current anti-rolling control systems still face several challenges, such as poor nonlinear adaptability and manual parameter adjustment. In this regard, this study developed a dynamic model for a ship anti-rolling system. In addition, based on deep reinforcement learning (DRL), an efficient anti-rolling controller was developed using a deep deterministic policy gradient (DDPG) algorithm. Finally, the developed system was applied to a ship anti-rolling device based on the Magnus effect. The advantages of reinforcement learning adaptive control enable controlling an anti-rolling system under various wave angles, ship speeds, and wavelengths. The results revealed that the anti-rolling efficiency of the intelligent ship anti-rolling control method using the DDPG algorithm surpassed 95% and had fast convergence. This study lays the foundation for developing a DRL anti-rolling controller for full-scale ships.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

This paper presents a parameter selection method of notch filters for suppressing mechanical resonances in industrial servo systems using the deep deterministic policy gradient (DDPG) algorithm. Several methods for tuning the notch filter parameters were studied such as the fast-Fourier-transform-based methods, extended-Kalman-filter-based methods and adaptive notch filter methods. However, these methods do not find the Q parameters of notch filters which play an important role in determining the system stability, and do not consider the cases in which multiple notch filters are required. Deep-Q-network-based method was developed to solve these problems, but the notch filter parameter tuning is limited to discrete action spaces. This paper develops a new parameter selection method of notch filters, using the DDPG algorithm. DDPG algorithm, which is a model-free and actor-critic algorithm using deep neural networks, is utilized for its capability to operate over continuous action spaces. Experiments are performed using an actual industrial servo system to demonstrate that the developed parameter selection method successfully finds the notch filter parameters to suppress the resonances of the system.

The UAV pursuit-evasion strategy based on Deep Deterministic Policy Gradient (DDPG) algorithm is a current research hotspot. However, this algorithm has the defect of low efficiency in sample exploration. To solve this problem, this paper uses the imitation learning (IL) to improve the DDPG exploration strategy. A kind of quasiproportional guidance control law is designed to generate effective learning samples, which are used as the data of the initial experience pool of DDPG algorithm. The UAV pursuit-evasion strategy based on DDPG and imitation learning (IL-DDPG) is proposed, and the algorithm obtains the data from the experience pool for experience playback learning, which improves the exploration efficiency of the algorithm in the initial stage of training and avoids the problem of too many useless exploration in the training process. The simulation results show that the trained pursuit-UAV can flexibly adjust the flight speed and flight attitude to pursuit the evasion-UAV quickly. It also verifies that the improved DDPG algorithm is more effective than the basic DDPG algorithm to improve the training efficiency.

A new ensemble deep graph reinforcement learning network for spatio-temporal traffic volume forecasting in a freeway network

A novel data-driven energy management strategy for fuel cell hybrid electric bus based on improved twin delayed deep deterministic policy gradient algorithm

Objective As the core issue of ship navigation safety, ship collision avoidance always depends on the captain's personal status and judgment, which has certain potential safety hazards. In order to coordinate all ships (cruise ships, cargo ships, etc.) in key waters and predict their paths, it is necessary to establish an anti-collision early warning mechanism. Methods Using the deep deterministic policy gradient (DDPG) algorithm and the ship domain model, the electronic chart is used to simulate the ship's navigation path, and the improved strategy of DDPG algorithm based on the key learning of failure region and the improved parameters of ship domain model according to the characteristics of cruise ships are proposed, Improve the accuracy of route prediction and anti-collision. Results Using the improved DDPG algorithm and the improved ship domain model, compared with the previous algorithm, the accuracy of ship collision avoidance is improved from 84.9% to 89.7%, and the average error between the simulated route and the real route is reduced from 25.2 m to 21.4 m. Conclusion Through the ship collision avoidance path planning based on the improved DDPG algorithm and the improved ship domain model, the ship route supervision function in water area can be realized, and when the predicted route intersects with other ships, the dispatcher will be alerted to realize the anti-collision early warning mechanism.

An improved Deep Deterministic Policy Gradient (DDPG) algorithm, integrating self-attention mechanisms, adversarial training, and prioritized experience replay, is proposed to address the limitations of state representation capability, sample efficiency, and algorithm robustness in the DDPG algorithm under complex dynamic environments. First, a multihead self-attention module is constructed within the Critic network to enhance the network’s spatial modeling capability for complex environments by parallel computing multidimensional state-action association features. This improves the accuracy of Q-value estimation and the stability of training. Second, an adversarial training mechanism is introduced, where adversarial samples are proportionally mixed into the training data, enhancing the algorithm’s adaptability to state disturbances. Meanwhile, a prioritized experience replay pool is designed based on the SumTree structure, improving the reuse efficiency of high-value samples. Finally, dynamic and static scenarios are built on the Gazebo simulation platform, and real-world experiments are conducted. The results show that, compared with the original DDPG algorithm, the improved algorithm achieves a 55.3% faster convergence speed in dynamic scenarios, an 80.2% reduction in Critic loss, and stronger generalization ability across different scenarios. Real-world tests further validate the superiority of the algorithm in dynamic obstacle avoidance and trajectory smoothness.

In the evolving landscape of financial technology, Deep Reinforcement Learning (DRL) has emerged as a potent tool for optimizing financial resource allocation, responding to the dynamic and complex nature of modern financial markets. This research investigates the application of the Deep Deterministic Policy Gradient (DDPG) algorithm, comparing its performance with traditional Q-learning and Random Selection methods in financial resource allocation. Our findings reveal that DDPG significantly outperforms the comparative methods, achieving a higher average return rate of 13.2%, showcasing superior risk management with a maximum drawdown of -5%, and demonstrating rapid convergence within 100 epochs. Furthermore, DDPG requires fewer parameter adjustments and maintains high stability and strategy diversity, leading to a lower portfolio risk. These results underline the effectiveness of DDPG in navigating the intricate financial market, providing a valuable framework for enhancing profitability and reducing investment risks. The study contributes to the financial technology field, advocating for the integration of DRL strategies in financial decision-making processes.

The Dual Active Bridge converter (DABC), known for its bidirectional power transfer capability and high efficiency, plays a crucial role in various applications, particularly in electric vehicles (EVs), where it facilitates energy storage, battery charging, and grid integration. The Dual Active Bridge Converter (DABC), when paired with a high-performance CLLC filter, is well-regarded for its ability to transfer power bidirectionally with high efficiency, making it valuable across a range of energy applications. While these features make the DABC highly efficient, they also complicate controller design due to nonlinear behavior, fast switching, and sensitivity to component variations. We have used a Fractional-order PID (FOPID) controller to benefit from the simple structure of classical PID controllers with lower complexity and improved flexibility because of additional filtering gains adopted in this method. However, for a FOPID controller to operate effectively under real-time conditions, its parameters must adapt continuously to changes in the system. To achieve this adaptability, a Multi-Agent Reinforcement Learning (MARL) approach is adopted, where each gain of the controller is tuned individually using the Deep Deterministic Policy Gradient (DDPG) algorithm. This structure enhances the controller’s ability to respond to external disturbances with greater robustness and adaptability. Meanwhile, finding the best initial gains in the RL structure can decrease the overall efficiency and tracking performance of the controller. To overcome this issue, Grey Wolf Optimization (GWO) algorithm is proposed to identify the most suitable initial gains for each agent, providing faster adaptation and consistent performance during the training process. The complete approach is tested using a Hardware-in-the-Loop (HIL) platform, where results confirm accurate voltage control and resilient dynamic behavior under practical conditions. In addition, the controller’s performance was validated under a battery management scenario where the DAB converter interacts with a nonlinear lithium-ion battery. The controller successfully regulated the State of Charge (SOC) through automated charging and discharging transitions, demonstrating its real-time adaptability for BMS-integrated EV systems. Consequently, the proposed MARL-FOPID controller reported better disturbance-rejection performance in different working cases compared to other conventional methods.

Deep reinforcement learning (DRL) has been one of the research hotspots in the areas of control. In this paper, we focus on the study of missile attitude control system using DRL. An novel PID controller based on deep deterministic policy gradient(DDPG) algorithm is presented, which could applied to the self-tuning of parameters. The framework of the adaptive DDPG-PID controller is given. The controller takes flight information as input and takes rudder angle as output. A reward function related to the system error is designed, which can be used to train the DDPG algorithm effectively. Simulation results show that the adaptive DDPG-PID controller has a faster convergence velocity, reduces the overshoot and oscillation, achieves higher accuracy tracking control to target.

Physical-assisted multi-agent graph reinforcement learning enabled fast voltage regulation for PV-rich active distribution network