Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning

A novel approach to hybrid dynamic environmental-economic dispatch of multi-energy complementary virtual power plant considering renewable energy generation uncertainty and demand response

This paper presents a Self-discipline economic operation method of integrated energy system (IES) considering fault state based on agent. Firstly, various energy supply equipment in the IES are modeled, and the objective function of system economic operation is designed. Then, the Markov decision process is introduced, the economic operation problem of IES is transformed into Markov decision problem, and the agent model of optimal operation is established. Then the agent training based on historical running data is completed by using deep deterministic policy gradient (DDPG) algorithm. Finally, taking the IES of a multi-energy park as an example, the rationality of the output change of each equipment after agent decision under different fault conditions is analyzed.

Anti-rolling devices are widely used in modern shipboard components. In particular, ship anti-rolling control systems are developed to achieve a wide range of ship speeds and efficient anti-rolling capabilities. However, factors that are challenging to solve accurately, such as strong nonlinearities, a complex working environment, and hydrodynamic system parameters, limit the investigation of the rolling motion of ships at sea. Moreover, current anti-rolling control systems still face several challenges, such as poor nonlinear adaptability and manual parameter adjustment. In this regard, this study developed a dynamic model for a ship anti-rolling system. In addition, based on deep reinforcement learning (DRL), an efficient anti-rolling controller was developed using a deep deterministic policy gradient (DDPG) algorithm. Finally, the developed system was applied to a ship anti-rolling device based on the Magnus effect. The advantages of reinforcement learning adaptive control enable controlling an anti-rolling system under various wave angles, ship speeds, and wavelengths. The results revealed that the anti-rolling efficiency of the intelligent ship anti-rolling control method using the DDPG algorithm surpassed 95% and had fast convergence. This study lays the foundation for developing a DRL anti-rolling controller for full-scale ships.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

Optimal dynamic economic dispatch of generation: A review

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

This paper presents a parameter selection method of notch filters for suppressing mechanical resonances in industrial servo systems using the deep deterministic policy gradient (DDPG) algorithm. Several methods for tuning the notch filter parameters were studied such as the fast-Fourier-transform-based methods, extended-Kalman-filter-based methods and adaptive notch filter methods. However, these methods do not find the Q parameters of notch filters which play an important role in determining the system stability, and do not consider the cases in which multiple notch filters are required. Deep-Q-network-based method was developed to solve these problems, but the notch filter parameter tuning is limited to discrete action spaces. This paper develops a new parameter selection method of notch filters, using the DDPG algorithm. DDPG algorithm, which is a model-free and actor-critic algorithm using deep neural networks, is utilized for its capability to operate over continuous action spaces. Experiments are performed using an actual industrial servo system to demonstrate that the developed parameter selection method successfully finds the notch filter parameters to suppress the resonances of the system.

The UAV pursuit-evasion strategy based on Deep Deterministic Policy Gradient (DDPG) algorithm is a current research hotspot. However, this algorithm has the defect of low efficiency in sample exploration. To solve this problem, this paper uses the imitation learning (IL) to improve the DDPG exploration strategy. A kind of quasiproportional guidance control law is designed to generate effective learning samples, which are used as the data of the initial experience pool of DDPG algorithm. The UAV pursuit-evasion strategy based on DDPG and imitation learning (IL-DDPG) is proposed, and the algorithm obtains the data from the experience pool for experience playback learning, which improves the exploration efficiency of the algorithm in the initial stage of training and avoids the problem of too many useless exploration in the training process. The simulation results show that the trained pursuit-UAV can flexibly adjust the flight speed and flight attitude to pursuit the evasion-UAV quickly. It also verifies that the improved DDPG algorithm is more effective than the basic DDPG algorithm to improve the training efficiency.

A novel data-driven energy management strategy for fuel cell hybrid electric bus based on improved twin delayed deep deterministic policy gradient algorithm

The triple exponentially weighted moving average (TriEWMA) controller integrates three distinct EWMAs to mitigate process dynamics and disturbances, enhancing system stability in the semiconductor manufacturing process. However, the performance of TriEWMA is highly dependent on its weight parameters, which are challenging to optimize manually. This work develops a composite strategy utilizing deep reinforcement learning to autonomously optimize these weights. Specifically, the weight adjustment process is modeled as a Markov decision process. Furthermore, the deep deterministic policy gradient (DDPG) algorithm is employed for optimization. The DDPG agent acts as an auxiliary controller to generate the ideal weights of TriEWMA, while the TriEWMA acts as the primary controller to generate appropriate recipes after obtaining the optimal combination of weights. Finally, the proposed algorithm demonstrates superior performance in handling process dynamics and disturbance rejection, as evidence in simulations of a chemical mechanical polishing process.

Intelligent connected vehicle (ICV) has been widely deemed as the key to reduce road accident rate and improve traffic efficiency. However, ensuring high communication reliability and low transmission delay in vehicular networks is challenging, especially in large-scale dynamic networks with diverse heterogeneous data exchange demands. In this paper, we investigate the potential of applying the deep reinforcement learning (DRL) technique to facilitate efficient transmission design in a class of complex multi-user vehicle-to-vehicle (V2V) networks, where conventional mathematical tools confront difficulties in solving the design optimization problems. The considered network contains several pairs of V2V links sharing the channel resource. Each link desires to communicate two types of delay-sensitive messages to support different safety-related applications with the maximum energy efficiency. We propose transforming the power/rate control problem into a Markov decision process and then solving it using the deep deterministic policy gradient (DDPG) algorithm. Simulation results show that in a two-user network our DRL-aided solution can achieve better performance than that with Lyapunov optimization. Extending the former to work in a larger network is straightforward, but it is not the case for the latter. The advantages of applying DRL to support wireless system design are thus demonstrated.

As global energy demand rises and climate change poses an increasing threat, the development of sustainable, low-carbon energy solutions has become imperative. This study focuses on optimizing shared energy storage (SES) and distribution networks (DNs) using deep reinforcement learning (DRL) techniques to enhance operation and decision-making capability. An innovative dynamic carbon intensity calculation method is proposed, which more accurately calculates indirect carbon emissions of the power system through network topology in both spatial and temporal dimensions, thereby refining carbon responsibility allocation on the user side. Additionally, we integrate user-side SES and ladder-type carbon emission pricing into DN to create a low-carbon economic dispatch model. By framing the problem as a Markov decision process (MDP), we employ the DRL, specifically the deep deterministic policy gradient (DDPG) algorithm, enhanced with prioritized experience replay (PER) and orthogonal regularization (OR), to achieve both economic efficiency and environmental sustainability. The simulation results indicate that this method significantly reduces the operating costs and carbon emissions of DN. This study offers an innovative perspective on the synergistic optimization of SES with DN and provides a practical methodology for low-carbon economic dispatch in power systems.

Objective As the core issue of ship navigation safety, ship collision avoidance always depends on the captain's personal status and judgment, which has certain potential safety hazards. In order to coordinate all ships (cruise ships, cargo ships, etc.) in key waters and predict their paths, it is necessary to establish an anti-collision early warning mechanism. Methods Using the deep deterministic policy gradient (DDPG) algorithm and the ship domain model, the electronic chart is used to simulate the ship's navigation path, and the improved strategy of DDPG algorithm based on the key learning of failure region and the improved parameters of ship domain model according to the characteristics of cruise ships are proposed, Improve the accuracy of route prediction and anti-collision. Results Using the improved DDPG algorithm and the improved ship domain model, compared with the previous algorithm, the accuracy of ship collision avoidance is improved from 84.9% to 89.7%, and the average error between the simulated route and the real route is reduced from 25.2 m to 21.4 m. Conclusion Through the ship collision avoidance path planning based on the improved DDPG algorithm and the improved ship domain model, the ship route supervision function in water area can be realized, and when the predicted route intersects with other ships, the dispatcher will be alerted to realize the anti-collision early warning mechanism.

An improved Deep Deterministic Policy Gradient (DDPG) algorithm, integrating self-attention mechanisms, adversarial training, and prioritized experience replay, is proposed to address the limitations of state representation capability, sample efficiency, and algorithm robustness in the DDPG algorithm under complex dynamic environments. First, a multihead self-attention module is constructed within the Critic network to enhance the network’s spatial modeling capability for complex environments by parallel computing multidimensional state-action association features. This improves the accuracy of Q-value estimation and the stability of training. Second, an adversarial training mechanism is introduced, where adversarial samples are proportionally mixed into the training data, enhancing the algorithm’s adaptability to state disturbances. Meanwhile, a prioritized experience replay pool is designed based on the SumTree structure, improving the reuse efficiency of high-value samples. Finally, dynamic and static scenarios are built on the Gazebo simulation platform, and real-world experiments are conducted. The results show that, compared with the original DDPG algorithm, the improved algorithm achieves a 55.3% faster convergence speed in dynamic scenarios, an 80.2% reduction in Critic loss, and stronger generalization ability across different scenarios. Real-world tests further validate the superiority of the algorithm in dynamic obstacle avoidance and trajectory smoothness.