A TD3 Algorithm Based Reinforcement Learning Controller for DC-DC Switching Converters

Autonomous navigation at unsignalized intersections: A coupled reinforcement learning and model predictive control approach

Reinforcement learning (RL) has attracted great interest from researchers in recent years. RL performs as human or better in many fields such as games and robot control. Although this technology is booming in computer science, it has not been practically applied in industrial process control. Up to now, proportional–integral–derivative (PID) control is still the most dominating and popular control method in industrial control. In this paper, we propose a combination of deep reinforcement learning (DRL) and PID control for better process control performance. The idea is generated by the following observations: for PID controller, its transient performance is not usually well enough to meet a strict requirement or in complex signal tracking tasks; For RL technology, a perfectly designed reward function is required for training. However, in practice, the reward function needs to be tested through trial and error, which will lead to a waste of computational power and time. By combining these two strategies, PID controller can help to improve the steady-state performance of RL control by its integral term, while the trained RL agent is able to improve the transient performance of PID controller. Several case studies with the water tank system are presented to demonstrate the effectiveness of the combined PID + RL control strategy.

This paper considers the effectiveness of neural-network-based reinforcement learning (RLNN) controller in discrete time for the underactuated ships. The path following problem of underactuated ships is solved by utilizing RLNN controller together with the Line-of-Sight (LOS) guidance law. The proper form of variables in RLNN controller compatible with the variables in ship maneuvering model is derived and the calculation of desired yaw angle in LOS is implemented in a concise way. The proposed controller is model independent with the consideration of output constraints. Our approach is evaluated by simulations on different ship models with different way-point generated paths.

Micro aerial vehicles are widely being researched and employed due to their relative low operation costs and high flexibility in various applications. We study the under-actuated quadrotor perching problem, designing a trajectory planner and controller which generates feasible trajectories and drives quadrotors to desired state in state space. This paper proposes a trajectory generating and tracking method for quadrotor perching that takes the advantages of reinforcement learning controller and traditional controller. We demonstrate the performance of the trained reinforcement learning controller generated trajectory information and manipulated quadrotor toward the perching point (manually throwing it up in the air with an initial velocity of 1 m/s). We show that this approach permits the control structure of trajectories and controllers enabling such aggressive maneuvers perching on vertical surfaces with relatively accurate. Computation time of evaluating the policy is only 0.03 sec per trajectory, which is two orders of magnitude less than common trajectory optimization algorithms with an approximated model.

This paper presents the analyses of a deep reinforcement learning, DRL approach for maximum power point tracking (MPPT) of a photovoltaic PV boost converter system. The DRL controller used is the Deep Deterministic Policy Gradient (DDPG) algorithm, which is designed to address the limitations of conventional perturb and observe, P&O methods with PID control, such as reduced efficiency under non – uniform irradiance or load variations. The DDPG framework offers an adaptive and computationally efficient alternative by learning the nonlinear mapping between PV states and duty cycle control. Thus, a comparative study of the DDPG with the conventional approach is performed under non – uniform irradiance and varying load resistance. Parameters of power and voltage were evaluated as well as the oscillatory performance for both controllers. The outcome of the results shows that the DDPG controller can achieve higher power extraction with significantly reduced under higher load impedances The conventional approach, however, achieves a better performance under uniform irradiation and low resistance conditions. This is due to the absence of multiple peaks that allows the conventional approach to rapidly converge to the MPP, while the exploratory nature DDPG introduces minor bias due to its exploratory learning process. These findings highlight the potential of DDPG to enhance the efficiency and adaptability of MPPT, particularly relevant in standalone DC microgrid applications.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

Lithium-ion batteries are highly sensitive to temperature, with excessive heat leading to accelerated aging, reduced cycle life, and safety risks such as thermal runaway. Traditional thermal management systems often struggle to maintain optimal battery temperature, especially under high ambient temperatures or dynamic operating conditions. This paper proposes a novel reinforcement learning (RL)-based control method for the coupled thermal management system of a power battery and air conditioning system in a Plug-in Hybrid Electric Vehicle (PHEV). The system integrates the battery cooling circuit with the air conditioning system via a heat exchanger, eliminating the need for a separate radiator. The Deep Deterministic Policy Gradient (DDPG) algorithm is employed to control the water pump speed, optimizing battery temperature in real-time. Experimental results under New European Driving Cycle (NEDC) conditions at 40°C ambient temperature demonstrate that the RL-controlled pump responds faster than the Model Predictive Control (MPC) algorithm, allowing the battery temperature to reach the target temperature of 28°C 1 min earlier with a maximum fluctuation of only 0.8°C. The RL-controlled pump consumes 4 W h more energy than the MPC-controlled pump, while the compressor energy consumption is only 1.68% higher. Additionally, the temperature fluctuations in the passenger cabin under RL control are only 0.1°C higher than under MPC, ensuring passenger comfort. This study highlights the potential of RL-based control for improving the efficiency and stability of integrated thermal management systems in electric vehicles.

Reinforcement learning (RL) can provide a basic framework for autonomous robots to learn to control and maximize future cumulative rewards in complex environments. To achieve high performance, RL controllers must consider the complex external dynamics for movements and task (reward function) and optimize control commands. For example, a robot playing tennis and squash needs to cope with the different dynamics of a tennis or squash racket and such dynamic environmental factors as the wind. In addition, this robot has to tailor its tactics simultaneously under the rules of either game. This double complexity of the external dynamics and reward function sometimes becomes more complex when both the multiple dynamics and multiple reward functions switch implicitly, as in the situation of a real (multi-agent) game of tennis where one player cannot observe the intention of her opponents or her partner. The robot must consider its opponent's and its partner's unobservable behavioral goals (reward function). In this article, we address how an RL agent should be designed to handle such double complexity of dynamics and reward. We have previously proposed modular selection and identification for control (MOSAIC) to cope with nonstationary dynamics where appropriate controllers are selected and learned among many candidates based on the error of its paired dynamics predictor: the forward model. Here we extend this framework for RL and propose MOSAIC-MR architecture. It resembles MOSAIC in spirit and selects and learns an appropriate RL controller based on the RL controller's TD error using the errors of the dynamics (the forward model) and the reward predictors. Furthermore, unlike other MOSAIC variants for RL, RL controllers are not a priori paired with the fixed predictors of dynamics and rewards. The simulation results demonstrate that MOSAIC-MR outperforms other counterparts because of this flexible association ability among RL controllers, forward models, and reward predictors.

Path-tracking control is a crucial process for autonomous vehicles, ensuring that the vehicle drives safely along the reference path, and the suitable controller parameters ensure the accuracy and stability of this process. To enhance the adaptability of traditional path-tracking controller parameters, this paper proposes an adaptive model predictive control (MPC) controller based on a preview-based PID controller and deep deterministic policy gradient (DDPG) algorithm to achieve adaptive tuning of the controller parameters. Starting with the design of the dynamics tracking error model of the vehicle and the MPC controller. Based on the actor-critic reinforcement learning architecture, the DDPG agent is designed to tune the prediction horizon and weight matrix of the MPC controller. A preview-based PID controller is proposed to improve the efficiency and stability of reinforcement learning and compensate for the error in vehicle modeling. The improved algorithm performance is verified through the simulation scenarios of high-speed lane changing and accelerated overtaking scenarios constructed by MATLAB/Simulink. The results show that the improved algorithm significantly improves the adaptive ability of the traditional MPC controller to time-varying conditions and achieves higher tracking accuracy and stability.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

Data center cooling accounts for a significant share of facility energy, yet traditional control strategies often rely on fixed schedules or detailed thermal models that limit adaptability. In this work, we implement a continuous-action deep reinforcement learning (DRL) controller based on the Deep Deterministic Policy Gradient (DDPG) algorithm to adjust supply air, chilled-water, and cooling-tower setpoints in an EnergyPlus simulation of a single-zone data hall. At 15-minute intervals, the agent observes IT load and ambient conditions and learns to minimize non-IT energy consumption (chiller power, fan power, evaporative water) while keeping server-zone outlet temperatures below safety limits. Compared to the default EnergyPlus logic, our DRL controller lowers average Power Usage Effectiveness (PUE) from 1.51 to 1.33 (≈12 %) and reduces peak outlet temperatures from 31.16 °C to 28.33 °C, with tighter performance variability. These results indicate that continuous, model-free DRL can implicitly enforce thermal constraints and exploit cooling subsystem trade-offs without prior calibration. Although evaluated on a simplified lumped-parameter model, this approach shows promise for deployment in heterogeneous or legacy data centers. Future work will extend the framework to multizone configurations, improve sample efficiency via offline pretraining, and incorporate economic factors like water usage and time-of-use electricity pricing.

This paper presents a parameter selection method of notch filters for suppressing mechanical resonances in industrial servo systems using the deep deterministic policy gradient (DDPG) algorithm. Several methods for tuning the notch filter parameters were studied such as the fast-Fourier-transform-based methods, extended-Kalman-filter-based methods and adaptive notch filter methods. However, these methods do not find the Q parameters of notch filters which play an important role in determining the system stability, and do not consider the cases in which multiple notch filters are required. Deep-Q-network-based method was developed to solve these problems, but the notch filter parameter tuning is limited to discrete action spaces. This paper develops a new parameter selection method of notch filters, using the DDPG algorithm. DDPG algorithm, which is a model-free and actor-critic algorithm using deep neural networks, is utilized for its capability to operate over continuous action spaces. Experiments are performed using an actual industrial servo system to demonstrate that the developed parameter selection method successfully finds the notch filter parameters to suppress the resonances of the system.

The UAV pursuit-evasion strategy based on Deep Deterministic Policy Gradient (DDPG) algorithm is a current research hotspot. However, this algorithm has the defect of low efficiency in sample exploration. To solve this problem, this paper uses the imitation learning (IL) to improve the DDPG exploration strategy. A kind of quasiproportional guidance control law is designed to generate effective learning samples, which are used as the data of the initial experience pool of DDPG algorithm. The UAV pursuit-evasion strategy based on DDPG and imitation learning (IL-DDPG) is proposed, and the algorithm obtains the data from the experience pool for experience playback learning, which improves the exploration efficiency of the algorithm in the initial stage of training and avoids the problem of too many useless exploration in the training process. The simulation results show that the trained pursuit-UAV can flexibly adjust the flight speed and flight attitude to pursuit the evasion-UAV quickly. It also verifies that the improved DDPG algorithm is more effective than the basic DDPG algorithm to improve the training efficiency.

Autonomous digital twin framework for gas turbine combined cycle control loops: Comparative study of proportional-integral control, reinforcement learning, and reinforcement learning with agents

This study proposes a novel reinforcement learning (RL) based method for a spacecraft to reorient its antenna towards the Earth to rebuild communication in the situation that unknown attitude failures have occurred. It is assumed that this extreme situation cannot be handled by conventional safe mode. Under such an emergency, an RL controller will learn from interaction with the environment and try to achieve the highest return based on a designed reward function. Preliminary results show that the RL controller can develop a control policy to collect as much rewards as possible, which in turn reorients the antenna towards the Earth steadily or regularly, therefore at least an intermittent communication is possible. The simulation results also reveal that the RL controller can either stabilize the satellite towards the Earth or guide it to a nearly periodic pointing.