Deep reinforcement learning with external control

The design and implementation of a deep reinforcement learning and quantum finance theory-inspired portfolio investment management system

A novel data-driven energy management strategy for fuel cell hybrid electric bus based on improved twin delayed deep deterministic policy gradient algorithm

Anti-rolling devices are widely used in modern shipboard components. In particular, ship anti-rolling control systems are developed to achieve a wide range of ship speeds and efficient anti-rolling capabilities. However, factors that are challenging to solve accurately, such as strong nonlinearities, a complex working environment, and hydrodynamic system parameters, limit the investigation of the rolling motion of ships at sea. Moreover, current anti-rolling control systems still face several challenges, such as poor nonlinear adaptability and manual parameter adjustment. In this regard, this study developed a dynamic model for a ship anti-rolling system. In addition, based on deep reinforcement learning (DRL), an efficient anti-rolling controller was developed using a deep deterministic policy gradient (DDPG) algorithm. Finally, the developed system was applied to a ship anti-rolling device based on the Magnus effect. The advantages of reinforcement learning adaptive control enable controlling an anti-rolling system under various wave angles, ship speeds, and wavelengths. The results revealed that the anti-rolling efficiency of the intelligent ship anti-rolling control method using the DDPG algorithm surpassed 95% and had fast convergence. This study lays the foundation for developing a DRL anti-rolling controller for full-scale ships.

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

A hybrid energy storage system (HESS) is adopted to tackle the traction network voltage fluctuation problem caused by high power and large energy demand during the starting and braking of urban rail trains. The system is composed of on-board ultracapacitors and ground lithium batteries, aiming to smooth out the power fluctuation to realize “peak-shaving and valley-filling”. Based on deep reinforcement learning (DRL) online sequence decision, a dynamic power allocation strategy is proposed to improve the energy-saving and voltage stabilization of DC traction networks as well as HESS life protection. Furthermore, to enhance the DRL’s efficiency under time-varying operating conditions, an annealing bias - priority experience replay twin delayed deep deterministic policy gradient algorithm (A-TD3) is proposed to train the replay buffer in DRL. The online learning and optimization strategy is implemented via the mechanism of “trial and error” and “feedback” of the agent. RT-LAB semi-physical real-time simulation systems are adopted to verify the effectiveness of the proposed strategy. Compared with the traditional filtering algorithms and DRL algorithms, the results show that the proposed method converges faster and is more energy saving and stable while effectively protecting the energy storage components.

Deep reinforcement learning optimized double exponentially weighted moving average controller for chemical mechanical polishing processes

Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm

We report on an aberration correction algorithm for a wavefront sensorless adaptive optics (WFSless AO) system based on deep reinforcement learning. First, it is verified that the reinforcement learning theory can be applied in our system. In addition, the deep deterministic policy gradient algorithm is introduced to build the control structure. After that, deep learning is used to deal with the messy raw images of far-field intensity distribution. We emphatically present how to design a feature extraction with the convolutional neural network in the control structure. To demonstrate the performance of this structure, some comparisons are made with the stochastic parallel gradient descent algorithm and the WFSless AO based on general modes algorithm. The results indicate that the correction speed of our method improves about 9 times and 2.5 times, respectively, for the similar correction effect.

In order to solve the problems of image perception and quality decision-making of wood defects with typical bionic intelligent algorithms, the existence of multidimensional degradation factors causes serious real problems of image distortion; the author proposes a wood defect image reconstruction and quality evaluation model based on deep reinforcement learning. The model introduced the deep learning mechanism and realized real-time and efficient reconstruction of multidimensional defect images of different wood by using the deep residual network for iterative training. In this model, a panoramic autonomous perception model was constructed for the fine segmentation and feature extraction of multidimensional defects of different wood and a shared resource pool of wood defect features of the magnitude of big data was constructed. Introduce the reinforcement learning mechanism, use the deep deterministic policy gradient algorithm, and establish a high-dimensional decision mapping between the iterative update of defect features, autonomous decision-making, panoramic visualization, depth prediction, and wood quality evaluation; it realizes the horizontal sharing integration of multidimensional difference wood defect image reconstruction and quality evaluation. The results obtained are as follows: in a typical environment, systematic wood quality evaluation, and autonomous intelligent decision-making performance, the coincidence rate with artificial defect recognition and evaluation efficiency can reach 90% and the loss of the training set can be controlled below 0.2%. Compared with the traditional wood quality grading system, the wood defect image reconstruction, and quality evaluation model system designed by the author, the quality evaluation decision-making efficiency rate was 90.19%, an increase of about 20%, and the system decision-making operation and maintenance loss was 2.23%, a decrease of about 10%. It is proved that the system designed by the author can realize the timely detection of wood quality defects very effectively and save a lot of manpower and material resources.

This paper proposed a deep reinforcement learning based reference speed planning strategy to co-optimize the fuel economy, driving safety, and travel efficiency of connected and automated hybrid electric vehicles in urban scenarios. Thus, in this work, a connected traffic environment is first developed based on Simulation Urban Mobility (SUMO) to simulate real-world urban scenarios. Then a deep reinforcement learning agent is designed based on a twin delayed deep deterministic policy gradient algorithm (TD3). It can quickly solve the reference speed according to the ego vehicle state and traffic information. In addition, by developing a rule-based safety module and integrating it into the reward function, the TD3 agent can be facilitated to learn safety car-following policies and adhere to traffic light rules. Finally, simulation results indicate that the proposed strategy can efficiently and safely control the ego vehicles in complex urban scenarios with signalized intersections. And the fuel consumption of the proposed strategy is reduced by about 15% compared with that of the intelligent driver model based speed planning strategy.

Attempts to address the production scheduling problem thus far rely on simplifying assumptions, such as static environment and inflexible size of the problem, which compromises the schedule performance in practice due to many unpredictable disruptions to the system. Thus, the study of scheduling in the presence of real-time events, termed dynamic scheduling, continues to attract attention given the agility, flexibility, and timeliness modern production systems must deliver. Additionally, the changing nature of the manufacturing system also raises new challenges to existing scheduling strategies. At the front-end, the development of advanced data creation and exchange frameworks such as the Internet of things and cyber-physical system and their applications to the industrial environment have created an abundance of industrial data, while at the backend, edge and cloud computing technologies greatly enhance the capacity to process that data. Industrial data must be mined and analyzed so that the investment in infrastructure is not wasted, and the production system managed more effectively and in real-time. Many data-driven technologies have been adopted in scheduling research, a promising candidate among them being reinforcement learning (RL) which is able to build a direct mapping from observation of environment to actions that improve its performance. In this thesis, a deep multi-agent reinforcement learning (deep MARL) architecture is proposed to solve the dynamic scheduling problem (DSP). The deep reinforcement learning (DRL) algorithm is used to train the decentralized scheduling agents, to capture the relationship between information on the factory floor and scheduling objectives, with the aim of making real-time decisions for a manufacturing system with frequent unexpected events. Two major aspects of deep MARL application to DSP are addressed in this work, namely the conversion from traditional static scheduling problem (SSP) to dynamic scheduling in a practical context, and the adaptation of existing deep MARL algorithms to solve the scheduling problem in such an environment. Some impractical constraints of traditional studies are removed to create a research context that is closer to actual practice, result in a scheduling problem of variable size and scope. Specialized state and action representations that can handle the ever-changing specification of problem are developed; the criteria of feature selection in dynamic environment are also discussed. Recent progressions in DRL and MARL research are integrated into the proposed approach after selection and adaptation. In addition, various improvements to common deep MARL architecture are proposed, including the lightweight multilayer perceptron (MLP) encoder that is efficient in handling unstructured industrial data, a training scheme under the multi-agent architecture to improve the stability of training and overall performance, and knowledge-based reward-shaping techniques to decompose the joint reward signal into individual utilities to speed up the learning and encourage cooperative behavior between agents. Simulation studies are then conducted for the ablation study and validation. In the first stage, the performance of the proposed approach, either as individual components or as an integrated model, are tested in iterative simulation runs within which a unique instance of production is created. Meanwhile, a set of DRL-based approaches from recent publications are run in parallel. Results suggest that the contribution of each improvement is significant; the integrated architecture also delivers stronger performance than peer DRL-based approaches. For the validation, a set of priority rules that have strong performance in specified context and are widely applied in actual production scheduling are used as the benchmark. Proposed approach also provides performance gain compared to the strongest rule, with a minor increase in computation cost and negligible latency in decision-making.

Deep Reinforcement Learning (DRL) is a powerful approach for generating control strategies for a variety of complex systems, representing an emerging paradigm in control applications. An important feature of Deep RL is that it does not explicitly model the process, but instead it relies on optimization-driven techniques to devise effective control policies. Despite its remarkable success in simulated environments, RL holds great potential in real-world applications. This article explores the complex challenges involved in implementing Deep Reinforcement Learning (DRL) algorithms on a cable-driven parallel robot. A key contribution of this work as specific advancement is the integration of a Proportional-Integral-Derivative (PID) controller within the RL framework, establishing a unique approach to CDPR control that leverages adaptive learning capabilities. A Reinforcement Learning (RL) agent for reference tracking is trained using the novel application of the adaptive-featured Twin Delayed Deep Deterministic (TD3) policy gradient algorithm, tailored to enhance CDPR adaptability and precision in dynamic environments. The first step is to test the performance of the trained agent on point-to-point robotic application tasks. As a result of such tasks, it is possible to evaluate the level of adaptability and performance of the RL agent. Multiple experiments are conducted to assess the versatility of the RL agent involving linear and circular scenarios. This research significantly advances the field by demonstrating the applicability of RL for complex robotic structures like CDPRs, showcasing promising results that underline the robustness and adaptability of the proposed approach. As a result of the TD3 adaptive learning process, the trained agent is able to perform the designated action in order to determine which policy stands out as the most rewarding.

In this study, we apply deep reinforcement learning (DRL) to control a robot manipulator and investigate its effectiveness by comparing the performance of several DRL algorithms, namely, deep deterministic policy gradient (DDPG) and distributed distributional deterministic policy gradient (D4PG) algorithms. We conducted computational training and testing experiments on a control model for a reaching task of the robot manipulator. Experimental results show that the D4PG algorithm achieves a higher learning success rate than the DDPG algorithm and demonstrate the potential application of DRL for controlling robot manipulators.

Deep reinforcement learning (DRL) has been one of the research hotspots in the areas of control. In this paper, we focus on the study of missile attitude control system using DRL. An novel PID controller based on deep deterministic policy gradient(DDPG) algorithm is presented, which could applied to the self-tuning of parameters. The framework of the adaptive DDPG-PID controller is given. The controller takes flight information as input and takes rudder angle as output. A reward function related to the system error is designed, which can be used to train the DDPG algorithm effectively. Simulation results show that the adaptive DDPG-PID controller has a faster convergence velocity, reduces the overshoot and oscillation, achieves higher accuracy tracking control to target.