Deep reinforcement learning-based adaptive trajectory tracking control for AUVs under ocean disturbances

This work proposes a non-singular fast terminal sliding mode control (NFTSMC) strategy based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and a nonlinear disturbance observer (NDO) to address the issues of modeling errors, motion disturbances, and transmission friction in robotic manipulators. Firstly, a novel modular serial 5-DOF robotic manipulator configuration is designed, and its kinematic and dynamic models are established. Secondly, a nonlinear disturbance observer is employed to estimate the total disturbance of the system and apply feedforward compensation. Based on boundary layer technology, an improved NFTSMC method is proposed to accelerate the convergence of tracking errors, reduce chattering, and avoid singularity issues inherent in traditional terminal sliding mode control. The stability of the designed control system is proved using Lyapunov stability theory. Subsequently, a deep reinforcement learning (DRL) agent based on the TD3 algorithm is trained to adaptively adjust the control gains of the non-singular fast terminal sliding mode controller. The dynamic information of the robotic manipulator is used as the input to the TD3 agent, which searches for optimal controller parameters within a continuous action space. A composite reward function is designed to ensure the stable and efficient learning of the TD3 agent. Finally, the motion characteristics of three joints for the designed 5-DOF robotic manipulator are analyzed. The results show that compared to the non-singular fast terminal sliding mode control algorithm based on a nonlinear disturbance observer (NDONFT), the non-singular fast terminal sliding mode control algorithm integrating a nonlinear disturbance observer and the Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3NDONFT) reduces the mean absolute error of position tracking for the three joints by 7.14%, 19.94%, and 6.14%, respectively, and reduces the mean absolute error of velocity tracking by 1.78%, 9.10%, and 2.11%, respectively. These results verify the effectiveness of the proposed algorithm in enhancing the trajectory tracking accuracy of the robotic manipulator under unknown time-varying disturbances and demonstrate its strong robustness against sudden disturbances.

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method

Vision-based motion planning of autonomous underwater vehicles (AUVs) is regarded as a critical requirement for marine intelligent transportation systems. However, the limited vision range and the uncertain model parameters of an AUV make it difficult to fulfill this requirement. This study focuses on a binocular-vision-based motion planning issue for an AUV. First, we develop an intelligent AUV system that mainly comprises binocular cameras for patrolling the target, a localization unit for acquiring the position information, and an acoustic modem for communicating with buoys. Accordingly, the parallax angles from the AUV to target are used to construct an optimal motion planning problem. To solve the aforementioned problem, we develop a deep reinforcement learning method called the improved twin delayed deep deterministic (TD3) policy gradient algorithm in order to minimize the reward function, such that the AUV can perpendicularly patrol the target with a fixed distance. The advantages of our solution are as follows: 1) the binocular-vision-based motion planning method can achieve a trade-off between motion stability and observation effectiveness; 2) the improved TD3 algorithm can accelerate the convergence compared to other algorithms, while it can simultaneously overcome the dependency on the model parameters of the AUV. Finally, simulation and experimental studies are conducted to verify the effectiveness.

A novel data-driven energy management strategy for fuel cell hybrid electric bus based on improved twin delayed deep deterministic policy gradient algorithm

Anti-rolling devices are widely used in modern shipboard components. In particular, ship anti-rolling control systems are developed to achieve a wide range of ship speeds and efficient anti-rolling capabilities. However, factors that are challenging to solve accurately, such as strong nonlinearities, a complex working environment, and hydrodynamic system parameters, limit the investigation of the rolling motion of ships at sea. Moreover, current anti-rolling control systems still face several challenges, such as poor nonlinear adaptability and manual parameter adjustment. In this regard, this study developed a dynamic model for a ship anti-rolling system. In addition, based on deep reinforcement learning (DRL), an efficient anti-rolling controller was developed using a deep deterministic policy gradient (DDPG) algorithm. Finally, the developed system was applied to a ship anti-rolling device based on the Magnus effect. The advantages of reinforcement learning adaptive control enable controlling an anti-rolling system under various wave angles, ship speeds, and wavelengths. The results revealed that the anti-rolling efficiency of the intelligent ship anti-rolling control method using the DDPG algorithm surpassed 95% and had fast convergence. This study lays the foundation for developing a DRL anti-rolling controller for full-scale ships.

This paper focuses on maintaining the liquid level in tank two in the coupled tank control system with the help of the sliding mode control (SMC). SMC is the potent controller for coupled tank liquid level system under tracking conditions. Sliding mode controller (SMC) is nonlinear control it applies a discontinuous control signal. The main advantages of sliding mode are low sensitivity to plant parameter variations and disturbances. To maintain desired liquid levels in a nonlinear coupled tank is a great challenging task so this problem better handled by SMC. To control this liquid level process a SMC strategy is used we have first order SMC and second order SMC. Regulating this system in less time with proper control signal is not possible in first order SMC due to chattering is obtained which yields unwanted control signal. To fill tank-2 in less time with smooth control signal is possible in higher order SMC. Next, second order SMC is applied and stability is reached fast and no chattering phenomenon. These can be shown in the simulation results with the help of MATLAB. This application report describes how to develop SMC for liquid level control in coupled tank system.

This paper deals with the control of underactuated autonomous underwater vehicles (AUVs). AUVs are needed in many applications such as the exploration of oceans, scientific and military missions, etc. There are many challenges in the control of AUVs due to the complexity of the AUV model, the unmodelled dynamics, the uncertainties and the environmental disturbances. A trajectory tracking control scheme is proposed in this paper; this control scheme is designed using the sliding mode control technique in order to be robust against bounded disturbances. The control performance of an example AUV, using the proposed method, is evaluated through computer simulations. These simulation studies, which consider different reference trajectories, show that the proposed control scheme is robust under bounded disturbances.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

This paper presents the design and implementation of a novel sliding mode control integrated with active disturbance rejection (SMCDR) for precise robust trajectory tracking of piezoelectric nanopositioning stages. The model uncertainties, nonlinearity, and external disturbances of the piezoelectric nanopositioning stage are regarded as a lumped disturbance, which is estimated by an extended state observer. The active disturbance rejection control (ADRC) strategy is used to realize a preliminary trajectory tracking, while the sliding mode control is adopted to handle the estimation error and residual uncertainties, and to improve the tracking performance. The exponential stability of the proposed SMCDR is proved using Lyapunov’s direct method. The proposed SMCDR controller combines the strength of both ADRC and sliding mode control, exhibits a simple structure, requires only the position measurements, and does not require any information of model parameters except for an approximate constant gain. Experimental results reveal the robustness of the SMCDR approach in suppressing the hysteresis of piezoelectric actuator, and illustrate the superior performance over conventional ADRC, integral sliding mode controller (ISMC) and PID controller for trajectory tracking control of piezoelectric nanopositioning stages.

There are many types of controllers had been used to control Autonomous Underwater Vehicle (AUV) such as Proportional Integral Derivative (PID), Linear Quadratic Regulator (LQR), state feedback linearization, integrator back-stepping, and Sliding-Mode Control (SMC). However, for PID and SMC in particular, it is difficult to determine the optimal control design parameters. The objective of this study is to design and develop a responsive motion control system with optimal parameters for an AUV. The contribution of this paper is in term of introducing a filter to smooth reference signal and proposing a brute forcing technique to find optimal controller parameters. The methodology starts with modeling the AUV, estimating the unknown parameters from a real AUV model, designing a control system based on PI and SMC methods, and finally optimizing the controller parameters. The controller design was onto controlling surge speed using PI, heading using SMC, and diving using SMC. Simulation-wise, the developed control system has an average value of 93.89 % of responsiveness to track desired trajectory while 82.33 % of responsiveness without using the smoothing filter. The tested input signals were unit step, ramp, parabolic, and sinusoidal.

Sliding mode heading control for AUV based on continuous hybrid model-free and model-based reinforcement learning

This paper presents the design of two predefined-time active fault-tolerant controllers for the trajectory tracking of autonomous underwater vehicles (AUVs) which can address actuator faults without causing actuator saturation. The first controller offers improved steady-state trajectory tracking precision, while the second ensures a nonsingular property. Firstly, a predefined-time sliding mode controller is formulated based on a predefined-time disturbance observer by integrating a novel predefined-time auxiliary system to prevent the control input from exceeding the actuator’s physical limitations. Subsequently, a non-singular backstepping controller is introduced to circumvent potential singularities in the sliding mode controller, guaranteeing that the trajectory tracking error is uniformly ultimately bounded (UUB) within the predefined time. Additionally, theoretical analysis and simulation results are presented to illustrate the advantages of the proposed method.

In the underwater environment, robust control algorithms are required to control autonomous underwater vehicles (AUVs) at high speed while preventing large nonlinearities and disturbances. Sliding mode control (SMC) is a well-known robust control theory and has been widely used not only in AUV control but also in systems such as industrial robots which have high nonlinearity in their system dynamics. However, SMC has the disadvantage of causing chattering on the control input, and it is difficult to apply this method to the control fins of an AUV system that cannot move its fins at high speed underwater. In this work, a design for a sliding mode control with sliding perturbation observer (SMCSPO) algorithm is applied to AUVs, and the simulation results under underwater disturbance conditions are discussed. From simulation using MATLAB, it is confirmed that AUV control using SMCSPO shows better trajectory tracking control performance without chattering than PID control.

The chattering phenomenon associated with the use of Sliding Mode Controllers (SMC) is considered a major drawback of the robust controllers. Ideally, SMCs are to exhibit infinite switching frequency but this is always compromised by the existence of parasitic dynamics which add to the input relative degree of the system and leads to high but finite switching frequency. The performance and robustness of the controller can be closely linked to the resulting switching frequency. Often, when controlling the speed of induction motors, SMCs are applied and used to generate what is assumed to be smooth control signals. These control signals are used as modulation inputs for various carrier based modulation schemes. In doing so, the sliding action of the controller is disrupted as the SMC's switching frequency is limited by the finite carrier frequency of the modulation process. This effect is carefully analysed in this paper. An alternative is considered in this paper where SMCs are applied to the inner most control loop without the use of any carrier based modulation schemes.

Adaptive sliding mode control with pre-specified performance settings for AUV’s trajectory tracking