Reinforcement Learning Based Path Planning Method for Underactuated AUV with Sonar Constraint

This paper focuses on Direct Policy Search (DPS) for cooperative path planning of a single beacon vehicle supporting Autonomous Underwater Vehicles (AUVs) performing surveying missions. Due to the lack of availability of GPS signals underwater, the position errors of the AUVs grow with time even though they are equipped with proprioceptive sensors for dead reckoning. One way to minimize this error is to have a moving beacon vehicle with good positioning data transmit its position acoustically from different locations to other AUVs. When the position is received, the AUVs can fuse this data with the range measured from the travel time of acoustic transmission to better estimate their own positions and keep the error bounded. In this work, we address the beacon vehicle’s path planning problem which takes into account the position errors being accumulated by the supported survey AUVs. We represent the path planning policy as state-action mapping and employ Variable-Length Genetic Algorithm (VLGA) to automatically discover the number of representative states and their corresponding action mapping. We show the resultant planned paths using the learned policy are able to keep the position errors of the survey AUVs bounded over the mission time.

Path planning is essential for autonomous underwater vehicles (AUVs) to perform tasks. Many existing single-objective path planning methods rely on prior knowledge of the underwater environment. However, extracting prior knowledge is challenging because the underwater environment is influenced by ocean current, complex terrain, and other factors. This paper proposes a two-stage reinforcement learning (RL) algorithm based on trajectory exploration and sequence modeling, called the Soft Actor Critic and Online Decision Transformer (SAC-ODT), which operates without prior knowledge. This algorithm utilizes the SAC strategy exploration capability to generate training data for ODT and uses the ODT strategy optimization capability to plan a smooth, energy-efficient, and safe path. In the first stage, a Multi-Reward Strategy Embedding (MRSE) method is designed to facilitate trajectory exploration with multiple strategies, enabling further training of an ODT with comprehensive decision-making capability. During the second stage, a Condition Prioritized Buffer Update and Sampling Strategy (CP-BUSS) is proposed to enhance the sensitivity of ODT to the reward function, enabling adaptation to various tasks while accelerating high-quality path learning. Experimental results demonstrate that compared to existing RL-based benchmarks, SAC-ODT reduces the path time and energy consumption by 2.7% and 2.5%, respectively, while improving path smoothness by 91.96%.

A novel reinforcement learning based tuna swarm optimization algorithm for autonomous underwater vehicle path planning

Path planning and formation control are two of the most significant concepts which can be considered in multi-vehicle systems and particularly in autonomous underwater vehicles (AUVs). The cooperative implementation of complicated commands would lead to desirable results and increase the probability of success in the missions. Due to the nonlinear dynamics and environmental conditions, the cooperative control of AUVs is a challenging topic. The developments in AUV applications demonstrate the significance of research and development in path planning and formation control. Unlike ground or aerial autonomous vehicles, this field of study has not attracted considerable attention and further exploration is required as a result. The present paper reviews the different structures of formation control in AUVs and discusses their advantages and disadvantages. Besides formation control, the cooperative path planning of AUVs along with the limitations specific to the cooperative structure is taken into consideration in the present study. Moreover, avoiding any obstacle collision and preventing any encounter between group members are considered as critical issues in the formation control and cooperative path planning. Some areas are still open to investigation as implied by the technological suggestions, which will facilitate future research. At the end of the article, a simulated sample is given of the triangular formation path planning for AUVs.

We present DDPG-AACO, a hierarchical method for welding manipulator path planning of complex components welding tasks that combines reinforcement learning (RL) with the intelligent optimisation algorithm. The RL agent, trained with the deep deterministic policy gradient (DDPG), learns local path planning policies that control the welding manipulator to safely move between two welding seam endpoints. Next, on a distance matrix constructed by the lengths of local paths between every two welding seam endpoints, the adaptive ant colony optimisation (AACO) algorithm with artificially changed value of pheromones is adopted to realise the global path planning that the welding manipulator traverses all welding seams under a welding direction constraint and has the shortest path length. The simulation results show the effectiveness of the method. The DDPG is better than the deep Q-learning-based methods when performing local path planning. Moreover, the length of the global path with direction constraint can converge to the minimum.

In this study, a set of different search strategies for locating submarine groundwater discharge (SGD) are investigated. This set includes pre-defined path planning (PPP), adapted random walk (RW), particle swarm optimisation (PSO), inertia Levy-flight (ILF), self-organising-migration-algorithm (SOMA), and bumblebee search algorithm (BB). The influences of self-localisation and communication errors and limited travel distance of the autonomous underwater vehicles (AUVs) on the performance of the proposed algorithms are investigated. This study shows that the proposed search strategies could not outperform the classic search heuristic based on full coverage path planning if all AUVs followed the same search strategy. In this study, the influence of self-localisation and communication errors was investigated. The simulations showed that, based on the median error of the search runs, the performance of SOMA was in the same order of magnitude regardless the strength of the localisation error. Furthermore, it was shown that the performance of BB was highly affected by increasing localisation errors. From the simulations, it was revealed that all the algorithms, except for PSO and SOMA, were unaffected by disturbed communications. Here, the best performance was shown by PPP, followed by BB, SOMA, ILF, PSO, and RW. Furthermore, the influence of the limited travel distances of the AUVs on the search performance was evaluated. It was shown that all the algorithms, except for PSO, were affected by the shorter maximum travel distances of the AUVs. The performance of PPP increased with increasing maximum travel distances. However, for maximum travel distances > 1800 m the median error appeared constant. The effect of shorter travel distances on SOMA was smaller than on PPP. For maximum travel distances < 1200 m, SOMA outperformed all other strategies. In addition, it can be observed that only BB showed better performances for shorter travel distances than for longer ones. On the other hand, with different search strategies for each AUV, the search performance of the whole swarm can be improved by incorporating population-based search strategies such as PSO and SOMA within the PPP scheme. The best performance was achieved for the combination of two AUVs following PPP, while the third AUV utilised PSO. The best fitness of this combination was 15.9. This fitness was 26.4% better than the performance of PPP, which was 20.4 on average. In addition, a novel mechanism for dynamically selecting a search strategy for an AUV is proposed. This mechanism is based on fuzzy logic. This dynamic approach is able to perform at least as well as PPP and SOMA for different travel distances of AUVs. However, due to the better adaptation to the current situation, the overall performance, calculated based on the fitness achieved for different maximum travel distances, the proposed dynamic search strategy selection performed 32.8% better than PPP and 34.0% better than SOMA.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

The path planning of the autonomous underwater vehicle (AUV) has shown great potential in various Internet of Underwater Things (IoUT) applications. Although considerable efforts had been made, prior studies are confronted with some limitations. For one thing, existing work only uses the ocean current simulation model without introducing real ocean information, having not been supported by real data. For another, traditional path planning algorithms have strong environment dependence and lack flexibility: once the environment changes, they need to be remodeled and replanned. To overcome these challenges, this article proposes comprehensive ocean information D3QN (COID), an AUV path planning scheme exploiting comprehensive ocean information and reinforcement learning (RL), which consists of three steps. First, we introduce the comprehensive real ocean data, including weather, temperature, thermohaline, current, etc., and apply them into the regional ocean modeling system to generated reliable ocean current. Next, through well-designed state transition function and reward function, we build a 3-D grid model of ocean environment for RL. Furthermore, based on the framework of the double dueling deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (D3QN), COID integrates local ocean current and position features to provide state input and uses priority sampling to accelerate network convergence. The performance of COID has been evaluated and proved by numerical results, which demonstrate efficient path planning and high flexibility for expansion into different ocean environments.

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method

Three-dimensional path planning for autonomous underwater vehicles based on a whale optimization algorithm

Autonomous underwater vehicles (AUVs) are used to execute an increasingly challenging set of missions in commercial, environmental and defense industries. The resources available to the AUV in service of these missions are typically a limited power supply and onboard sensing of its local environment. Optimal path planning is needed to maximize the chances that these AUVs will successfully complete long-endurance missions within their power budget. A time-optimal path planner has been recently developed to minimize AUV mission time required to traverse a dynamic ocean environment at a specified speed through the water. For many missions, time minimization is appropriate because the AUVs operate at a fixed propeller speed. However, the ultimate limiting constraint on AUV operations is often the onboard power supply, rather than mission time. While an empirical or theoretical relationship between mission time and power could be applied to estimate power usage in the path planner, the real power usage and availability on an AUV varies mission-to-mission, as a result of multiple factors, including vehicle buoyancy, battery charge cycle, fin configuration, and water type or quality. In this work, we use data collected from two mid-size AUVs operating in various conditions to learn the mission-to-mission variability in the power budget so that it could be incorporated into the mission planner.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

The autonomous underwater vehicle (AUV) frequently operates in harsh underwater environments, and timely fault diagnosis of the AUV can prevent mission failure and equipment loss. Data‐driven methods based on a single data source have been widely utilized for fault diagnosis of the AUV because they do not require the construction of complex mechanism models and have high fault diagnosis accuracy. However, these methods face challenges in accomplishing complex fault diagnosis tasks because the single data source provides very restricted fault features. To address this issue, an attention mechanism‐based multisensor data fusion neural network (MDFNN) for AUV fault diagnosis is proposed in this work. First, a feature extraction layer based on the two‐dimensional (2D) convolutional method with a 1D kernel is introduced to extract features from each sensor data separately, significantly optimizing the model architecture. Second, an efficient channel attention mechanism‐based feature fusion layer is proposed to reassign weights to the features of each sensor data, enabling the model to focus more on crucial features. Finally, the fused features are input to the fully connected layers and softmax layer to realize the fault diagnosis of multisensor data. In the end, the diagnostic performance of the proposed MDFNN is evaluated utilizing real AUV experimental data. The experiment shows that the proposed MDFNN has a very fast convergence speed and 98.37% fault diagnosis accuracy, demonstrating its excellent fault diagnosis performance. The proposed MDFNN provides a generalized and simply structured fault diagnosis framework for the AUV with multiple types of sensor data, providing significant engineering value.

Due to the large range of dynamic parameters during the Autonomous Underwater Vehicle (AUV) navigation process, the AUV control system exhibits strong non-linear characteristics. Therefore, it has strong requirements on the control performance. For normal sliding mode control, however, the state tracking error does not converge to zero over a finite time. Aiming at dealing with the nonlinear AUV control problem with variable parameters, a Terminal sliding mode variable structure control system with fast convergence speed, high control precision and strong robustness is designed. In this method, firstly, the AUV kinematics and dynamics model are established, and the equations of heading motion and longitudinal motion are described respectively, and the linearization is carried out. Secondly, the sliding mode motion switching function is introduced, and the longitudinal port sliding mode controller is designed respectively. The whole process of the control system from the initial state to the equilibrium state is robust to the disturbancs. Finally, the Lyapunov function is used to verify that the dynamic error has global asymptotic stability. The simulation results show the effectiveness of the control method and confirm the correctness of the theoretical analysis.

Integrated path planning and control through proximal policy optimization for a marine current turbine