A Two-Stage Reinforcement Learning Algorithm for AUV Path Planning Based on Trajectory Exploration and Sequence Modeling

Path planning for Autonomous Underwater Vehicle(AUV) is challenging for the reason that the dynamic system of AUV is under-actuated and the main sensor called sonar suffers from a limit detection range. This paper proposed a reinforcement learning (RL) based path planning method for the high speed AUV considering multi constrains. Firstly, an actor-critic based RL algorithm is designed including the constrains of sonar's detection distance, detect angle and AUV's angular rate. Secondly, an improved reward function is proposed considering total path length, angle change and the position of obstacles and target area. Finally, a priority experience replay(PER) technology is applied to RL algorithm to speed the training process. Simulation results show that the improved algorithm plan the reasonable path with mentioned constrains. Compared with conventional method in simulation, the improved algorithm finishes the path planning task with faster convergence speed, shorter path length and smaller cost about angle action.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

A novel reinforcement learning based tuna swarm optimization algorithm for autonomous underwater vehicle path planning

This paper focuses on Direct Policy Search (DPS) for cooperative path planning of a single beacon vehicle supporting Autonomous Underwater Vehicles (AUVs) performing surveying missions. Due to the lack of availability of GPS signals underwater, the position errors of the AUVs grow with time even though they are equipped with proprioceptive sensors for dead reckoning. One way to minimize this error is to have a moving beacon vehicle with good positioning data transmit its position acoustically from different locations to other AUVs. When the position is received, the AUVs can fuse this data with the range measured from the travel time of acoustic transmission to better estimate their own positions and keep the error bounded. In this work, we address the beacon vehicle’s path planning problem which takes into account the position errors being accumulated by the supported survey AUVs. We represent the path planning policy as state-action mapping and employ Variable-Length Genetic Algorithm (VLGA) to automatically discover the number of representative states and their corresponding action mapping. We show the resultant planned paths using the learned policy are able to keep the position errors of the survey AUVs bounded over the mission time.

The path planning problem for autonomous underwater vehicle (AUV) is to generate real-time collision-free trajectories from beginning to end in an underwater environment. Although in the past few decades the study of AUV path planning has obtained many achievements, but many methods only consider the optimal path of AUV, while ignoring the can't get to the obstacles in the practical underwater environment. Another problem is that the generated path may not meet the limitation that the length of the path should be minimized when the method is applied in point-to-point path planning. In order to solve these problems, a potential field bio-inspired neural network (PBNN) method is proposed. The bio-inspired neural network approach can be used to plan real-time collision free paths. The potential field introduced by the proposed approach can plan a real-time comfortable trajectory by overcoming the too close shortcomings. The proposed approach of introducing attractive potential field in point to point path planning can optimize the path. Simulation studies verify that the proposed method can safely and effectively navigate of an AUV.

Most existing path planning algorithms focus on generating piecewise linear functions, while the smoothness of the path is usually ignored. Route stability and energy saving are key issues for an autonomous underwater vehicle (AUV) working in the underwater environment. To ensure that AUV can pass through the target area steadily and safely, a path smoother is required. This paper puts forward a novel path planning method based on extreme learning machine (ELM), which generates a smooth and safe path at a quite fast speed. First a Voronoi preprocessor is employed to label the obstacles in the map and generate a rough path connecting the initial and the goal positions. Due to the property of nonlinear separating surface as well as fast learning speed, ELM is then applied to regenerate and smooth the path to ensure that the vehicle drives automatically and safely. Related experiments also validate the performance of our proposed method as expected. More analysis and some possible limitations are also discussed.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

We present DDPG-AACO, a hierarchical method for welding manipulator path planning of complex components welding tasks that combines reinforcement learning (RL) with the intelligent optimisation algorithm. The RL agent, trained with the deep deterministic policy gradient (DDPG), learns local path planning policies that control the welding manipulator to safely move between two welding seam endpoints. Next, on a distance matrix constructed by the lengths of local paths between every two welding seam endpoints, the adaptive ant colony optimisation (AACO) algorithm with artificially changed value of pheromones is adopted to realise the global path planning that the welding manipulator traverses all welding seams under a welding direction constraint and has the shortest path length. The simulation results show the effectiveness of the method. The DDPG is better than the deep Q-learning-based methods when performing local path planning. Moreover, the length of the global path with direction constraint can converge to the minimum.

P2P power trading based on reinforcement learning for nanogrid clusters

Autonomous underwater vehicles (AUVs) as an efficient underwater exploration means have been used to perform various marine missions. However, limited by the technologies of underwater acoustic communications and intelligent autonomy, the most current and advanced AUVs only perform a limited number of tasks in the small-scale area and the known underwater environment. Therefore, in this paper, a one path planning model was proposed combining the global path planning and the local path planning for the large-scale complex marine environment. More specifically, the B-spline curve was used to represent the smooth path for the requirement of kinematic constraints of AUVs. After considering the various constraints, such as the energy/time consumption, the turning radius limitation, the marine environment, and the ocean current, the path planning was abstractly modeled as a multi-objective optimization model with the time cost, the curvature cost, the map cost, and the ocean current cost. The swarm hyper-heuristic algorithm (SHH) with the online learning ability was proposed to solve this model with real-time performance and stability. The results showed that the proposed online learning SHH algorithm had obvious advantages in terms of time efficiency, stability, and optimal performance compared with the results of two traditional heuristic algorithms, both particle swarm optimization (PSO) and firefly algorithm (FFA). The time efficiency of the online learning SHH algorithm improved at least 20% compared with PSO and FFA.

An autonomous underwater vehicle (AUV) must move from a start point to a target point without colliding with other vehicles or obstacles to execute various missions. Path planning is a crucial function that causes AUVs to find the shortest and safest path without human intervention. This paper presents a path planning algorithm that adopts a safety distance to keep the vehicle as safe as possible while avoiding collisions with obstacles. The algorithm uses a combination of fuzzy logic (FL) and an artificial potential field (APF), which together overcome the shortcomings of global path planning. The proposed algorithm simultaneously considers the shortest path to the target position while maintaining a safe distance from obstacles. The effectiveness of the proposed algorithm is demonstrated with computer simulations.

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

Three-dimensional path planning for autonomous underwater vehicles based on a whale optimization algorithm

The path planning of the autonomous underwater vehicle (AUV) has shown great potential in various Internet of Underwater Things (IoUT) applications. Although considerable efforts had been made, prior studies are confronted with some limitations. For one thing, existing work only uses the ocean current simulation model without introducing real ocean information, having not been supported by real data. For another, traditional path planning algorithms have strong environment dependence and lack flexibility: once the environment changes, they need to be remodeled and replanned. To overcome these challenges, this article proposes comprehensive ocean information D3QN (COID), an AUV path planning scheme exploiting comprehensive ocean information and reinforcement learning (RL), which consists of three steps. First, we introduce the comprehensive real ocean data, including weather, temperature, thermohaline, current, etc., and apply them into the regional ocean modeling system to generated reliable ocean current. Next, through well-designed state transition function and reward function, we build a 3-D grid model of ocean environment for RL. Furthermore, based on the framework of the double dueling deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (D3QN), COID integrates local ocean current and position features to provide state input and uses priority sampling to accelerate network convergence. The performance of COID has been evaluated and proved by numerical results, which demonstrate efficient path planning and high flexibility for expansion into different ocean environments.

Aiming at the global three-dimensional path planning of autonomous underwater vehicles (AUV) in ocean currents, a three-dimensional path planning method based on the improved imperial competition algorithm (ICA) is proposed. First, establish a three-dimensional environment model and a cone three-dimensional ocean current model, and derive a cost function that measures path length, energy consumption, and sailing time to establish a multi-objective performance evaluation model to achieve global comprehensive optimization of AUV path planning; secondly, in the algorithm The colonial revolution part added the differential evolution algorithm to ensure that the algorithm will not lose outstanding individuals while increasing the diversity of the population; finally, the algorithm proposed in this article and the other three algorithms are simulated and analyzed on three different scale maps. The simulation results show that the improved ICA has a strong global search capability, improves the accuracy of the optimization process, strengthens the global optimization, and effectively solves the problem that traditional ICA is easy to fall into local optimal solutions, and can make AUVs Obtain the global optimal path, and the comprehensive performance index of the path is better than other comparison algorithms. KeywordsAutonomous underwater vehicleThree-dimensional path planningOcean current environmentMulti-objective optimizationImperial competition algorithm