Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships

This paper presents a real-time path planning approach for controlling the motion of space-based robots. The algorithm can plan three-dimensional trajectories for agents in a complex environment which includes numerous static and dynamic obstacles, path constraints, and/or performance constraints. This approach is extended based on the dynamic window approach (DWA). As the classic reactive method for obstacle avoidance, DWA uses an optimized function to select the best motion command. The original DWA optimization function consists of three weight terms. Changing the weights of these terms will change the behavior of the algorithm. In this paper, to improve the evaluation ability of the optimization function and the robot’s ability to adapt to the environment, a new optimization function is designed and combined with fuzzy logic to adjust the weights of each parameter of the optimization function. Given that DWA has the defect of local minima, which makes the robot hard to escape U-shaped obstacles, a dual dynamic window method and local goals are adopted in this article to help the robot escape local minima. By comparison, the proposed method is superior to traditional DWA and fuzzy DWA (F_DWA) in terms of computational efficiency, smoothness and security.

It is essential to investigate the path planning and tracking problem for maritime autonomous surface ships (MASS). This study proposed an automatic collision avoidance method based on the separation and recovery mechanism with formation of MASS (F-MASS). First, the marine environment model utilized S-57 electric navigation chart (ENC) data for construction. The Scan-searching method was proposed for multipath and path-optimized redistribution method was developed to address MASS congestion. The formation pattern transformation was settled by separation and recovery mechanism. Considered the ‘unmanned–manned’ encounter situation, formulated avoidance strategies for both the MASS formation and manned ships encounters. Finally, the simulation experiment was carried out in the sea near Zhoushan port to verify the effectiveness of the algorithm. The results indicated that the method can satisfy the integrated operation of collision avoidance in a complex environment and can offer effective collision avoidance strategies for F-MASSs navigating in complex marine environments.

This work presents a hybrid mobile robot navigation system for simulated environments. The system integrates the Dynamic Window Approach (DWA) with a deep reinforcement learning (DRL) using an actor-critic architecture. Initially, the robot uses DWA, while concurrently, a DRL agent generates alternative control speeds using actor and critic networks. An entropy-based mechanism dynamically weights the DWA and DRL speeds, transitioning gradually to DRL control. The system's performance is evaluated using success rate and trajectory length, comparing the hybrid approach to DWA and DRL alone. Results demonstrate improved robustness and performance, particularly in complex scenarios.

To address issues such as insufficient real-time performance, poor obstacle avoidance flexibility, and slow emergency stop response in path planning for transport robots operating within complex, dynamic hospital environments, this paper proposes a hybrid path planning strategy that integrates an improved A* algorithm with the Dynamic Window Algorithm (DWA). This strategy is implemented within the ROS2 framework. First, the traditional $\mathbf{A}^{*}$ algorithm is optimized by introducing a dynamic weighted Manhattan distance <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</sup> as the heuristic function. This incorporates path smoothness and safety distance penalty terms, enhancing the rationality and efficiency of the global path. Second, at the local planning layer, a multi-modal perception fusion dynamic window algorithm is designed. This method dynamically embeds real-time evaluations of LiDAR, visual semantic information, and UWB positioning data into the evaluation function, enabling adaptive obstacle avoidance for diverse obstacles such as pedestrian flows, IV stands, and mobile hospital beds. Experiments demonstrate that the improved algorithm reduces planning time by approximately 35% while maintaining near-optimal path length. Emergency stop response time is $\boldsymbol{\leqslant} 180 \mathrm{~ms}$, and dynamic obstacle avoidance success rate increases to 98.7%, effectively supporting high-frequency, high-priority medication transport tasks in medical settings.

The autonomous navigation in the crowded environment is a challenging task due to the sensor occlusion and the complex nature of the abstract social interactions. And yet, humans are capable of navigating in such complex environment. In this paper, we propose an effective navigation method that combines the learning-based and model-based methods in a way that a cost function that includes human imitation factor learned via deep learning is integrated into the dynamic window approach (DWA) [1]. The experiments conducted on simulations show that by training the robot to imitate the human trajectory, our navigation method is safer and more efficient than the state-of-the-art methods. Additionally, we successfully deployed a physical robot in an actual environment, and we validate that our navigation quality shares similar tendency with human in the path length, travel time, and the collision avoidance.

Collision-free motion is essential for mobile robots. Most approaches to collision-free and efficient navigation with wheeled robots require parameter tuning by experts to obtain good navigation behavior. In this paper, we aim at learning an optimal navigation policy by deep reinforcement learning to overcome this manual parameter tuning. Our approach uses proximal policy optimization to train the policy and achieve collision-free and goal-directed behavior. The output of the learned network are the robot’s translational and angular velocities for the next time step. Our method combines path planning on a 2D grid with reinforcement learning and does not need any supervision. Our network is first trained in a simple environment and then transferred to scenarios of increasing complexity. We implemented our approach in C++ and Python for the Robot Operating System (ROS) and thoroughly tested it in several simulated as well as real-world experiments. The experiments illustrate that our trained policy can be applied to solve complex navigation tasks. Furthermore, we compare the performance of our learned controller to the popular dynamic window approach (DWA) of ROS. As the experimental results show, a robot controlled by our learned policy reaches the goal significantly faster compared to using the DWA by closely bypassing obstacles and thus saving time.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

With the development of Maritime Autonomous Surface Ships (MASS), considerable research is undertaken to secure their safety. One of the critical aspects of MASS is collision avoidance, and multiple collision avoidance algorithms have been developed. However, due to various reasons, collision avoidance of autonomous merchant vessels appears to be far from resolved. With this study, we aim to discuss the current state of Collision Avoidance Methods (CAMs) and the challenges lying ahead—from a joint academic and practical point of view. To this end, the key Rules from International Regulations for Preventing Collisions at Sea (COLREG) have been reviewed with a focus on their practical application for MASS. Moreover, the consideration of the COLREG Rules in contemporary collision avoidance algorithms has been reviewed. The ultimate objective is to identify aspects of COLREG requiring additional attention concerning MASS developments in terms of collision avoidance. Our conclusions indicate that although a lot of progress has been achieved recently, the feasibility of CAMs for MASS remains questionable. Reasons for so are the ambiguous character of the regulations, especially COLREG, as well as virtually all existing CAMs being at best only partly COLREG-compliant.

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

Follow the Gap Method (FGM) is an obstacle avoidance method which uses gap arrays. This method recursively directs the robot to the goal state while avoiding the obstacles through the safest gap. Since FGM is a geometric method, it does not consider the robot dynamics. For this reason, oscillations or collisions due to robot dynamics are possible. On the other hand, FGM calculates the desired heading angle, but it does not give linear and angular velocity reference. Dynamic Window Approach (DWA) is one of the most popular obstacle avoidance algorithms which does take robot dynamics into consideration. It calculates best angular and linear velocity pair which is chosen by an objective function. In this paper, an FGM-DW approach which uses the strongest elements of FGM and DWA methods to achieve safe, smooth and fast navigation is proposed. The FGM-DW approach provides these concerns and meets the low level angular and rotational velocity requirement of FGM. In this paper, the performance and analysis of FGM-DW are shown by both simulations and real-world experimental tests.

Bounding volume based approaches in velocity obstacle (VO) provide a good solution for collision avoidance of mobile robots with uncertainty. However, the VO built with the bounding footprint always has over-constraining problems which may lead to conservative maneuvers of the mobile robots. Addressing this problem, a vertical ellipse based velocity obstacle (VEVO) collision avoidance method is proposed in this paper. The method mitigates the over-constraining situation by building the footprint probability ellipse whose minor axis is vertical to the direction of the obstacle to minimize the VO area. Based on VEVO, a DWA (Dynamic Window Approach) integrated method is proposed to provide a set of available velocities in speed selection. According to different collision avoidance objectives like collision safety, shortest time consumption and shortest trajectory length, a multi-objective velocity selecting strategy is proposed to provide optimal velocities for motion planning. Furthermore, a dynamic local path adjustment method is proposed to help robots react to the closest obstacle (dynamic or static) according to different collision safety requirements. We validate our methods in a simulated workspace with different numbers of robots going to their goal points. Experimental results show VEVO method could improve the collision avoidance performance in crowded multi-robot environment and robots could achieve their different objectives when suitable parameters are set in the velocity evaluation function. The proposed dynamic local path adjustment method only affects the trajectories in local areas and could ensure collision avoidance safety and performance at the same time.

This article proposes, a novel obstacle avoidance algorithm for a mobile robot based on finite memory filtering (FMF) in unknown dynamic environments. To overcome the limitations of the existing dynamic window approach (DWA), we propose a new version of the DWA, called the finite distribution estimation-based dynamic window approach (FDEDWA), which is an algorithm that avoids dynamic obstacles through estimating the overall distribution of obstacles. FDEDWA estimates the distribution of obstacles through the FMF, and predicts the future distribution of obstacles. The FMF is derived to minimize the effect of the measurement noise through the Frobenius norm, and covariance matrix adaptation evolution strategy. The estimated information is used to derive the control input for the robust mobile robot navigation effectively. FDEDWA allows for the fast perception of the dynamic environment, and superior estimation performance, and the mobile robot can be controlled by a more optimal path while maintaining real-time performance. To demonstrate the performance of the proposed algorithm, simulations, and experiments were carried out under dynamic environments by comparing the latest dynamic window for dynamic obstacle, and the existing DWA.

As a classic algorithm for realizing robot local path planning, the dynamic window approach (DWA) uses an objective function to choose the optimal velocity commands. However, the path generated by the DWA in the complex environment is not smooth. Therefore, this paper proposes an improved DWA algorithm to make the path of the robot more smoother when avoiding obstacles. At the same time, in view of the condition that the weight coefficients of the evaluation function of the original DWA remain unchanged, this paper will add a fuzzy controller to realize the weight coefficients adaptation, so as to adapt to a more complex environment, and the generated path can be smoother.

This paper proposes a hybrid Unmanned Aerial Vehicle (UAV) path planning method that combines the Rapidly-exploring Random Tree (RRT) algorithm with Proximal Policy Optimization (PPO). The proposed method aims to enhance the efficiency and adaptability of UAV path planning in complex and dynamic environments. The RRT algorithm excels at quickly generating a feasible path from a start point to a goal point. However, the quality of its paths is often suboptimal, and it lacks adaptability in dynamic settings. In contrast, PPO, a deep reinforcement learning algorithm, optimizes paths through iterative policy updates, enabling the UAV to adapt to environmental changes. Our approach first employs RRT to generate an initial path, which is subsequently refined by PPO to improve smoothness and adaptability. Experimental results demonstrate that the RRT-PPO hybrid method performs favorably in terms of path length, computational time, and obstacle avoidance capability, effectively improving the task completion efficiency of UAVs in complex environments.

Water quality plays a vital role in aquaculture sustainability, particularly for sensitive freshwater species such as Salmo letnica, which require strict physicochemical conditions for survival. This study evaluates the suitability of household water sources from eight rural villages in North Macedonia for supporting Salmo letnica in home-based tank environments. Seven key water quality parameters were measured and compared against biological thresholds derived from ecological literature. A Deep Reinforcement Learning (DRL) agent, trained using the Proximal Policy Optimization (PPO) algorithm, was developed to classify the suitability of water samples based on these parameters. While the DRL model exhibited low precision and recall due to the limited sample size, it provided a framework for interpretability through reward dynamics and parameter correlations. Among the eight villages, Forino and Gradec were found to meet all critical biological thresholds, while Kamenjane and Vrapçishte were identified as marginally suitable. The remaining locations exhibited insufficient oxygen levels or excessive nutrient concentrations. These findings demonstrate the potential of AI-based classification models in supporting aquaculture planning and ecological risk assessment. Future work will focus on data expansion, reward function refinement, and field-level model deployment. Unlike traditional supervised classifiers, the DRL agent enables autonomous learning and decision-making without requiring large labeled datasets. This makes the approach suitable for real-time, remote water quality monitoring in resource-limited rural areas.