A Study of Reinforcement Learning for Indoor Path Planning with Dynamic Obstacles

Integrated path planning and control through proximal policy optimization for a marine current turbine

Background: With the development of robotics, more and more robots are being used in industrial production. However, as the production environment surrounding the robots becomes increasingly complex, there is a need for more intelligent industrial robots. For improving the intelligence of industrial robots, the most important thing is to ensure that they can carry out safe industrial production activities in a complex production environment. Therefore, studying the autonomous dynamic obstacle avoidance path planning of industrial robots in complex environments is of great significance for improving the intelligence of industrial robots and the application of human-machine collaboration. Objective: The main purpose of this paper is to improve the traditional artificial potential field method. It aims to improve the disadvantages of the traditional artificial potential field method, such as falling into the local minimum and failing to reach the target. Secondly, the background difference method, which is based on binocular vision and Kalman filtering algorithm, is used, and the environmental map containing the static and dynamic obstacles is obtained. After obtaining the information on the position of static and dynamic obstacles, the robot arm can make good use of the improved artificial potential field method to plan its own trajectory, thus realizing the dynamic obstacle avoidance of the robot arm in a complex environment. Methods: This paper proposes an improved artificial potential field method. First of all, in order to solve the problem of not being able to achieve the goal, the method of modifying the repulsion field function, as proposed by Wang Huili and others, has been cited. In the traditional repulsion function, the relative distance between the robot and the target is introduced. Due to this, the target point is always the minimum point of the potential field. It is necessary to ensure that the robot can reach the target position smoothly. For the local minimum problem, by adding a gravitational increase factor β, the gravitational force received is greater than the repulsive force, thereby breaking the balance of the resultant force. Finally, for the traditional artificial potential field method that does not consider the path planning problem in the presence of dynamic obstacles, the velocity vector of the dynamic obstacle is brought into the potential field function to improve the traditional artificial potential field method. Results: The robot easily falls into the local optimum during path planning, and the improved artificial potential field method overcomes this shortcoming. Therefore, the manipulator can perform autonomous dynamic obstacle avoidance path planning in an environment with dynamic obstacles, finally reaching the target point safely. Conclusions: The industrial production field faces an increasing demand for intelligent industrial robots. This paper improves on the traditional artificial potential field method based on binocular vision so that the mechanical arm can avoid dynamic obstacles autonomously in the presence of dynamic obstacles, thereby improving the degree of intelligence of the robotic arm, enabling the technology to meet the needs of future industrial development and contribute to the development of future industrial robot technology.

Controlling 6-Degree-of-Freedom (6-DOF) robotic arms for precise manipulation tasks is challenging due to kinematic redundancy and the complexity of existing simulation environments like MuJoCo or ROS-Gazebo. This paper presents ArmReach6DOFEnv, a lightweight, open-source simulation framework built on PyBullet for rapid Reinforcement Learning (RL) prototyping on 6-DOF robotic arms. Using a Universal Robot Description Format (URDF) model, the environment supports a continuous state-action space for a 3D reaching task, with a reward function balancing accuracy and control effort. We evaluate two state-of-the-art RL algorithms, Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG), implemented via Stable-Baselines3, comparing their convergence, success rate, and motion smoothness. Experimental results demonstrate DDPG’s superior performance (69% success rate vs. PPO’s 34%) and smoother trajectories, despite PPO’s faster convergence. This framework enables accessible RL experimentation on resource-constrained systems, with potential for future sim-to-real transfer.

This paper proposes a hybrid Unmanned Aerial Vehicle (UAV) path planning method that combines the Rapidly-exploring Random Tree (RRT) algorithm with Proximal Policy Optimization (PPO). The proposed method aims to enhance the efficiency and adaptability of UAV path planning in complex and dynamic environments. The RRT algorithm excels at quickly generating a feasible path from a start point to a goal point. However, the quality of its paths is often suboptimal, and it lacks adaptability in dynamic settings. In contrast, PPO, a deep reinforcement learning algorithm, optimizes paths through iterative policy updates, enabling the UAV to adapt to environmental changes. Our approach first employs RRT to generate an initial path, which is subsequently refined by PPO to improve smoothness and adaptability. Experimental results demonstrate that the RRT-PPO hybrid method performs favorably in terms of path length, computational time, and obstacle avoidance capability, effectively improving the task completion efficiency of UAVs in complex environments.

Aiming at the problem of dynamic path planning for UAVs in complex environment, a path planning algorithm combining dynamic obstacle trajectory estimation and variable step rolling window deduction is proposed. When the dynamic obstacle distance is close to the UAV, the Kalman filter is used to estimate the dynamic obstacle trajectory. Then the window deduction is performed to judge the track area of the deduction window, so that the algorithm has certain "forward-looking". In turn, the window step size, the flight speed of the drone and the flight direction are changed, and the avoidance of the dynamic obstacle is completed, and the final goal is reached by the rolling plan. The experimental analysis shows that the algorithm can complete the dynamic path planning of UAVs in complex environment in real time.

Aiming at the problem of multi-objective path planning (MOPP) and dynamic obstacle avoidance of manipulator in dynamic environment, this paper proposes a solution based on hierarchical reinforcement learning (HRL) framework. Traditional path planning methods have some problems in dynamic scenes, such as poor real-time performance, difficult to balance multi-objective conflicts and insufficient adaptability to environmental changes. Therefore, this paper designs a two-tier architecture including global path planning layer and local obstacle avoidance layer, which is trained by Proximal policy optimization (PPO) and Soft Actor-Critic (SAC) algorithms respectively, and achieves collaborative optimization through an efficient information exchange mechanism between layers. At the same time, a multi-objective reward function based on dynamic weight adjustment strategy is introduced. Fuzzy logic is used to adaptively balance the relationship among path length, obstacle avoidance safety and energy consumption according to environmental complexity. Combined with Long Short-Term Memory (LSTM), the trajectory of obstacles is predicted, and the potential field method is further introduced to modify the obstacle avoidance reward, which improves the real-time response ability and robustness of the algorithm in dynamic environment. The experimental results show that the HRL-SAC-PPO method proposed in this paper shows superior performance in both static and dynamic scenarios. In the static scene, the success rate of this method reaches 100%, the average path length is shortened to 2.13m, no collision occurs, and the energy consumption is reduced to 1.12 kJ, which shows a good multi-objective optimization effect. In the dynamic scene, the trajectory error of obstacles predicted by LSTM is only 4.2%, and the safe distance between the robot arm and obstacles is improved by 35%, which significantly enhances the reliability of obstacle avoidance. In addition, the average decision delay of this method is only 11.3ms, and the peak delay is 23ms, which is much lower than that of the contrast algorithm, showing stronger real-time response ability. The ablation experiment further verified the key role of LSTM trajectory prediction, dynamic weight adjustment and layered structure on the overall performance. In the welding task verification of the real UR5 manipulator, the success rate of the system in dynamic environment is 95.3%, the average path length is 3.41m, and the maximum joint acceleration is only 0.87 radian/s², which is far below the safety threshold, indicating that the algorithm has good stability and obstacle avoidance ability in practical application. The comprehensive performance comparison shows that this method performs well in different industrial scenarios, and has stronger environmental adaptability and comprehensive path planning ability.

In dynamic environments with continuous variability, such as those affected by typhoons, ship path planning must account for both navigational safety and the maneuvering characteristics of the vessel. However, current methods often struggle to accurately capture the continuous evolution of dynamic obstacles and generally lack adaptive exploration mechanisms. Consequently, the planned routes tend to be suboptimal or incompatible with the ship’s maneuvering constraints. To address this challenge, this study proposes a Space–Time Integrated Q-Learning (STIQ-Learning) algorithm for dynamic path planning under typhoon conditions. The algorithm is built upon the following key innovations: (1) Spatio-Temporal Environment Modeling: The hazardous area affected by the typhoon is decomposed into temporally and spatially dynamic obstacles. A grid-based spatio-temporal environment model is constructed by integrating forecast data on typhoon wind radii and wave heights. This enables a precise representation of the typhoon’s dynamic evolution process and the surrounding maritime risk environment. (2) Optimization of State Space and Reward Mechanism: A time dimension is incorporated to expand the state space, while a composite reward function is designed by combining three sub-reward terms: target proximity, trajectory smoothness, and heading correction. These components jointly guide the learning agent to generate navigation paths that are both safe and consistent with the maneuverability characteristics of the vessel. (3) Priority-Based Adaptive Exploration Strategy: A prioritized action selection mechanism is introduced based on collision feedback, and the exploration factor ϵ is dynamically adjusted throughout the learning process. This strategy enhances the efficiency of early exploration and effectively balances the trade-off between exploration and exploitation. Simulation experiments were conducted using real-world scenarios derived from Typhoons Pulasan and Gamei in 2024. The results demonstrate that in open-sea environments, the proposed STIQ-Learning algorithm achieves reductions in path length of 14.4% and 22.3% compared to the D* and Rapidly exploring Random Trees (RRT) algorithms, respectively. In more complex maritime environments featuring geographic constraints such as islands, STIQ-Learning reductions of 2.1%, 20.7%, and 10.6% relative to the DFQL, D*, and RRT algorithms, respectively. Furthermore, the proposed method consistently avoids the hazardous wind zones associated with typhoons throughout the entire planning process, while maintaining wave heights along the generated routes within the vessel’s safety limits.

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3).

The increasing integration of mobile robots in various industries necessitates efficient navigation strategies amidst dynamic environments. Path planning plays a crucial role in guiding mobile robots from their starting points to target destinations, contributing to automation and enhancing human-robot collaboration. This study focuses on devising a tailored path-planning approach for a fleet of mobile robots to navigate through dynamic obstacles and reach designated trajectories efficiently. Leveraging particle swarm optimization (PSO), our methodology optimizes the path while considering real-time environmental changes. We present a simulation-based implementation of the algorithm, where each robot maintains position, velocity, cost, and personal best information to converge towards the global optimal solution. Different obstacles consist of circles, squares, rectangles, and triangles with various colors and five handle-points used. Our findings demonstrate that PSO achieves a global best cost of 5.1017, indicative of the most efficient path, minimizing overall distance traveled.

Dynamic path planning for unmanned surface vehicle in complex offshore areas based on hybrid algorithm

In order to solve the forced shaking problem of the Autonomous Underwater Vehicle (AUV) facing dynamic obstacles in the path planning when the Artificial Potential Field Method is used, a method of removing the shaking state based on a favorable path is proposed, and on this basis, Visibility Graph Method is used to optimize the path of the AUV to avoid dynamic obstacles. By judging the relative motion direction of AUV and dynamic obstacle, the moving direction of AUV is adjusted based on Visibility Graph Method when it is far away from the obstacle. When it is close to the obstacle, the direction conducive to approaching the target point is selected as the favorable forward direction to bypass the obstacle. The simulation results show that the dynamic obstacle avoidance path optimization method based on the Visibility Graph Method reduces the number of moving steps by about 19% compared with the traditional method when dealing with a single dynamic obstacle, and can be applied in complex environments with multiple dynamic obstacles.

Autonomous drone racing has emerged as a challenging benchmark for high-speed robotic perception, decision-making, and control, requiring systems to operate at the limits of agility and safety in complex, dynamic environments. This work presents an abstract overview of autonomous drone racing using deep reinforcement learning (DRL), where an agent learns end-to-end control policies that map raw sensory inputs directly to flight commands through trial-and-error interactions with the environment. Unlike traditional model-based or trajectory optimization approaches that rely on accurate system models and prior knowledge of the racing track, DRL enables drones to learn adaptive strategies that generalize across varying track layouts, gate configurations, and environmental conditions. By formulating drone racing as a sequential decision-making problem, the learning agent is trained to maximize a reward function that balances speed, stability, collision avoidance, and successful gate traversal. Advanced DRL algorithms such as Deep Q-Networks, Proximal Policy Optimization, and Soft Actor-Critic are commonly employed to handle continuous control, high-dimensional state spaces, and stochastic dynamics. Training is typically conducted in high-fidelity simulation environments that incorporate realistic aerodynamics, sensor noise, and delays, followed by sim-to-real transfer techniques to deploy learned policies on physical drones. The use of domain randomization, curriculum learning, and imitation learning from expert demonstrations further improves training efficiency and robustness. Experimental results reported in recent studies demonstrate that DRL-based autonomous drones can achieve competitive lap times, smooth trajectories, and resilient performance under disturbances, often rivaling or surpassing traditional planning-based methods. Overall, deep reinforcement learning provides a scalable and flexible framework for autonomous drone racing, advancing the development of intelligent, high-speed aerial robots and contributing valuable insights applicable to broader domains such as autonomous navigation, search and rescue, and agile robotic

Trajectory planning of robotic manipulators in complex environments involves generating smooth and collision-free paths, and key aspects to consider include dynamic environment perception, path planning, trajectory smoothing and optimization, and obstacle avoidance. This paper discusses different algorithms and finally proposes a fusion of the improved Dung Beetle Optimizer (IDBO) algorithm using chaotic mapping, sinusoidal random mutation, and nonlinear convergence strategies with the bidirectional Rapidly exploring Random Tree (RRT) algorithm to achieve manipulator trajectory planning and minimum jerk optimization trajectory. In the experiment, the IDBO algorithm was compared with other heuristic algorithms. The path obtained was shortened by 9.18% on average, and the calculation time was shortened by 33.81% on average. Then, the paper explored Cartesian coordinate space trajectory planning and joint space trajectory planning when the robot was working in 3D space, and verified the superiority of the latter. In controlling the movement of the robot, by combining the minimum angle jerk planning and the bidirectional RRT obstacle avoidance algorithm to drive the joint angle parameters obtained by the spatial trajectory planning, the trajectory of the robot manipulator can be smoother, more efficient, and avoid obstacles in complex 3D space. This research helps improve the performance, safety, and technical support of robots, and is of great research significance.

With the popularity of service robots in dynamic environments such as hospitals and shopping malls, traditional path planning algorithms have problems such as inflection points of planning paths and difficulty in effectively avoiding dynamic obstacles. In response to these problems, this paper proposes an optimization algorithm based on A* and dynamic window method (DWA). First, in the global path planning, the number of turning points is reduced by deleting redundant turning points. Secondly, by establishing the motion model of the obstacle, the range of the dynamic obstacle is updated to the static environment, and the dynamic obstacle avoidance ability of the DWA algorithm is improved. Finally, the optimized A* algorithm is combined with the DWA algorithm to realize the path planning of the service robot in a dynamic environment. Experimental results show that the algorithm reduces the global path length without disturbing dynamic obstacles, the path is smoother, and the dynamic obstacle avoidance ability is more reliable.

In the task of path planning, the unmanned surface vessels (USV) are required to reach the destination while avoiding obstacle. However, it is difficult for USV to prioritize the two sub-target tasks of destination arrival and obstacle avoidance in a complex dynamic environment using traditional path planning methods. Based on the above considerations, a path planning method for USVs based on strategy integration is proposed. Firstly, the path planning scenario containing dynamic obstacles and two sub-target scenarios are set up. In order to strengthen its ability to reach the destination and avoid dynamic obstacles, the dynamic obstacles and the destination are removed separately in the two sub-target scenarios. Furthermore, the model for strategy selection is trained on action decisions in sub-target scenarios through double deep Q-Network with prioritized experience replay. Finally, a framework for strategy integration is designed to optimize strategy selection for USV. The simulation results show that the proposed method achieves a good overall performance and a higher path planning success rate than the traditional path planning method.