Robotic manipulation planning using dynamic RRT

In this paper, a hierarchical reinforcement learning (HRL) architecture, namely a “Hierarchical Deep Deterministic Policy Gradient (HDDPG)” has been proposed and studied. A HDDPG utilizes manager and worker formation similar to other HRL structures. However, unlike others, the HDDPG enables sharing an identical environment and state among workers and managers, while a unique reward system is required for each Deep Deterministic Policy Gradient (DDPG) agent. Therefore, the HDDPG allows easy structural expansion with probabilistic action selection of a worker by the manager. Due to its innate structural advantage, the HDDPG has a merit in building a general AI to deal with a complex time-horizon tasks with various conflicting sub-goals. The experimental results demonstrated its usefulness with a manipulator motion planning problem in a dynamic environment, where path planning and collision avoidance conflict each other. The proposed HDDPG is compared with an HAM and a single DDPG for performance evaluation. The result shows that the HDDPG demonstrated more than 40% of reward gain and more than two times the reward improvement rate. Another important feature of the proposed HDDPG is the biased manager training capability. By adding a preference factor to each worker, the manager can be trained to prefer a certain worker to achieve better success rate for a specific objective if needed.

The problem of collision-free motion planning in dynamic environments has received considerable attention in computational geometry. In this thesis, we study the point robot motion planning problems among dynamic obstacles. A growing disc is a disc obstacle that grows over time. A transient obstacle is an obstacle that appears at a particular time in the environment and may disappear at some later time.

Even though mobile robots have been around for decades, trajectory optimization and continuous time collision avoidance remain subject of active research. Existing methods trade off between path quality, computational complexity, and kinodynamic feasibility. This work approaches the problem using a nonlinear model predictive control (NMPC) framework, that is based on a novel convex inner approximation of the collision avoidance constraint. The proposed Convex Inner ApprOximation (CIAO) method finds kinodynamically feasible and continuous time collision free trajectories, in few iterations, typically one. For a feasible initialization, the approach is guaranteed to find a feasible solution, i.e. it preserves feasibility. Our experimental evaluation shows that CIAO outperforms state of the art baselines in terms of planning efficiency and path quality. Experiments show that it also efficiently scales to high-dimensional systems. Furthermore real-world experiments demonstrate its capability of unifying trajectory optimization and tracking for safe motion planning in dynamic environments.

動的環境下におけるロボットの動作生成

Service robots have a great potential of improving human quality of life by aiding in everyday tasks. However, robots that share an environment and interact with humans still face some challenges that limits their acceptance. One of these challenges is how to move and behave among groups of people, which is a task performed seamlessly by humans and some animals. Motion planning in dynamic environments has been addressed mostly by predicting future position of humans and avoiding them. However, with the increase of the number of persons in such environments, techniques that are based only on the prediction of the movement of humans can fail, as they usually ignore the human's reaction to the presence of the robot. Instead of trying to model the complex human motion behavior, this work proposes to rely on humans to guide the robot through difficult situations, where modern approaches would fail to find a solution. This will be accomplished by a probabilistic approach for selecting a human leader, according to the robot's desired destination. In this way, the robot can take advantage of the humans' paths and behavior, effortlessly avoiding dynamic and static features together with the human leader, relieving the robot from the burden of having to generate its own path in difficult situations.

Motion planning in dynamic environment using maneuverability maps

A robot in a dynamic environment needs to avoid static obstacles and perform motion planning while considering dynamic obstacles. This paper presents a Configuration-time-space (C-time-space), in which time-series information is added to the configuration space (C-space). The C-time-space is represented by the mapping position and joint displacement of the robot at each time in the generalized coordinate space. Motion planning using the C-time-space can generate a trajectory that avoids dynamic obstacles. The experimental evaluation results indicate that compared to the conventional method, the trajectory distance of the robot arm can be shortened in environments where dynamic objects exist using the proposed method.

An object pick-and-place system with a camera, a six-degree-of-freedom (DOF) robot manipulator, and a two-finger gripper is implemented based on the robot operating system (ROS) in this paper. A collision-free path planning method is one of the most fundamental problems that has to be solved before the robot manipulator can autonomously pick-and-place objects in complex environments. In the implementation of the real-time pick-and-place system, the success rate and computing time of path planning by a six-DOF robot manipulator are two essential key factors. Therefore, an improved rapidly-exploring random tree (RRT) algorithm, named changing strategy RRT (CS-RRT), is proposed. Based on the method of gradually changing the sampling area based on RRT (CSA-RRT), two mechanisms are used in the proposed CS-RRT to improve the success rate and computing time. The proposed CS-RRT algorithm adopts a sampling-radius limitation mechanism, which enables the random tree to approach the goal area more efficiently each time the environment is explored. It can avoid spending a lot of time looking for valid points when it is close to the goal point, thus reducing the computing time of the improved RRT algorithm. In addition, the CS-RRT algorithm adopts a node counting mechanism, which enables the algorithm to switch to an appropriate sampling method in complex environments. It can avoid the search path being trapped in some constrained areas due to excessive exploration in the direction of the goal point, thus improving the adaptability of the proposed algorithm to various environments and increasing the success rate. Finally, an environment with four object pick-and-place tasks is established, and four simulation results are given to illustrate that the proposed CS-RRT-based collision-free path planning method has the best performance compared with the other two RRT algorithms. A practical experiment is also provided to verify that the robot manipulator can indeed complete the specified four object pick-and-place tasks successfully and effectively.

This paper proposes a hybrid Unmanned Aerial Vehicle (UAV) path planning method that combines the Rapidly-exploring Random Tree (RRT) algorithm with Proximal Policy Optimization (PPO). The proposed method aims to enhance the efficiency and adaptability of UAV path planning in complex and dynamic environments. The RRT algorithm excels at quickly generating a feasible path from a start point to a goal point. However, the quality of its paths is often suboptimal, and it lacks adaptability in dynamic settings. In contrast, PPO, a deep reinforcement learning algorithm, optimizes paths through iterative policy updates, enabling the UAV to adapt to environmental changes. Our approach first employs RRT to generate an initial path, which is subsequently refined by PPO to improve smoothness and adaptability. Experimental results demonstrate that the RRT-PPO hybrid method performs favorably in terms of path length, computational time, and obstacle avoidance capability, effectively improving the task completion efficiency of UAVs in complex environments.

There are many sources of uncertainty facing the fuzzy logic controller (FLC) for autonomous mobile robots navigating in changing and dynamic unstructured environments. The traditional type-1 FLC using precise type-1 fuzzy sets cannot fully handle such uncertainties. A type-2 fuzzy logic controller (FLC) using type-2 fuzzy sets can handle such uncertainties to produce a better performance. In this paper, we present the type-2 FLC and its novel application for the real time control of mobile robots. We used the type-2 FLC to implement different robotic behaviours on different robotic platforms in indoor and outdoor unstructured and challenging environments. The type-2 FLCs dealt with the uncertainties facing mobile robots in unstructured environments and resulted in a very good performance that outperformed the type-1 FLCs whilst using smaller rule bases.

This paper presents a learning-based model predictive trajectory planning controller for automated driving in unstructured, dynamic environments with obstacle avoidance. We first address the problem of lacking prior knowledge in unstructured environments by introducing a risk map that maps the density and motion of obstacles and the road to an occupancy risk. Model predictive control is then used to integrate trajectory planning and tracking control into one framework to bridge the gap between planning and control. Meanwhile, we use Gaussian Process (GP) regression to learn the residual model uncertainty for improving the model accuracy. An objective function considering both risks within the feasible region and vehicle dynamics is carefully formulated to obtain collision-free and kinematically-feasible local trajectories. Field experiments are performed on real unstructured environments with our automated vehicle. Experimental results demonstrate the effectiveness of the proposed algorithm for successful obstacle avoidance in various complex unstructured scenarios.

In recent years, the perceptual capabilities of robots have been significantly enhanced. However, the task execution of the robots still lacks adaptive capabilities in unstructured and dynamic environments. In this paper, we propose an ontology based autonomous robot task processing framework (ARTProF), to improve the robot's adaptability within unstructured and dynamic environments. ARTProF unifies ontological knowledge representation, reasoning, and autonomous task planning and execution into a single framework. The interface between the knowledge base and neural network-based object detection is first introduced in ARTProF to improve the robot's perception capabilities. A knowledge-driven manipulation operator based on Robot Operating System (ROS) is then designed to facilitate the interaction between the knowledge base and the robot's primitive actions. Additionally, an operation similarity model is proposed to endow the robot with the ability to generalize to novel objects. Finally, a dynamic task planning algorithm, leveraging ontological knowledge, equips the robot with adaptability to execute tasks in unstructured and dynamic environments. Experimental results on real-world scenarios and simulations demonstrate the effectiveness and efficiency of the proposed ARTProF framework. In future work, we will focus on refining the ARTProF framework by integrating neurosymbolic inference.

The rapidly-exploring random trees (RRT) is a sampling-based path planner which utilizes simultaneously kinematics and dynamics of a robot. However, since the RRT has produced a robot path without taking the existence of dynamic obstacles into consideration, RRT-based navigation has the risk of a collision with dynamic obstacles. We proposed a path replanning technique for the RRT applied to robot navigation in dynamic environments, which is named grafting. The proposed technique replans a safe and efficient path in real time instead of the original path which may cause a collision with dynamic obstacles. Moreover, the replanned path can be easily merged into the original RRT path because the grafting technique preserves the property of the RRT. The grafting technique was tested by simulations in various dynamic environments, which revealed that the grafting technique was capable of replanning a safe and efficient path for RRT-based navigation in real time.

As a sampling based path planning method, Rapidly-exploring Random Tree (RRT) is powerful for its high efficiency in finding feasible solutions. Optimal RRT (RRT*) improves RRT by asymptotically searching the optimal solution, however, at the cost of time consuming. This paper proposes an efficient optimal planning algorithm based on RRT* with adaptive restricted sampling space, which is inspired by the tropism of plants and named as tropistic RRT*. Tropistic RRT* improves the efficiency of the path planning by restricting the sampling space to approach the goal position just like the phototropism of plants. To validate our proposed algorithm, we carry out both path planning and motion planning experiments in 2D environments. In the path planning experiments, we compare the proposed algorithm with RRT* and Informed RRT* and experimental results show that our method outperforms the other two in the convergence rate, final solution cost and robustness. Furthermore, we also implement our algorithm in solving motion planning problem for a nonholonomic mobile robot and compare the time cost in the robot navigation process. Compared results exhibit that the time cost of our algorithm is much less than that of Real-Time RRT* in both static and dynamic environments.

In dynamic environments with continuous variability, such as those affected by typhoons, ship path planning must account for both navigational safety and the maneuvering characteristics of the vessel. However, current methods often struggle to accurately capture the continuous evolution of dynamic obstacles and generally lack adaptive exploration mechanisms. Consequently, the planned routes tend to be suboptimal or incompatible with the ship’s maneuvering constraints. To address this challenge, this study proposes a Space–Time Integrated Q-Learning (STIQ-Learning) algorithm for dynamic path planning under typhoon conditions. The algorithm is built upon the following key innovations: (1) Spatio-Temporal Environment Modeling: The hazardous area affected by the typhoon is decomposed into temporally and spatially dynamic obstacles. A grid-based spatio-temporal environment model is constructed by integrating forecast data on typhoon wind radii and wave heights. This enables a precise representation of the typhoon’s dynamic evolution process and the surrounding maritime risk environment. (2) Optimization of State Space and Reward Mechanism: A time dimension is incorporated to expand the state space, while a composite reward function is designed by combining three sub-reward terms: target proximity, trajectory smoothness, and heading correction. These components jointly guide the learning agent to generate navigation paths that are both safe and consistent with the maneuverability characteristics of the vessel. (3) Priority-Based Adaptive Exploration Strategy: A prioritized action selection mechanism is introduced based on collision feedback, and the exploration factor ϵ is dynamically adjusted throughout the learning process. This strategy enhances the efficiency of early exploration and effectively balances the trade-off between exploration and exploitation. Simulation experiments were conducted using real-world scenarios derived from Typhoons Pulasan and Gamei in 2024. The results demonstrate that in open-sea environments, the proposed STIQ-Learning algorithm achieves reductions in path length of 14.4% and 22.3% compared to the D* and Rapidly exploring Random Trees (RRT) algorithms, respectively. In more complex maritime environments featuring geographic constraints such as islands, STIQ-Learning reductions of 2.1%, 20.7%, and 10.6% relative to the DFQL, D*, and RRT algorithms, respectively. Furthermore, the proposed method consistently avoids the hazardous wind zones associated with typhoons throughout the entire planning process, while maintaining wave heights along the generated routes within the vessel’s safety limits.