Path planning based on reinforcement learning

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Path planning is essential for mobile robots to operate efficiently in uncertain environments. As the number of robots increases, centralised approaches struggle to provide feasible solutions within real-time constraints. To address this, an adaptive hybrid strategy-based decentralised path planning algorithm is proposed to solve the real-time path planning problem through decentralised computing. First, various path planning strategies are introduced, including a neural computing planning strategy, a dynamic search planning strategy, and a cluster coordination planning strategy. Then, an intelligent strategy selection mechanism with an adaptive strategy adjustment factor is designed, allowing each robot to dynamically select optimal planning strategies based on their current state and ensuring the planned paths exhibit greater flexibility and adaptability. Finally, the results of the ablation experiment indicate that all three strategies effectively enhance the navigation capabilities of robots in a decentralised mode. The algorithm comparison experiment demonstrates that the proposed algorithm achieves a higher task completion rate and a lower detour percentage in various environments. The decision response experiment shows that our approach has an average decision-making time of approximately 60 ms, which meets the real-time requirements for decentralised path planning of mobile robots in most scenarios.

PurposeThe purpose of this paper is to put forward a rolling assistant robot with two rolling modes, and the multi-mode rolling motion strategy with path planning algorithm, which is suitable to this multi-mode mobile robot, is proposed based on chessboard-shaped grid division (CGD).Design/methodology/approachBased on the kinematic analysis and motion properties of the mobile parallel robot, the motion strategy based on CGD path planning algorithm of a mobile robot with two rolling modes moving to a target position is divided into two parts, which are local self-motion planning and global path planning. In the first part, the mobile parallel robot can move by switching and combining the two rolling modes; and in the second part, the specific algorithm of the global path planning is proposed according to the CGD of the moving ground.FindingsThe assistant robot, which is a novel 4-RSR mobile parallel robot (where R denotes a revolute joint and S denotes a spherical joint) integrating operation and rolling locomotion (Watt linkage rolling mode and 6R linkage rolling mode), can work as a moving spotlight or worktable. A series of simulation and prototype experiment results are presented to verify the CGD path planning strategy of the robot, and the performance of the path planning experiments in simulations and practices shows the validation of the path planning analysis.Originality/valueThe work presented in this paper is a further exploration to apply parallel mechanisms with two rolling modes to the field of assistant rolling robots by proposing the CGD path planning strategy. It is also a new attempt to use the specific path planning algorithm in the field of mobile robots for operating tasks.

Mobile robots are mostly used in many industrial and military applications. Researches in the path planning is one of the most important aspects in mobile robot researches. Path planning for a mobile robot need to find a collision free path through the robot's environment with obstacles from the specified start location to a desired(target) goal location while satisfying certain optimization conditions. Existing path planning methods like graphical methods such as visibility graph, the potential field and the cell decomposition are designed for static environments, in which there are stationary obstacles. In practical systems such as Marine Science Research, Robots in Industry, and military combat applications, robots usually face dynamic environments where both moving and stationary obstacles exist. This project proposes a heuristic based method to search the feasible initial path efficiently. The heuristic based method is then combined into the simulated annealing algorithm based approach for dynamic robot path planning. Therefore the quality of the solution is characterised by the length of the planned path and it is improved with the combined heuristic method in the simulated annealing based approach for both runtime and offline path planning.

Purpose This review paper aims to provide a comprehensive review of Dubins robot path planning, which has become increasingly important in mobile robotics, autonomous vehicles and industrial applications. The authors identify key developments, analyze existing approaches and highlight the fundamental constraints, algorithms and future opportunities involving curvature-constrained, forward-only path planning. Design/methodology/approach The review consists of three parts: basic Dubins theory, seven decades of development and future opportunities. In the first part, the authors summarize problem formulations and representative computational strategies/algorithms. In the second part, the seventy years of development are divided into four stages, with each stage summarizing developments in important topics such as point-to-point planning, obstacle avoidance and variable speed planning. In addition, the authors analyze current challenges and evaluate recent extensions in cooperative, real-time and learning-based planning frameworks. Findings Dubins path planning has evolved from simple analytical models to complex multirobot and dynamic scenarios. While exact solutions exist for basic tasks, the scalability, obstacle constraints and real-time responsiveness remain active challenges. Promising future directions include integrating machine learning for adaptive planning, and extending to Reeds–Shepp and nonholonomic systems in three-dimensional environments. Originality/value This review offers a structured and in-depth overview of path planning for Dubins robots, connecting core theoretical concepts with the latest developments and real-world implementations. Unlike prior surveys that roughly treat general nonholonomic planning or sampling-based methods, this paper for the first time emphasizes the curvature-constrained planning, especially under the Dubins constraints. By identifying underexplored areas such as dynamic environments, real-time applications, this review offers actionable insights and future research directions valuable to both researchers and practitioners in mobile and industrial robots.

This paper deals with a new approach based on Q-learning for solving the problem of mobile robot path planning in complex unknown static environments. As a computational approach to learning through interaction with the environment, reinforcement learning algorithms have been widely used for intelligent robot control, especially in the field of autonomous mobile robots. However, the learning process is slow and cumbersome. For practical applications, rapid rates of convergence are required. Aiming at the problem of slow convergence and long learning time for Q-learning based mobile robot path planning, a state-chain sequential feedback Q-learning algorithm is proposed for quickly searching for the optimal path of mobile robots in complex unknown static environments. The state chain is built during the searching process. After one action is chosen and the reward is received, the Q-values of the state-action pairs on the previously built state chain are sequentially updated with one-step Q-learning. With the increasing number of Q-values updated after one action, the number of actual steps for convergence decreases and thus, the learning time decreases, where a step is a state transition. Extensive simulations validate the efficiency of the newly proposed approach for mobile robot path planning in complex environments. The results show that the new approach has a high convergence speed and that the robot can find the collision-free optimal path in complex unknown static environments with much shorter time, compared with the one-step Q-learning algorithm and the Q(λ)-learning algorithm.

The application of reinforcement learning algorithms to motion planning is a research hotspot in robotics in recent years. However, training reinforcement learning agents from scratch has low training efficiency and difficulty in convergence. In this paper, a robot motion planning method based on residual reinforcement learning is proposed. This method divides the agent's policy of motion planning into initial policy and residual policy. The initial policy is composed of a neural network motion planner responsible for guiding the training direction of residual policy. The residual policy is composed of Proximal Policy Optimization (PPO) algorithm in reinforcement learning. A motion planning experiment is carried out in a simulation environment, and the result shows that the method can successfully perform motion planning. The comparison experiment between PPO and the proposed algorithm demonstrates that the proposed algorithm has better motion planning performance.

According to the problem of obstacles avoidance path planning in mobile robot system, this article uses the Q-learning algorithm in reinforcement learning to solve it. The original Q-learning algorithm has the problem of low learning efficiency, then an improved algorithm is proposed that adds a layer of learning process based on it. So that the mobile robot can find obstacles and target positions as soon as possible, which accelerates the efficiency of path planning and improves the efficiency. Finally, the path planning is established by a grid method on Python, and the comparison between the original algorithm, the improved algorithm proves that the learning efficiency of the algorithm is improved.

Intelligent manufacturing is the theme and main direction of the “Made in China 2025” development strategy, and the application of industrial robots is an important direction of intelligent manufacturing. In the next few years, industrial robots will be widely used in various enterprises, which will inevitably require a large number of high-tech industries. Industrial robots are high-tech products of modern society and play an important role in the process of economic development, especially in the manufacturing industry. Industrial robot technology is widely used in automated production lines, so that industrial production efficiency can be greatly improved. Instead of manual labor to carry out various complex production and operations, to achieve industrial production automation. Analyze the application of industrial robots in automated production lines at this stage, and explore their future development directions, so that they can better serve the manufacturing industry. This article takes the industrial robots and PLCs in the industrial automation production line as the research object, and takes the automated production line for mass production of a product as an example. It explains the joint control scheme of the PLC controller and the industrial robot controller, and provides for the construction of joint control of similar production lines in the future.

The recent advances in robotics has resulted in a more convenient use of mobile robots in alications such as assisting the disabled, deliveries and domestic purposes. The main challenge faced by mobile robots is navigation in a dynamic environment, which is path planning for dynamic obstacle avoidance. This paper proposes a novelty method for solving the path planning problem for mobile robots posed by dynamic obstacles based on SLAM (Simultaneous Localisation and Maing) algorithm and Reinforcement Learning. The algorithms implemented relied on the Kinect sensor for maing and rotary encoder for localisation of the robot in the map. The SLAM algorithm implemented resulted in a mean error metric of 4.07%. The modified Q-learning algorithm implemented in this paper allowed the mobile robot to avoid dynamic obstacles by re-planning the path to find another optimal path different from the previously set global optimal path. From the investigation, it was shown that it is possible for a robot to navigate in a dynamic using the Reinforcement Learning technique.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

This paper presents an adaptive approach based on the Reinforcement Learning (RL) method to manipulate and transport parts and also assemble 3-D structures in a moderately constrained and dynamic environment using a quad-rotor. Nowadays, complex construction tasks using mobile robots are characterized by two fundamental problems such as task planning and motion planning. However, to obtain the task and path planning that define a specific sequence of operations for construction of a given structure is generally very complex. In this context, we propose and investigate a system in which an aerial robot learns the assembly and construction tasks of multiple 3-D structures. This process involves the learning of the sequence of maneuvers of a vehicle, the assembly sequence of the parts and also the correct types of structural elements for each assembly point of the structure. A heuristic search algorithm is used in the learning process to find the optimal path for the quad-rotor so that its navigation through the dynamic environment is performed. The experimental results show that a 3-D structure can be built using the task planning approach derived from a learning algorithm combined with a heuristic search method.

The study deals with automatic path planning method for cutting operation using an industrial robot. In general, studies of motion planning for industrial robots are divided in two approaches, considering a robot configuration or a tool configuration. However, in case of cutting operation using an ultrasonic cutter, generation of optimal tool path becomes difficult, because it is required consideration of continuous tool posture, wide movable range of robot, and high tool feed speed. In this study, we propose the use of an original local configuration space that generated by two angles that determine tool posture at each cutting point, and generate tool path using an A-star search algorism based on the evaluations of interference, movable range and mobility of robot that varied according to relationship of tool feed speed and angular speed of each joint. As a result, it is found that the proposed path generation method is more effective than conventional methods.

AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop

We present DDPG-AACO, a hierarchical method for welding manipulator path planning of complex components welding tasks that combines reinforcement learning (RL) with the intelligent optimisation algorithm. The RL agent, trained with the deep deterministic policy gradient (DDPG), learns local path planning policies that control the welding manipulator to safely move between two welding seam endpoints. Next, on a distance matrix constructed by the lengths of local paths between every two welding seam endpoints, the adaptive ant colony optimisation (AACO) algorithm with artificially changed value of pheromones is adopted to realise the global path planning that the welding manipulator traverses all welding seams under a welding direction constraint and has the shortest path length. The simulation results show the effectiveness of the method. The DDPG is better than the deep Q-learning-based methods when performing local path planning. Moreover, the length of the global path with direction constraint can converge to the minimum.