Grid Path Planning for Mobile Robots with Improved Q-learning Algorithm

This paper deals with a new approach based on Q-learning for solving the problem of mobile robot path planning in complex unknown static environments. As a computational approach to learning through interaction with the environment, reinforcement learning algorithms have been widely used for intelligent robot control, especially in the field of autonomous mobile robots. However, the learning process is slow and cumbersome. For practical applications, rapid rates of convergence are required. Aiming at the problem of slow convergence and long learning time for Q-learning based mobile robot path planning, a state-chain sequential feedback Q-learning algorithm is proposed for quickly searching for the optimal path of mobile robots in complex unknown static environments. The state chain is built during the searching process. After one action is chosen and the reward is received, the Q-values of the state-action pairs on the previously built state chain are sequentially updated with one-step Q-learning. With the increasing number of Q-values updated after one action, the number of actual steps for convergence decreases and thus, the learning time decreases, where a step is a state transition. Extensive simulations validate the efficiency of the newly proposed approach for mobile robot path planning in complex environments. The results show that the new approach has a high convergence speed and that the robot can find the collision-free optimal path in complex unknown static environments with much shorter time, compared with the one-step Q-learning algorithm and the Q(λ)-learning algorithm.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

The problem of path exploration and planning of agents in unknown environments is a popular application problem in the field of reinforcement learning. In this paper, we propose an improved reinforcement learning algorithm called the Q-Learning algorithm for adaptive exploration based on simulated annealing (AE-SAQL). We apply the algorithm to the agent path planning problem, improve the setting of the reward function and add the feedback information of the environment. By simulating the Metropolis criterion in the annealing algorithm and adding an adaptive adjustment mechanism, the agent fully explores the environment and makes full use of the environmental information, solving the exploration-utilization dilemma during the algorithm and finally enabling the agent to reach the target location safely. Compared with the standard Q-Learning algorithm and SARSA algorithm, AE-SAQL achieves better.

Path planning is an essential algorithm in any autonomous mobile robot, including agricultural robots. One of the reinforcement learning methods that can be used for mobile robot path planning is the Q-Learning algorithm. However, the conventional Q-learning method explores all possible robot states in order to find the most optimum path. Thus, this method requires extensive computational cost especially when there are considerable grids to be computed. This study modified the original Q-Learning algorithm by removing the impassable area, so that these areas are not considered as grids to be computed. This modified Q-Learning method was simulated as path finding algorithm for autonomous mobile robot operated at the Agribusiness and Technology Park (ATP), IPB University. Two simulations were conducted to compare the original Q-Learning method and the modified Q-Learning method. The simulation results showed that the state reductions in the modified Q-Learning method can lower the computation cost to 50.71% from the computation cost of the original Q-Learning method, that is, an average computation time of 25.74s as compared to 50.75s, respectively. Both methods produce similar number of states as the robot’s optimal path, i.e. 56 states, based on the reward obtained by the robot while selecting the path. However, the modified Q-Learning algorithm is capable of finding the path to the destination point with a minimum learning rate parameter value of 0.2 when the discount factor value is 0.9.

With the accelerated development of smart cities, the concept of a “smart industrial park” in which unmanned ground vehicles (UGVs) have wide application has entered the industrial field of vision. When faced with multiple tasks and heterogeneous tasks, the task execution efficiency of a single UGV is inefficient, thus the task planning research under multi-UGV cooperation has become more urgent. In this paper, under the anti-collision cooperation mechanism for multi-UGV path planning, an improved algorithm with optimized-weighted-speedy Q-learning (OWS Q-learning) is proposed. The slow convergence speed of the Q-learning algorithm is overcome to a certain extent by changing the update mode of the Q function. By improving the selection mode of learning rate and the selection strategy of action, the relationship between exploration and utilization is balanced, and the learning efficiency of multi-agent in complex environments is improved. The simulation experiments in static environment show that the designed anti-collision coordination mechanism effectively solves the coordination problem of multiple UGVs in the same scenario. In the same experimental scenario, compared with the Q-learning algorithm and other reinforcement learning algorithms, only the OWS Q-learning algorithm achieves the convergence effect, and the OWS Q-learning algorithm has the shortest collision-free path for UGVS and the least time to complete the planning. Compared with the Q-learning algorithm, the calculation time of the OWS Q-learning algorithm in the three experimental scenarios is improved by 53.93%, 67.21%, and 53.53%, respectively. This effectively improves the intelligent development of UGV in smart parks.

The Q-Learning (QL) algorithm is widely used for path planning. As the scene in which the mobile robot is located becomes complex, the algorithm suffers from the limitations of low convergence speed and long exploration paths. Therefore, a Max Reward-Q-learning (MR-QL) path planning algorithm based on maximum reward is proposed for complex unknown scenarios. The original algorithm’s discrete reward function and action selection strategy are improved, and a new reward function is designed to dynamically adjust the reward mechanism to heuristically guide the robot motion. The action selection strategy is also optimized to avoid invalid exploration and improve the algorithm’s convergence. Finally, three experimental environments with different complexity are constructed to demonstrate the feasibility of the proposed algorithm. The simulation results show that the MR-QL algorithm is about 50% of the original algorithm in terms of exploration step length and training time, and the convergence speed of the algorithm is better than the original algorithm.

We present path-planning techniques for a multiple mobile robot system. The mobile robot has the shape of a cylinder, and its diameter, height, and weight are 8 cm, 15 cm, and 1.5 kg, respectively. The controller of the mobile robot is an MCS-51 chip, and it acquires detection signals from sensors through I/O pins. It receives commands from the supervising computer via a wireless RF interface, and transmits the status of the robots to the supervising computer via a wireless RF interface. The mobile robot system is a module-based system, and contains a controller module (including two DC motors and drivers), an obstacle detection module, a voice module, a wireless RF module, an encoder module, and a compass detection module. We propose an evaluation method to arrange the position of the multiple mobile robot system, and develop a path-planning interface on the supervising computer. In the experimental results, the mobile robots were able to receive commands from the supervising computer, and to move their next positions according to the proposed method.

In response to the problem of low efficiency and weak obstacle avoidance ability in finding the optimal or suboptimal path for 3D path planning of drones in complex dynamic environments, this paper uses Q-learning algorithm to complete the 3D path planning of drones, aiming to improve their path planning and obstacle avoidance capabilities. Firstly, by constructing a three-dimensional gridded environment model, the system calculates the reward for each state under the influence of natural environment and obstacles, and then guides the drone to avoid obstacles and find the optimal path. The system uses the ε - greedy strategy for exploration and learning, optimizing decisions by continuously updating the Q-table value table. The experimental results show that the drone has a success rate of 93.3% in obstacle avoidance in complex and multi obstacle scenes. Moreover, in terms of average path length, the Q-learning algorithm has shortened it by approximately 20.00%, 11.45%, and 40.39% compared to ant colony algorithm, A* algorithm, and RRT algorithm, respectively. In dynamic wind speed environments, the Q-learning algorithm reduces the path length by about 4% to 11% compared to other algorithms, further demonstrating its effectiveness and advantages in complex environments.

The construction of a smart autonomous mobile robot system that can operate in an anonymous environment can be difficult. Building an autonomous mobile robot requires collecting data from sensors and cameras in order to recognize the surrounding environment, get knowledge of existing landmarks, and figure out its position and orientation. Many research studies have been carried out in recent years providing algorithms for achieving autonomous motion of mobile robot systems and object detection. The most common platform utilized for robot motion design is ROS (Robot Operating System) and for object detection YOLO (You Only Look Once) is a common choice. Additionally, the autonomous system needs motion planning in a systematic way to avoid obstacles by using LIDAR or other sensing methods generating the feedback signals required for control of motion around occupied spaces. One of the most effective and versatile feedback systems for mobile robot control can be constructed using visual feedback, which relies on a high-resolution camera that processes the image in real-time and the result is fed back to the control system for decision making. Visual feedback can be employed alone without sensors for feedback control of autonomous mobile robots, and it can be used for multiple tasks like path planning, obstacle avoidance, object detection, object recognition in addition to motion control. In this research, results of design and construction of an autonomous mobile rescue robot for finding victims through video data are presented. It is aimed to find victims via applying video as a data collection and feedback generation tool in the autonomous mobile robot control system. The visual feedback and YOLO v3 are used not only for finding victims but also for motion control. Our autonomous mobile robot is designed to move smoothly to the victim's direction and avoids obstacles by using ROS Global Planner. The robot is tested in multiple scenarios in different locations, various obstacle and victim numbers, and satisfactory results with good performances are obtained.

This manuscript looks into details into the hardware implementations of an active stereo vision path planning navigation for a mobile robotics system. This mobile robotics system has been designated as the (KSU-IMR), as it a test-bed for research within the area of mobile robotics system navigation through the use of an active stereo vision for planning a path within an unstructured environment at King Saud University in KSA. The system is making use of fast image processing tools, as in addition to fast algorithms for processing the dynamic images. The approach followed here is to employ the Epipolar Geometry to compute the 3-D details of the environment where the mobile is moving. The algorithm mixes stereo vision, occupancy grid mapping, and potential field path planning procedures to form a robu

PurposeThe purpose of this paper is to put forward a rolling assistant robot with two rolling modes, and the multi-mode rolling motion strategy with path planning algorithm, which is suitable to this multi-mode mobile robot, is proposed based on chessboard-shaped grid division (CGD).Design/methodology/approachBased on the kinematic analysis and motion properties of the mobile parallel robot, the motion strategy based on CGD path planning algorithm of a mobile robot with two rolling modes moving to a target position is divided into two parts, which are local self-motion planning and global path planning. In the first part, the mobile parallel robot can move by switching and combining the two rolling modes; and in the second part, the specific algorithm of the global path planning is proposed according to the CGD of the moving ground.FindingsThe assistant robot, which is a novel 4-RSR mobile parallel robot (where R denotes a revolute joint and S denotes a spherical joint) integrating operation and rolling locomotion (Watt linkage rolling mode and 6R linkage rolling mode), can work as a moving spotlight or worktable. A series of simulation and prototype experiment results are presented to verify the CGD path planning strategy of the robot, and the performance of the path planning experiments in simulations and practices shows the validation of the path planning analysis.Originality/valueThe work presented in this paper is a further exploration to apply parallel mechanisms with two rolling modes to the field of assistant rolling robots by proposing the CGD path planning strategy. It is also a new attempt to use the specific path planning algorithm in the field of mobile robots for operating tasks.

The use of Mobile Robots in diverse sectors are increasing day by day. Nowadays, mobile robots are widely used in industrial sectors, but real-time path planning and collision free path tracking raises more challenging issues. A good path planning process for these mobile robots is needed in order for them to perform better. In path planning problems, the $A^{\ast}$ algorithm has been widely investigated and applied, but the cost and efficiency of the path is not completely taken into consideration. This algorithm has been tweaked and improved in a number of ways to facilitate path planning. Taking into account the variable-step-length $A^{\ast}$ search algorithm, which can take steps longer than one. This method uses a fixed step length and produces a better path than the $A^{\ast}$ Search algorithm. However, if the step length can be modified in the process as required, a more optimal direction can be obtained. As a result, this paper proposes an improved updated variable-step-length based $A^{\ast}$ algorithm that can update or change step length. This algorithm will efficiently find a path with a shorter length. In order to check the feasibility and efficiency of the proposed system, simulation and practical experiments are carried out. The findings show that the improved $A^{\ast}$ algorithm will increase the planned route's safety and efficiency while also reducing the robot's movement time in difficult terrain.

With the development of science and technology, a new handling method has emerged in the industrial automation industry - handling robots. Handling robot vehicles are called AGVs (Automatic Guided Vehicles), which are multi-wheel drive robots. Produced by the interdisciplinary application, it integrates automatic control, mechanical devices, image transmission and processing, computer communication, electronic information and other disciplines and technologies. It is widely used in various aspects. The path planning and obstacle avoidance problem is the leading research problem in applying the handling robot system. It is of great theoretical and practical significance to study the handling robot's path planning and obstacle avoidance algorithm. In this paper, the grid method is used to simulate the environment of the formal path, and the A* algorithm is used to improve the algorithm. We establish the global congestion model and local collision model of the robot and establish the dynamic path planning problem of the robot. At the same time, the global congestion problem is done Analysis and processing, complete the anti-collision prediction and automatic obstacle avoidance strategy analysis, and use the A* algorithm to improve the ant colony algorithm results for comparison.

Robotic path planning is a field of research which is gaining traction given the broad domains of interest to which path planning is an important systemic requirement. The aim of path planning is to optimise the efficacy of robotic movement in a defined operational environment. For example, robots have been employed in many domains including: Cleaning robots (such as vacuum cleaners), automated paint spraying robots, window cleaning robots, forest monitoring robots, and agricultural robots (often driven using satellite and geostationary positional satellite data). Additionally, mobile robotic systems have been utilised in disaster areas and locations hazardous to humans (such as war zones in mine clearance). The coverage path planning problem describes an approach which is designed to determine the path that traverses all points in a defined operational environment while avoiding static and dynamic (moving) obstacles. In this paper we present our proposed Smooth-STC model, the aim of the model being to identify an optimal path, avoid all obstacles, prevent (or at least minimise) backtracking, and maximise the coverage in any defined operational environment. The experimental results in a simulation show that, in uncertain environments, our proposed smooth STC method achieves an almost absolute coverage rate and demonstrates improvement when measured against alternative conventional algorithms.

For the issues of the ant colony algorithm (ACO) to solving the problems in mobile robot path planning, such as the slow optimization speed and the redundant paths in planning results, a high-precision improved ant colony algorithm (IPACO) with fast optimization and compound prediction mechanism is proposed. Firstly, aiming at maximizing the possibility of optimal node selection in the process of path planning, a composite optimal node prediction model is introduced to improve the state transition function. Secondly, a pheromone model with initialize the distribution and “reward or punishment” update mechanism is used to updates the global pheromone concentration directionally, which increases the pheromone concentration of excellent path nodes and the heuristic effect; Finally, a prediction-backward mechanism to deal with the “deadlock” problem in the ant colony search process is adopted in the IPACO algorithm, which enhance the success rate in the ACO algorithm path planning. Five groups of different environments are selected to compare and verify the performance of IPACO algorithm, ACO algorithm and three typical path planning algorithms. The experimental simulation results show that, compared with the ACO algorithm, the convergence speed and the planning path accuracy of the IPACO algorithm are improved by 57.69% and 12.86% respectively, and the convergence speed and the planning path accuracy are significantly improved; the optimal path length, optimization speed and stability of the IPACO algorithm are improved. Which verifies that the IPACO algorithm can effectively improve the environmental compatibility and stability of the ant colony algorithm path planning, and the effect is significantly improved.