Explore-Bench: Data Sets, Metrics and Evaluations for Frontier-based and Deep-reinforcement-learning-based Autonomous Exploration

In this paper, we propose an off-line planning algorithm for cooperative tasks of multiple mobile robots. Sweeping means a motion that a robot covers a 2-dimensional area by its effector. Sweeping of a whole work area is fundamental and essential task of mobile robots. For efficient cooperation, setting appropriate burden onto each robot is very important because interference of robots and overlaps of their effecters make efficiency low. The cost of sweeping depends on both sweeping ability of a robot and a shape of a work area to be swept. Evaluation and distribution of the cost are most important issues. In the proposed algorithm, the cost is evaluated by means of length on which robot should move. We introduce both edges of the configuration space and Voronoi diagram so as to compute paths in the whole area. We generate the a tour for traversing all the paths by applying the algorithm of the Chinese Postman Problem. According to the cost evaluation, appropriate paths of the tour are assigned to each robot. The efficiency of the proposed algorithm is verified by simulations and an experiment.

To train a mobile robot to navigate using end-to-end approach which maps sensors data into actions, we can use deep reinforcement learning (DRL) method by providing training environments with proper reward functions. Although some studies have shown the success of DRL in navigation task for mobile robots, the method needs appropriate hyperparameter settings such as the environment's timestep size and the robot's velocity range to produce a good navigation policy. The previous existing DRL framework has proposed the use of odometry sensor to generate dynamic timestep size in the environment to solve the mismatch problem between the timestep size and the robot's velocity. However, the framework lacks a procedure for checking terminal conditions which may occur during action executions resulting inconsistency in the environment and poor navigation policies. In the case of navigation task, terminal conditions may happen when the robot achieves the navigation goal position or collides with obstacles while performing an action in one timestep. To cope with this problem, we propose a state transition checking method in the DRL environment which is specific for navigation task that leverages odometry and laser sensor to ensure that the environment follows Markov Decision Process with dynamic timestep size. We also introduce a velocity increment scheduling to stabilize the mobile robot during training. Our experiment results show that state transition checking along with the velocity increment scheduling are able to make the robot navigate faster with higher success rate compared to other existing DRL frameworks.

Mobile Additive Manufacturing (MAM) systems are transforming large-scale fabrication across various industries, particularly in building and construction. This review explores recent advancements and ongoing challenges in deploying mobile robots within dynamic additive manufacturing (AM) environments. A primary focus is placed on mobile robots' path planning and real-time navigation methods, identified as critical knowledge gaps that impact the accuracy of printing trajectories. AI-driven techniques, such as deep learning and reinforcement learning, are presented as promising solutions to these challenges, offering improvements in trajectory optimisation, obstacle avoidance, and multi-robot cooperation. However, significant obstacles remain, particularly in scaling up MAM operations while maintaining both precision and efficiency. This review provides analysis of the current state of mobile robotic AM, outlines potential pathways for future research, and underscores the alignment of these technologies with Industry 4.0 objectives, emphasising the ongoing need for innovation to unlock the full potential of mobile robotics in large-scale manufacturing.

Aiming at the problems existing in traditional upper limb rehabilitation training, such as low rehabilitation efficiency, single and passive training methods, etc. Deep reinforcement learning can provide a better trajectory planning scheme for the upper limb rehabilitation robot working in multi-dynamic environment. It can make more efficient and active rehabilitation training plan for rehabilitation patients. In the actual rehabilitation training, it is necessary to smooth the motion track without mutation and avoid the singular region. In this paper, DDPG deep learning reinforcement learning algorithm is used to propose an improved reward function to provide global guidance for exploring optimal trajectory planning. At the same time, the nonlinear function of the feedback regulation mechanism is constructed by dividing the working interval property to accelerate the exploration process and avoid the singular region. Experiments show that the convergence of the reward value of the improved model is significantly improved compared with the traditional model. The method proposed in this paper provides a good theoretical basis for the research of active rehabilitation control system of upper limb rehabilitation robot.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Autonomous exploration in expansive and complicated environments poses a significant challenge. When the dimensions of the environment expand, exploration algorithms encounter substantial overhead, which can overpower the computational capacity of mobile platforms. In this paper, we propose a novel 3D LiDAR-based end-to-end autonomous exploration network architecture, which allows mobile robots to learn to explore autonomously in expansive environments through deep reinforcement learning. Specifically, we utilize both scans from the LiDAR sensor and maps obtained by SLAM as exploration information to predict the robot's linear and angular actions simultaneously. Furthermore, in order to enhance exploration capability, intrinsic rewards are also used during training. Compared to the existing methods, our proposed approach demonstrates improved learning efficiency and adaptability for various environments. Moreover, the proposed method can complete exploration in unknown environments with a shorter trajectory length than state-of-the-art methods. Additionally, experiments are conducted on the physical robot. which indicates that the trained network can be seamlessly transferred from the simulation to the real world.

We consider an autonomous exploration problem in which a range-sensing mobile robot is tasked with accurately mapping the landmarks in an a priori unknown environment efficiently in real-time; it must choose sensing actions that both curb localization uncertainty and achieve information gain. For this problem, belief space planning methods that forward- simulate robot sensing and estimation may often fail in real-time implementation, scaling poorly with increasing size of the state, belief and action spaces. We propose a novel approach that uses graph neural networks (GNNs) in conjunction with deep reinforcement learning (DRL), enabling decision-making over graphs containing exploration information to predict a robot's optimal sensing action in belief space. The policy, which is trained in different random environments without human intervention, offers a real-time, scalable decision-making process whose high-performance exploratory sensing actions yield accurate maps and high rates of information gain.

In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar performance as genie-aided methods without using the future information.

This paper proposes an intelligent game confrontation model for a wargame based on imitation learning and deep reinforcement learning, given light of the supremacy of reinforcement learning for training in complex environments. In the context of simulating the battle between the red team and the blue team, a pre-training model based on expert empirical data is produced using imitation learning, and a TD3 algorithm based on the attention mechanism is further designed to build an experience pool using priority experience replay. Finally, the model is enhanced using the self-play approach to increase its training efficiency. Experiments conducted after training demonstrate that the model has a superior training impact, and the winning rate in simulation training is enhanced by 8% compared to the original model.

Autonomous exploration and mapping in unknown environments is a critical capability for robots. Existing exploration techniques (e.g., heuristic-based and learning-based methods) do not consider the regional legacy issues, i.e., the great impact of smaller unexplored regions on the whole exploration process, which results in a dramatic reduction in their later exploration efficiency. To this end, this paper proposes a Local-and-Global Strategy (LAGS) algorithm that combines a local exploration strategy with a global perception strategy, which considers and solves the regional legacy issues in the autonomous exploration process to improve exploration efficiency. Additionally, we further integrate Gaussian process regression (GPR), Bayesian optimization (BO) sampling, and deep reinforcement learning (DRL) models to efficiently explore unknown environments while ensuring the robot’s safety. Extensive experiments show that the proposed method could explore unknown environments with shorter paths, higher efficiencies, and stronger adaptability on different unknown maps with different layouts and sizes.

More and more researchers are trying to use deep reinforcement learning (DRL) for mobile robot tasks due to its powerful inference capability. However, deep reinforcement learning requires a large amount of data for DRL training in the pre-experimental stage, which hinders the application of the algorithm. On the other hand, the inconsistency between the ROS data interface and DRL GYM-Like data interface leads to a high cost of migration of the algorithm verification. This paper proposes a fast simulator generation method using linear approximate kinematics model and bake-based lidar rendering methods to generate a fast approximate simulator used in the pre-experiment stage to solve the problem of data cost. At the same time, an experimental system design scheme that converts the ROS interface into a GYM-like interface is also proposed to simplify the deployment process of deep reinforcement learning. We evaluate our proposed method on collision avoidance tasks in a variety of kinematics models and lidar scenarios. Our Method achieves about 14.2 times kinematics simulation speedup and 2.56 times lidar rendering speedup. We open-sourced our simulation environment and robot system software at https://github.com/efc-robot/MultiVehicleEnv and https://github.com/efc-robot/NICS_MultiRobot_Platform

Deep reinforcement learning methods have been applied to mobile robot navigation to find the optimal path to the target. The rewards are usually given when the task is completed, which may lead to the local optima during the training procedure. It seriously affects the training efficiency and navigation performance of the mobile robot. To this end, this paper proposes an intrinsic reward mechanism with intrinsic curiosity module and randomness enhanced module, combining the TD3 (twin-delayed deep deterministic policy gradient) reinforcement learning algorithm for mobile robot navigation. It effectively resolves the issue of slow convergence caused by sparse rewards in continuous action spaces. It also encourages mobile robots to explore unknown areas and reduces the occurrence of local optima. The experimental results show that the proposed navigation method significantly improves the training efficiency of mobile robots. Out of 1000 test episodes, only 3 exceeded the maximum step limit. This approach significantly reduces the occurrence of local optima. Furthermore, it increases the success rate to an impressive 83.5%, outperforms the existing navigation methods.

Multiple mobile robots are a practical platform for realizing the concept of distributed manipulation. New approach for manipulating large number of objects using distributive actuators or agent has been suggested. Multiple mobile robots provide the ability to conduct actions simultaneously from various locations and directions. Using two mobile robots and flexible tool such as a piece of string or rope, a large number of objects that are placed at sparse locations can be sorted and grouped by single action. The Voronoi separation algorithm is proposed as a path generation method for preplanning cooperative actions of multiple robots. The sorting task is conducted on-line using fuzzy logic controller with preplanned subgoals. The proposed control framework was verified by an experiment using two autonomous mobile robots with a flexible tool attachment of fixed length.

In this paper, a novel DRL algorithm based on a DQN is proposed for multiple mobile robots to find optimized paths. The multiple robots’ states are the inputs of the DQN. The DQN estimates the Q-value of the agents’ actions. After selecting the action with the maximum Q-value, the multiple robots’ actions are calculated and sent to them. Then, the robots will explore the area and detect the obstacles. In the area, there are static obstacles. The robots should detect the static obstacles using a LiDAR sensor. The other moving robots are recognized as dynamic obstacles that need to be avoided. The robots will give feedback on the reward and the robots’ new states. A positive reward will be given when a robot successfully arrives at its goal point. If it is in a free space, zero reward will be given. If the robot collides with a static obstacle or other robots or reaches its start point, it will receive a negative reward. Multiple robots explore safe paths to the goals at the same time, in order to improve learning efficiency. If a robot collides with an obstacle or other robots, it will stop and wait for the other robots to complete their exploration tasks. The episode will end when all robots find safe paths to reach their goals or when all of them have collisions. This collaborative behavior can reduce the risk of collisions between robots, enhance overall efficiency, and help avoid multiple robots attempting to navigate through the same unsafe path simultaneously. Moreover, storage space is used to store the optimal safe paths of all robots. Finally, the multi-robots will learn the policy to find the optimized paths to go to the goal points. The goal of the simulations and experiment is to make multiple robots efficiently and safely move to their goal points.

In order to make the robot obtain the optimal action directly from the original visual perception without any hand-crafted features and features matching, a novel end-to-end path planning method-mobile robot path planning using deep reinforcement learning is proposed. Firstly, a deep Q-network (DQN) is designed and trained to approximate the mobile robot state-action value function. Then, the Q value corresponding to each possible mobile robot action (i.e., turn left, turn right, forward) is determined by the well trained DQN, here, the input of the DQN is the original RGB image (image pixels) captured from the environment without any hand-crafted features and features matching; Finally, the current optimal mobile robot action is selected by the action selection strategy. Mobile robot reach to the goal point while avoiding obstacles ultimately. 30 times path planning experiments are conducted in the seekavoid_arena_01 environment on DeepMind Lab platform. The experimental results show that our deep reinforcement learning based robot path planning method is an effective end-to-end mobile robot path planning method.