Cooperative control for swarming systems based on reinforcement learning in unknown dynamic environment

Autonomous area covering robots are being increasingly deployed in residential and commercial settings for a variety of purposes. These robots usually employ universal area covering algorithms to cover indoor environments. The performance of such algorithms heavily depends on room geometry as well as obstacle location, and often suffers from significant overlap leading to inordinately long coverage time, especially in realistic unknown environments with dynamic obstacles. Hence, deploying smarter algorithms that adapt to the environment can improve the performance significantly. In this study, we explore deep reinforcement learning (RL) algorithms for efficient coverage in unknown environments with multiple dynamic obstacles. Through experiments in grid-based environments and Gazebo simulator, we demonstrate the superior performance of RL based coverage algorithms in environments with dynamic obstacles. The performance of RL based algorithm is compared with the BA* algorithm with dynamic re-planning to demonstrate the advantages of the former over one-shot algorithms. Further, by employing transfer learning the trained RL agent learns to cover unseen unknown environments with minimal additional sample complexity. Importantly, we show that RL agents trained in smaller environments can be deployed for coverage in larger unknown environments with marginal additional sample complexity.

In this article, an admittance control strategy for aerial manipulators is proposed to achieve contact force tracking in unknown dynamic environments. First, considering the impact of variations in unknown environments on force tracking performance, an adaptive variable stiffness feature is incorporated into an advanced admittance model. The stiffness coefficient is dynamically adjusted using position and force feedback to generate the desired reference trajectory. Second, to address the issue of reference trajectory tracking under disturbances, a pose controller composed of a disturbance observer and barrier Lyapunov function is utilized to achieve stable tracking performance. In the absence of prior knowledge of disturbances, the state variables converge to a constrained range within a finite time, without introducing excessively high control gains. Finally, the stability of the proposed strategy is rigorously analyzed via Lyapunov tools. Both simulations and real-world experimental investigations are conducted to demonstrate the feasibility of the control strategy, highlighting its robust performance in maintaining a stable contact force during interaction with unknown dynamic environments.

In this paper, we study the problem of maintaining a stable mobile robot formation, steering and localizing all robots in an unknown dynamic environment consisting of multiple periodically moving objects, without the presence of a global positioning system or a robot tracking system. We propose a distributed observer such that each agent can estimate global positions of all mobile robots and that of moving landmarks in an unknown environment. By combining the proposed distributed observer with the distributed formation control and centroid tracking control law, we show that the formation shape can be maintained by utilizing its available relative measurements and the estimated relative measurements to its neighbors, and the group's centroid follows a desired trajectory. We present stability analysis of the closed‐loop system. Finally, we validate the proposed methods in a simulation result where a group of mobile robots can maintain a robust formation and maneuver in an unknown dynamic environment.

PurposeThis study aims to introduce the fast reactive tree (FRT*) algorithm for enhancing replanning speed and reducing the overall cost of navigation in unknown dynamic environments.Design/methodology/approachFRT* comprises four key components: inverted tree build, convex hull construction, dead nodes inform activation and lazy-rewiring replanning. First, an initial path is found from the inverted tree where the valid structure is preserved to minimise re-exploration areas during the replanning phase. As the robot encounters environment changes, convex hulls are extracted to sparsely describe impacted areas. Next, the growth direction of the modified tree is biased by the inform activation of dead nodes to avoid unnecessary exploration. In the replanning phase, the tree structure is optimized using the proposed lazy-rewiring replanning to find a high-quality path with low computation burden.FindingsA series of comprehensive simulation experiments demonstrate that the proposed FRT* algorithm can efficiently replan short-cost feasible paths in unknown dynamic environments. The differential wheeled mobile robot with varying reference linear velocities is used to validate the effectiveness and adaptability of the proposed strategy in real word scenarios. Furthermore, ablation studies are conducted to analyze the significance of the key components of FRT*.Originality/valueThe proposed FRT* algorithm introduces a novel approach to addressing the challenges of navigation in unknown dynamic environments. This capability allows mobile robots to safely and efficiently navigate through unknown and dynamic environments, making the method highly applicable to real-world scenarios.

A general scheme of cooperative hunting for a moving target by multiple mobile robots in unknown environments is presented. To realize successful hunting, the point stabilization control algorithm is used for path planning of hunting mobile robots. Simultaneously, collision risk based obstacle avoidance scheme is used to keep away from random obstacles during hunting. The synthesized behavior is obtained by fusing avoidance behavior, encirclement behavior and pincer attack behavior. Simulated hunting experiments were conduct under different conditions in Microsoft Robotics Studio (MRS) and simulation results demonstrate the correctness and effectiveness of the proposed scheme.

We present a planning algorithm for a mobile robot in unknown indoor dynamic environments. The algorithm takes into account the robot's own dynamics and the dynamic obstacles’ future behavior in a safety hierarchy. It also uses an appropriate time horizon for planning and respects the timing constraints on various modules of the planner that are imposed by interleaving of planning and execution. We provide a formulation of this safety hierarchy in a single step via a composite costmap. The planning algorithm has been implemented in Player/Stage. We have run extensive simulations and compared the performance of the full safety hierarchy with various permutations of safety levels, as well as with some baseline algorithms over three different performance measures. We show that our safety hierarchy performs statistically significantly better across most performance measures and most simulation scenarios.

As mobile robots are becoming more and more widely used, it is of great significance to design an efficient path planning method for multi-robot systems (MRS) that can adapt to complex and unknown environments. In this paper, we present a deep reinforcement learning (DRL) based method to navigate the MRS with collision avoidance in unknown dynamic environments, which consists of centralized learning and decentralized executing paradigm. The proposed policy maps original laser information into robot control commands without constructing global maps. The learned policy is tested in Gazebo environments with three robot systems, which shows the effective performance in terms of success rate, extra time rate, and formation maintenance rate.

PurposeThe purpose of this paper is to solve the problem that mobile robots are still based on reactive collision avoidance in unknown dynamic environments leading to a lack of interaction with obstacles and limiting the comprehensive performance of mobile robots. A dynamic window approach with multiple interaction strategies (DWA-MIS) is proposed to solve this problem.Design/methodology/approachThe algorithm firstly classifies the moving obstacle movement intention, based on which a rule function is designed to incorporate positive incentives to motivate the robot to make correct avoidance actions. Then, the evaluation mechanism is improved by considering the time cost and future information of the environment to increase the motion states. Finally, the optimal objective function is designed based on genetic algorithm to adapt to different environments with time-varying multiparameter optimization.FindingsFaced with obstacles in different states, the mobile robot can choose a suitable interaction strategy, which solves the limitations of the original DWA evaluation function and avoids the defects of reactive collision avoidance. Simulation results show that the algorithm can efficiently adapt to unknown dynamic environments, has less path length and iterations and has a high comprehensive performance.Originality/valueA DWA-MIS is proposed, which increases the interaction capability between mobile robots and obstacles by improving the evaluation function mechanism and broadens the navigation strategy of DWA at a lower computational cost. After real machine verification, the algorithm has a high comprehensive performance based on real environment and provides a new idea for local path planning methods.

A new hybrid solution is presented to improve the efficiency of intelligent warehouses with multirobot systems, where the genetic algorithm (GA) based task scheduling is combined with reinforcement learning (RL) based path planning for mobile robots. Reinforcement learning is an effective approach to search for a collision-free path in unknown dynamic environments. Genetic algorithm is a simple but splendid evolutionary search method that provides very good solutions for task allocation. In order to achieve higher efficiency of the intelligent warehouse system, we design a new solution by combining these two techniques and provide an effective and alternative way compared with other state-of-the-art methods. Simulation results demonstrate the effectiveness of the proposed approach regarding the optimization of travel time and overall efficiency of the intelligent warehouse system.

The online cooperative path planning problem is discussed for multi-quadrotor maneuvering in an unknown dynamic environment. Based on the related basic concepts, typical three-dimensional obstacle models, such as spherical and cubic, and their collision checking criteria are presented in this article. An improved rapidly exploring random tree (RRT) algorithm with goal bias and greed property is proposed based on the heuristic search strategy to overcome the shortcomings of the classical RRT algorithm. Not only are the kinematic constraints of the quadrotor established but the time and space coordination strategy matching with the improved RRT algorithm is also presented in this article. Furthermore, a novel online collision avoidance strategy according to the partial information of the surrounding environment is proposed. On the basis of the above work, a distributed online path planning strategy is proposed to obtain the feasible path for each quadrotor. Numerical simulation results show that the improved RRT algorithm has better search efficiency than the classical RRT algorithm. And the satisfactory path planning and path tracking results prove that the above model and related planning strategies are reasonable and effective.

Mobile robots generally face formidable challenges in unknown dynamic environments, particularly in the aspects of rapidly searching feasible and high-quality solutions. In this article, Gaussian-heuristic fast marching tree (GH-FMT), including heuristic expansion and sampling, is proposed to provide high-quality solutions while simultaneously improving efficiency of planning and replanning in unknown dynamic environments. Specifically, an adaptive expansion region is defined using a boundary-based Gaussian distribution, minimizing potential redundant collision checking and supports effective and comprehensive exploration of the environments. A hybrid incremental search strategy is designed to reduce sample density and prioritized exploration of the search tree in promising directions using a heuristic single sampling method. Moreover, a hybrid optimization strategy is employed under time constraints to identify potential nodes, enabling efficient convergence toward high-quality solutions. Finally, through a series of challenging simulation scenarios and real-world experimental investigations, and by benchmarking against current-leading variants in the sampling-based planning class, the proposed GH-FMT demonstrated favorable advantages in flexibility, safety, and rapidity. It is shown that the search time is reduced on average by 15.8% and the path cost is reduced by on average 13.5%.

This paper investigates the development of a physical multi-robot system, where a group of intelligent robots work cooperatively to transport an object to a goal location and orientation in an unknown dynamic environment. Multi-agent technology and machine learning are integrated into the same physical platform to provide innovative capabilities for carrying out the task. First, a new multi-agent architecture is developed. Second, two methods that facilitate optimized machine learning, Reinforcement learning (RL) and genetic algorithms (GA), are integrated into the decision-making agent in the architecture. An arbitrator is incorporated into a probabilistic switching scheme for selecting the optimal strategy in a given state of the task. Finally, the robot force/motion control and local modeling are integrated into the architecture to implement a physical multi-robot system. Simulation and experimental studies are carried out to demonstrate the feasibility of the system.

This paper deals with the search-and-rescue tasks of a mobile robot with multiple interesting targets in an unknown dynamic environment. The problem is challenging because the mobile robot needs to search for multiple targets while avoiding obstacles simultaneously. To ensure that the mobile robot avoids obstacles properly, we propose a mixed-strategy Nash equilibrium based Dyna-Q (MNDQ) algorithm. First, a multi-objective layered structure is introduced to simplify the representation of multiple objectives and reduce computational complexity. This structure divides the overall task into subtasks, including searching for targets and avoiding obstacles. Second, a risk-monitoring mechanism is proposed based on the relative positions of dynamic risks. This mechanism helps the robot avoid potential collisions and unnecessary detours. Then, to improve sampling efficiency, MNDQ is presented, which combines Dyna-Q and mixed-strategy Nash equilibrium. By using mixed-strategy Nash equilibrium, the agent makes decisions in the form of probabilities, maximizing the expected rewards and improving the overall performance of the Dyna-Q algorithm. Furthermore, a series of simulations are conducted to verify the effectiveness of the proposed method. The results show that MNDQ performs well and exhibits robustness, providing a competitive solution for future autonomous robot navigation tasks.

In this paper, we address the autonomous control of a 3D snake-like robot through the use of reinforcement learning, and we apply it in a dynamic environment. In general, snake-like robots have high mobility that is realized by many degrees of freedom, and they can move over dynamically shifting environments such as rubble. However, this freedom and flexibility leads to a state explosion problem, and the complexity of the dynamic environment leads to incomplete learning by the robot. To solve these problems, we focus on the properties of the actual operating environment and the dynamics of a mechanical body. We design the body of the robot so that it can abstract small, but necessary state-action space by utilizing these properties, and we make it possible to apply reinforcement learning. To demonstrate the effectiveness of the proposed snake-like robot, we conduct experiments; from the experimental results we conclude that learning is completed within a reasonable time, and that effective behaviors for the robot to adapt itself to an unknown 3D dynamic environment were realized.

UAVs are widely used in fields such as industrial inspection, disaster relief and package transportation. Therefore, it has great significance to realize safe and smooth path planning in an unknown dynamic environment. For this reason, this paper designs a path planning system suitable for unknown environment based on the hybrid A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">*</sup> algorithm by the following three modules, the planning module, the navigation module and the control module to achieve the above-mentioned functions. We built a drone model and experimental environment in the gazebo simulator, verified the performance of the system with smoothness as an indicator.