Online Area Covering Robot in Unknown Dynamic Environments

Robots can be used for various purposes in environments humans cannot reach. Dynamic environments with static and dynamic obstacles are challenging areas in robotic implementations. In this proposed study, robotic exploration is presented in an unknown dynamic environment using Random Network Distillation(RND) algorithm, which was introduced for Atari games. RND algorithm is used here for robotic navigation depending on obstacle location and environmental features. The proposed route planning robot is implemented using the Robot Operating System(ROS) and the Gazebo simulator. In previous works, Deep Reinforcement Learning based agents have shown promising outcomes. However, the reward system needs to be more robust for this path-planning task. In order to unravel the sparse reward issue, a revised RND approach is suggested. In order to dynamically alter intrinsic and extrinsic rewards, this algorithm determines the variation of the training steps count in each training episode. Laser inputs from Gazebo are fed to the RND algorithm to get the actions. Actions will define by avoiding walls or obstacles. The robotic path and map are visualized on the Gazebo. The results show the robot's actions and combine intrinsic and extrinsic rewards from RND on the application. Further, employing the RND-trained agent learns to cover environments presented as a novel part of the research. Four simulations are conducted based on the environment to validate the algorithm, including without obstacles, static obstacles, dynamic obstacles, and both. The experimental results indicate that RND agents trained in different environments can reach the goal accurately without crashes.

Cooperative control for swarming systems based on reinforcement learning in unknown dynamic environment

In this article, an admittance control strategy for aerial manipulators is proposed to achieve contact force tracking in unknown dynamic environments. First, considering the impact of variations in unknown environments on force tracking performance, an adaptive variable stiffness feature is incorporated into an advanced admittance model. The stiffness coefficient is dynamically adjusted using position and force feedback to generate the desired reference trajectory. Second, to address the issue of reference trajectory tracking under disturbances, a pose controller composed of a disturbance observer and barrier Lyapunov function is utilized to achieve stable tracking performance. In the absence of prior knowledge of disturbances, the state variables converge to a constrained range within a finite time, without introducing excessively high control gains. Finally, the stability of the proposed strategy is rigorously analyzed via Lyapunov tools. Both simulations and real-world experimental investigations are conducted to demonstrate the feasibility of the control strategy, highlighting its robust performance in maintaining a stable contact force during interaction with unknown dynamic environments.

We present a planning algorithm for a mobile robot in unknown indoor dynamic environments. The algorithm takes into account the robot's own dynamics and the dynamic obstacles’ future behavior in a safety hierarchy. It also uses an appropriate time horizon for planning and respects the timing constraints on various modules of the planner that are imposed by interleaving of planning and execution. We provide a formulation of this safety hierarchy in a single step via a composite costmap. The planning algorithm has been implemented in Player/Stage. We have run extensive simulations and compared the performance of the full safety hierarchy with various permutations of safety levels, as well as with some baseline algorithms over three different performance measures. We show that our safety hierarchy performs statistically significantly better across most performance measures and most simulation scenarios.

In this paper, we study the problem of maintaining a stable mobile robot formation, steering and localizing all robots in an unknown dynamic environment consisting of multiple periodically moving objects, without the presence of a global positioning system or a robot tracking system. We propose a distributed observer such that each agent can estimate global positions of all mobile robots and that of moving landmarks in an unknown environment. By combining the proposed distributed observer with the distributed formation control and centroid tracking control law, we show that the formation shape can be maintained by utilizing its available relative measurements and the estimated relative measurements to its neighbors, and the group's centroid follows a desired trajectory. We present stability analysis of the closed‐loop system. Finally, we validate the proposed methods in a simulation result where a group of mobile robots can maintain a robust formation and maneuver in an unknown dynamic environment.

This paper describes a reinforcement learning algorithm that allows a point robot to learn navigation strategies within initially unknown indoor environments with fixed and dynamic obstacles. The knowledge is encoded in two surfaces, called reward and penalty surfaces, that are updated either when a target is found or whenever the robot moves respectively. The proposed policy is suitable for both planning and reactive behaviour. The tests involve different kinds of obstacles: a fixed passage, a barrier, a U-shape obstacle and a simple maze. The results suggest that the model solves the goal-directed exploration problem. Thus, the robot is able to reach a desired goal, starting its movement from any position within the environment, avoiding obstacles, and following a viable trajectory. The robot may get stuck in dynamic obstacles, may depend on randomness to avoid them, and generally does not solve the goal-directed reinforcement learning problem.

As mobile robots are becoming more and more widely used, it is of great significance to design an efficient path planning method for multi-robot systems (MRS) that can adapt to complex and unknown environments. In this paper, we present a deep reinforcement learning (DRL) based method to navigate the MRS with collision avoidance in unknown dynamic environments, which consists of centralized learning and decentralized executing paradigm. The proposed policy maps original laser information into robot control commands without constructing global maps. The learned policy is tested in Gazebo environments with three robot systems, which shows the effective performance in terms of success rate, extra time rate, and formation maintenance rate.

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

A new hybrid solution is presented to improve the efficiency of intelligent warehouses with multirobot systems, where the genetic algorithm (GA) based task scheduling is combined with reinforcement learning (RL) based path planning for mobile robots. Reinforcement learning is an effective approach to search for a collision-free path in unknown dynamic environments. Genetic algorithm is a simple but splendid evolutionary search method that provides very good solutions for task allocation. In order to achieve higher efficiency of the intelligent warehouse system, we design a new solution by combining these two techniques and provide an effective and alternative way compared with other state-of-the-art methods. Simulation results demonstrate the effectiveness of the proposed approach regarding the optimization of travel time and overall efficiency of the intelligent warehouse system.

This paper investigates the development of a physical multi-robot system, where a group of intelligent robots work cooperatively to transport an object to a goal location and orientation in an unknown dynamic environment. Multi-agent technology and machine learning are integrated into the same physical platform to provide innovative capabilities for carrying out the task. First, a new multi-agent architecture is developed. Second, two methods that facilitate optimized machine learning, Reinforcement learning (RL) and genetic algorithms (GA), are integrated into the decision-making agent in the architecture. An arbitrator is incorporated into a probabilistic switching scheme for selecting the optimal strategy in a given state of the task. Finally, the robot force/motion control and local modeling are integrated into the architecture to implement a physical multi-robot system. Simulation and experimental studies are carried out to demonstrate the feasibility of the system.

PurposeThis study aims to introduce the fast reactive tree (FRT*) algorithm for enhancing replanning speed and reducing the overall cost of navigation in unknown dynamic environments.Design/methodology/approachFRT* comprises four key components: inverted tree build, convex hull construction, dead nodes inform activation and lazy-rewiring replanning. First, an initial path is found from the inverted tree where the valid structure is preserved to minimise re-exploration areas during the replanning phase. As the robot encounters environment changes, convex hulls are extracted to sparsely describe impacted areas. Next, the growth direction of the modified tree is biased by the inform activation of dead nodes to avoid unnecessary exploration. In the replanning phase, the tree structure is optimized using the proposed lazy-rewiring replanning to find a high-quality path with low computation burden.FindingsA series of comprehensive simulation experiments demonstrate that the proposed FRT* algorithm can efficiently replan short-cost feasible paths in unknown dynamic environments. The differential wheeled mobile robot with varying reference linear velocities is used to validate the effectiveness and adaptability of the proposed strategy in real word scenarios. Furthermore, ablation studies are conducted to analyze the significance of the key components of FRT*.Originality/valueThe proposed FRT* algorithm introduces a novel approach to addressing the challenges of navigation in unknown dynamic environments. This capability allows mobile robots to safely and efficiently navigate through unknown and dynamic environments, making the method highly applicable to real-world scenarios.

Reinforcement learning, as a branch of machine learning, has been gradually applied in the control field. However, in the practical application of the algorithm, the hyperparametric approach to network settings for deep reinforcement learning still follows the empirical attempts of traditional machine learning (supervised learning and unsupervised learning). This method ignores part of the information generated by agents exploring the environment contained in the updating of the reinforcement learning value function, which will affect the performance of the convergence and cumulative return of reinforcement learning. The reinforcement learning algorithm based on dynamic parameter adjustment is a new method for setting learning rate parameters of deep reinforcement learning. Based on the traditional method of setting parameters for reinforcement learning, this method analyzes the advantages of different learning rates at different stages of reinforcement learning and dynamically adjusts the learning rates in combination with the temporal-difference (TD) error values to achieve the advantages of different learning rates in different stages to improve the rationality of the algorithm in practical application. At the same time, by combining the Robbins–Monro approximation algorithm and deep reinforcement learning algorithm, it is proved that the algorithm of dynamic regulation learning rate can theoretically meet the convergence requirements of the intelligent control algorithm. In the experiment, the effect of this method is analyzed through the continuous control scenario in the standard experimental environment of ”Car-on-The-Hill” of reinforcement learning, and it is verified that the new method can achieve better results than the traditional reinforcement learning in practical application. According to the model characteristics of the deep reinforcement learning, a more suitable setting method for the learning rate of the deep reinforcement learning network proposed. At the same time, the feasibility of the method has been proved both in theory and in the application. Therefore, the method of setting the learning rate parameter is worthy of further development and research.

A general scheme of cooperative hunting for a moving target by multiple mobile robots in unknown environments is presented. To realize successful hunting, the point stabilization control algorithm is used for path planning of hunting mobile robots. Simultaneously, collision risk based obstacle avoidance scheme is used to keep away from random obstacles during hunting. The synthesized behavior is obtained by fusing avoidance behavior, encirclement behavior and pincer attack behavior. Simulated hunting experiments were conduct under different conditions in Microsoft Robotics Studio (MRS) and simulation results demonstrate the correctness and effectiveness of the proposed scheme.

Integrated sensing and communications (ISAC) has been envisioned as a pivotal technology in solving the exacerbated spectrum congestion problem. Meanwhile, intelligent reflective surface (IRS) presents a promising avenue for enhancing both communications and sensing in severe obstruction scenarios, such as dense urban area. In this paper, we propose a reinforcement learning (RL)-based IRS-assisted ISAC design that enables simultaneous multi-user downlink communications and multi-target sensing via optimizing the IRS reflection coefficients in unknown high-obstruction scenarios. In order to facilitate the IRS-assisted ISAC design, we first introduce a two-phase method that can achieve precise channel estimation with a small number of pilot signals. Subsequently, we employ the RL to co-design the dual-functional IRS, striving for an optimal performance balance between communications and sensing in unknown environments. Specifically, to address the non-convex problems arising from the RL training, we propose to design the Gramian matrix first, followed by matrix decomposition to obtain the optimal IRS reflection coefficients. Extensive simulations have demonstrated that the two-phase method can accurately estimate the channel, and the IRS-assisted ISAC designed via RL exhibits an excellent dual-functional performance in the unknown environment.

Using Deep Reinforcement Learning (DRL) algorithm to deal with autonomous driving tasks usually have unsatisfied performance due to lack of robustness and means to escape local optimum. In this article, we designs a Survival-Oriented Reinforcement Learning (SORL) model that tackle these problems by setting survival rather than maximize total reward as first priority. In SORL model, we model autonomous driving task as Constrained Markov Decision Process (CMDP) and introduce Negative-Avoidance Function to learn from previous failure. The SORL model greatly speed up the training process and improve the robustness of normal Deep Reinforcement Learning algorithm.