High-speed autonomous flight and obstacle avoidance for quadrotors in unknown dynamic environments based on imitation learning

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

In this article, an admittance control strategy for aerial manipulators is proposed to achieve contact force tracking in unknown dynamic environments. First, considering the impact of variations in unknown environments on force tracking performance, an adaptive variable stiffness feature is incorporated into an advanced admittance model. The stiffness coefficient is dynamically adjusted using position and force feedback to generate the desired reference trajectory. Second, to address the issue of reference trajectory tracking under disturbances, a pose controller composed of a disturbance observer and barrier Lyapunov function is utilized to achieve stable tracking performance. In the absence of prior knowledge of disturbances, the state variables converge to a constrained range within a finite time, without introducing excessively high control gains. Finally, the stability of the proposed strategy is rigorously analyzed via Lyapunov tools. Both simulations and real-world experimental investigations are conducted to demonstrate the feasibility of the control strategy, highlighting its robust performance in maintaining a stable contact force during interaction with unknown dynamic environments.

Artificial intelligence (AI), imitation learning, big data, cloud and distributed computing, robotics cells, and information communication technology, are some of the key tools and elements of the future digital and smart manufacturing facility. There are a number of challenges that digital and smart manufacturing is facing, especially with the complication of AI (i.e., machine, deep and cognitive learning) algorithms, great amount of data to process, and essential complex coding required, which makes immediate changes needed in manufacturing facilities not straightforward. This is notable in small manufacturing cells which is an integrated part of future smart factories such manufacturing facilities are usually needed some annual and regular updates to meet the update in the design specifications of next generation of products. Imitation learning is offering a great opportunity to overcome these challenges and simplify such complications, where human skills, ability to perform specific tasks, knowledge, and talent could be transferred. This is conveying the knowledge, and skills transfer using imitation learning. However, smart manufacturing and industrial revolution needs robotics cells that has skills beyond this, especially when it comes to process optimisation. Therefore, deep imitation learning could come in to help in the development of self-learning robotic systems and cells. Of course with the powerful tools such as distributed computing, blockchain, cloud computing, edge computing, and 5G the collaboration between such self-learning robotic cells will be possible. This will certainly not eliminate human existence but will enhance the manufacturing environment. This paper is focused on presenting the outcomes of CAD simulation and modelling phase of the ongoing research programme that focused on developing a self-learning robotic system using imitation learning. CAD tools have been used and some initial results is presented. Further work is still undertaken, and this will focus on learning from more than one expert, optimisation, impact of dynamic manufacturing environment.

In this paper, we study the problem of maintaining a stable mobile robot formation, steering and localizing all robots in an unknown dynamic environment consisting of multiple periodically moving objects, without the presence of a global positioning system or a robot tracking system. We propose a distributed observer such that each agent can estimate global positions of all mobile robots and that of moving landmarks in an unknown environment. By combining the proposed distributed observer with the distributed formation control and centroid tracking control law, we show that the formation shape can be maintained by utilizing its available relative measurements and the estimated relative measurements to its neighbors, and the group's centroid follows a desired trajectory. We present stability analysis of the closed‐loop system. Finally, we validate the proposed methods in a simulation result where a group of mobile robots can maintain a robust formation and maneuver in an unknown dynamic environment.

This paper addresses the problems related to the mapless navigation control of wheeled mobile robots based on deep learning technology. The traditional navigation control framework is based on a global map of the environment, and its navigation performance depends on the quality of the global map. In this paper, we proposes a mapless Light Detection and Ranging (LiDAR) navigation control method for wheeled mobile robots based on deep imitation learning. The proposed method is a data-driven control method that directly uses LiDAR sensors and relative target position for mobile robot navigation control. A deep convolutional neural network (CNN) model is proposed to predict motion control commands of the mobile robot without the requirement of the global map to achieve navigation control of the mobile robot in unknown environments. While collecting the training dataset, we manipulated the mobile robot to avoid obstacles through manual control and recorded the raw data of the LiDAR sensor, the relative target position, and the corresponding motion control commands. Next, we applied a data augmentation method on the recorded samples to increase the number of training samples in the dataset. In the network model design, the proposed CNN model consists of a LiDAR CNN module to extract LiDAR features and a motion prediction module to predict the motion behavior of the robot. In the model training phase, the proposed CNN model learns the mapping between the input sensor data and the desired motion behavior through end-to-end imitation learning. Experimental results show that the proposed mapless LiDAR navigation control method can safely navigate the mobile robot in four unseen environments with an average success rate of 75%. Therefore, the proposed mapless LiDAR navigation control system is effective for robot navigation control in an unknown environment without the global map.

Cooperative control for swarming systems based on reinforcement learning in unknown dynamic environment

Autonomous area covering robots are being increasingly deployed in residential and commercial settings for a variety of purposes. These robots usually employ universal area covering algorithms to cover indoor environments. The performance of such algorithms heavily depends on room geometry as well as obstacle location, and often suffers from significant overlap leading to inordinately long coverage time, especially in realistic unknown environments with dynamic obstacles. Hence, deploying smarter algorithms that adapt to the environment can improve the performance significantly. In this study, we explore deep reinforcement learning (RL) algorithms for efficient coverage in unknown environments with multiple dynamic obstacles. Through experiments in grid-based environments and Gazebo simulator, we demonstrate the superior performance of RL based coverage algorithms in environments with dynamic obstacles. The performance of RL based algorithm is compared with the BA* algorithm with dynamic re-planning to demonstrate the advantages of the former over one-shot algorithms. Further, by employing transfer learning the trained RL agent learns to cover unseen unknown environments with minimal additional sample complexity. Importantly, we show that RL agents trained in smaller environments can be deployed for coverage in larger unknown environments with marginal additional sample complexity.

The goal of the study is to understand what data structure is necessary to teach a robot to perform simple manipulation tasks. Training will take place thanks to Imitation learning, using the behavior cloning algorithm. Imitation learning is extremely useful for training any robotic system, especially with multi-jointed robots. This approach allows you to avoid writing complex and cumbersome heuristic algorithms. Heuristic algorithms often contain many hyperparameters that must be chosen empirically. Imitation learning algorithms allow you to develop an End-to-End policy from the collected data, which simplifies the system and increases its reliability. In the realm of training robotic systems through Imitation Learning, the criticality of data collection cannot be overstated. The efficacy of the Behavior Cloning algorithm hinges on the quality and speed at which data is gathered. Properly curated and swiftly acquired datasets serve as the bedrock for training algorithms that emulate human behavior, allowing the robot to seamlessly replicate intricate manipulations. The efficiency of data gathering is paramount in addressing the challenges posed by the dynamic nature of real-world environments. Swift adaptation to varying conditions and unforeseen scenarios necessitates a data collection process that is not only rapid but also comprehensive. A well-curated dataset not only expedites the training process but also enhances the model's adaptability and robustness in real-world applications. In conclusion, the success of Imitation Learning algorithms, particularly those underpinned by the Behavior Cloning approach, is contingent on the meticulous and expeditious gathering of data. This underscores the importance of streamlined data collection processes, ensuring that the resultant models are not only accurate but also capable of navigating the complexities of diverse and dynamic environments. Fastidious attention to data gathering is imperative to ensure a diverse and representative dataset. The xArm7 robot, in collaboration with the Realsense D435 camera, serves as an ideal synergy for capturing a myriad of movements and scenarios. The precision of data collection directly influences the algorithm's ability to generalize, making it adept at handling a spectrum of manipulation tasks.

This paper presents our solution for enabling a quadrotor helicopter, equipped with a laser rangefinder sensor, to autonomously explore and map unstructured and unknown indoor environments. While these capabilities are already commodities on ground vehicles, air vehicles seeking the same performance face unique challenges. In this paper, we describe the difficulties in achieving fully autonomous helicopter flight, highlighting the differences between ground and helicopter robots that make it difficult to use algorithms that have been developed for ground robots. We then provide an overview of our solution to the key problems, including a multilevel sensing and control hierarchy, a high-speed laser scan-matching algorithm, an EKF for data fusion, a high-level SLAM implementation, and an exploration planner.1 Finally, we show experimental results demonstrating the helicopter's ability to navigate accurately and autonomously in unknown environments.

This paper discusses the importance, the complexity and the challenges of mapping mobile robot's unknown and dynamic environment, besides the role of sensors and the problems inherited in map building. These issues remain largely an open research problems in developing dynamic navigation systems for mobile robots. The paper presenst the state of the art in map building and localization for mobile robots navigating within unknown environment, and then introduces a solution for the complex problem of autonomous map building and maintenance method with focus on developing an incremental grid based mapping technique that is suitable for real-time obstacle detection and avoidance. In this case, the navigation of mobile robots can be treated as a problem of tracking geometric features that occur naturally in the environment of the robot. The robot maps its environment incrementally using the concept of occupancy grids and the fusion of multiple ultrasonic sensory information while wandering in it and stay away from all obstacles. To ensure real-time operation with limited resources, as well as to promote extensibility, the mapping and obstacle avoidance modules are deployed in parallel and distributed framework. Simulation based experiments has been conducted and illustrated to show the validity of the developed mapping and obstacle avoidance approach.

This paper studies delay-aware formation control for (unmanned aerial vehicle) UAV swarms operating under realistic air-to-air communication latency. An attractive–repulsive distance-based potential-field (ARD-PF) controller is used as an expert to generate demonstrations for imitation learning in multi-UAV cooperative systems. By augmenting the training data with communication delay, the learned policy implicitly compensates for outdated neighbor information and improves swarm coordination during autonomous flight. Extensive simulations across different swarm sizes, formation spacings, and delay levels show that delay-robust imitation learning significantly enlarges the probabilistic stability region compared with classical ARD-PF control and non-robust learning baselines. Formation control performance is evaluated using internal geometric error, global offset, and multi-run stability metrics. In addition, a predictive delay–stability model is introduced, linking the maximum admissible communication delay to swarm size and inter-agent spacing, with low fitting error against simulated stability boundaries. The results provide quantitative insights for designing communication-aware UAV swarm systems under latency constraints.

PurposeThis study aims to introduce the fast reactive tree (FRT*) algorithm for enhancing replanning speed and reducing the overall cost of navigation in unknown dynamic environments.Design/methodology/approachFRT* comprises four key components: inverted tree build, convex hull construction, dead nodes inform activation and lazy-rewiring replanning. First, an initial path is found from the inverted tree where the valid structure is preserved to minimise re-exploration areas during the replanning phase. As the robot encounters environment changes, convex hulls are extracted to sparsely describe impacted areas. Next, the growth direction of the modified tree is biased by the inform activation of dead nodes to avoid unnecessary exploration. In the replanning phase, the tree structure is optimized using the proposed lazy-rewiring replanning to find a high-quality path with low computation burden.FindingsA series of comprehensive simulation experiments demonstrate that the proposed FRT* algorithm can efficiently replan short-cost feasible paths in unknown dynamic environments. The differential wheeled mobile robot with varying reference linear velocities is used to validate the effectiveness and adaptability of the proposed strategy in real word scenarios. Furthermore, ablation studies are conducted to analyze the significance of the key components of FRT*.Originality/valueThe proposed FRT* algorithm introduces a novel approach to addressing the challenges of navigation in unknown dynamic environments. This capability allows mobile robots to safely and efficiently navigate through unknown and dynamic environments, making the method highly applicable to real-world scenarios.

A general scheme of cooperative hunting for a moving target by multiple mobile robots in unknown environments is presented. To realize successful hunting, the point stabilization control algorithm is used for path planning of hunting mobile robots. Simultaneously, collision risk based obstacle avoidance scheme is used to keep away from random obstacles during hunting. The synthesized behavior is obtained by fusing avoidance behavior, encirclement behavior and pincer attack behavior. Simulated hunting experiments were conduct under different conditions in Microsoft Robotics Studio (MRS) and simulation results demonstrate the correctness and effectiveness of the proposed scheme.

In this paper autonomous formation control for Unmanned Aerial Vehicles (UAVs) has been discussed and a real time solution has been put forward by benefiting General Purpose Graphical Processing Units (GPGPU) accelerated potential field approach while ensuring obstacle and collision avoidance in unknown environment by using real-time sensors. GPGPU accelerated real time formation control for UAVs was designed and the basic model of the approach has been explained in our previous work (Cetin and Yilmaz 2014). As the deficiencies of the previous approach, autonomous real time collision and coordinated obstacle avoidance features in unknown environments are also handled while maintaining formation flight conditions in this work. With these features, improved autonomous formation control approach is discussed as a real time solution. The computation is performed by using Graphical Processing Units (GPUs) as parallel computation architectures by benefiting from Single Instruction Multiple Data (SIMD) type parallel algorithms. Classic binary map conversation, connected component labeling and minimum bounding box algorithms which are commonly used for image processing applications, has been evaluated for real time obstacle detection and avoidance features by developing GPGPU suitable parallel algorithms. Real-time solution has been developed by integrated these parallel algorithms with parallel Artificial Potential Field (APF) computation algorithm. Simulation results are proved that this novel autonomous improved formation control approach is successful and it would be used in real time applications like UAV formation flight missions.

As mobile robots are becoming more and more widely used, it is of great significance to design an efficient path planning method for multi-robot systems (MRS) that can adapt to complex and unknown environments. In this paper, we present a deep reinforcement learning (DRL) based method to navigate the MRS with collision avoidance in unknown dynamic environments, which consists of centralized learning and decentralized executing paradigm. The proposed policy maps original laser information into robot control commands without constructing global maps. The learned policy is tested in Gazebo environments with three robot systems, which shows the effective performance in terms of success rate, extra time rate, and formation maintenance rate.