Autonomous Navigation of UAV in Dynamic Unstructured Environments via Hierarchical Reinforcement Learning

Autonomous navigation is a highly desirable capability for Unmanned Aerial Vehicle (UAV). In this paper, the problem of autonomous navigation of UAV in unknown dynamic environments is addressed. Specifically, we propose a visual-inertial multi-modal sensor data perception framework based on a hierarchical reinforcement learning paradigm. This model consists of a high-level behavior selection model and a low-level policy execution model. The high-level model learns a stable and reliable behavior selection strategy. The low-level model decomposes the UAV navigation task into two simpler subtasks, which respectively achieve obstacle avoidance and goal approximation, which effectively learns high-level semantic information about the scene and narrows the strategy space. Furthermore, extensive simulation results are provided to confirm the superiority of the proposed method in terms of convergence and effectiveness compared to state-of-the-art methods.

Assisting elderly individuals in indoor environments is crucial, as they often encounter mobility and safety challenges. Autonomous Unmanned Aerial Vehicles (UAVs) navigation offers a transformative solution, significantly enhancing safety, efficiency, and quality of life. These UAVs can perform tasks such as monitoring, delivering essential items, and responding to emergencies, providing invaluable support and promoting greater independence for elderly individuals. To achieve effective assistance, autonomous UAV navigation relies on robust path planning with obstacle avoidance to ensure optimal performance. A newly developed path planning, based on a new hybrid Water Cycle Algorithm (WCA), stands out by addressing the challenge of avoiding obstacles while adhering to non-holonomic constraints and conserving energy in complex, cluttered indoor environments. In Population-Based Algorithms (PBAs), the initial population plays a crucial role, as it greatly impacts the algorithm’s efficiency in exploring the search space and its rate of convergence. The new developed hybrid algorithm, called WCA-HS, is based on using the Harmony Search (HS) algorithm to identify the optimal initial population for the WCA. The results indicate that the hybrid WCA-HS outperforms classical WCA and other metaheuristic algorithms, such as HS, Firefly Algorithm (FA), Cuckoo Search (CS), Genetic Algorithm (GA), Differential Evolution (DE), Ant Colony Optimization (ACO), and Artificial Bee Colony (ABC), over 20 independent runs, highlighting the effectiveness of the population initialization technique. Additionally, the developed indoor path planning system was evaluated in highly cluttered and low-light environments, showcasing its robustness and real-world applicability, making it highly effective for assisting elderly individuals in complex indoor environments.

Unmanned Aerial Vehicles (UAVs) have a major issue in autonomous navigation, particularly in dynamic or GPS-denied environments. Conventional rule-based or GPS-dependent strategies lack the flexibility and resilience to support real-time decision-making, especially in difficult terrain. We propose a novel deep learning-based framework, DeepFusion-NavNet, that couples reinforcement learning with semantic segmentation to perform end-to-end autonomous UAV navigation in an attempt to overcome these limitations. It employs a Deep Q-Network (DQN) to learn the optimal trajectory of the UAV based on policies learned once spatial features are extracted from aerial images by a U-Net-based semantic segmentation model. The proposed method was experimented with in both urban and forest environments utilizing the AirSim simulation tool. Relative to existing deep reinforcement learning baselines, experimental validation proves that DeepFusion-NavNet results in a navigation success rate of 97.5%, reduces collision incidents by 91.2%, and improves route efficiency by 22.8%. These results indicate that DeepFusion-NavNet significantly enhances UAV safety and autonomy under challenging operating conditions.

Chapter 21 - Event-driven programming-based path planning and navigation of UAVs around a complex urban environment

Visual odometry (VO) has become a popular technique for autonomous naviga-tion of unmanned aerial vehicles (UAVs) in GPS-denied environments. Although, the concept of VO is not new, it has been successfully implemented for on-ground vehi-cles only. There are several product level solutions of autonomous navigation, which rely on VO or contain VO as an important subpart. Meanwhile, a proper system of VO for UAVs is not yet achieved and continues to be a subject of an active research. In this paper, we show the comparison of the latest VO techniques for UAV nav-igation. We begin by describing the basic principles of VO, then review the different types of VO algorithms commonly used in UAV navigation, such as monocular, stereo, and RGB-D approaches. For each type of algorithm, we discuss the advantages and limitations, and provide examples of recent research and applications. Finally, we dis-cuss some of the emerging trends and future directions in visual odometry for UAV navigation, such as Visual inertial odometry approaches and real-time implementa-tions. This review paper provides a comprehensive overview of the current state-of-the-art in visual odometry for UAV autonomous navigation, highlighting the ad-vantages and limitations of different types of algorithms and identifying key challeng-es and future research directions.

Unmanned Aerial Vehicle (UAV) has received wide interest in recent years because of their promising potentials for several applications that include border surveillance, intrusion, detection, emergent item delivery, wildlife monitoring, rescue operations etc. The growing number of entrances of drones into airspace has emerged a significant interest in the collision and obstacle avoidance in autonomous operation. One of major challenges for safer deployment of UAVs is collision avoidance in order to make UAVs autonomous and self-sustainable. In today’s technological era, it is possible for UAVs to avoid obstacles and prevent collisions while navigation by implementing cutting edge computer vision algorithms and visual tracking systems. But it is challenging due to the limited payload in UAV that only permits vision based sensors such as camera, inertial measurement unit such for control and obstacle detection. This paper aims to summarize existing solutions of collision avoidance strategies used in the navigation of UAV. Collision avoidance strategies based on geometry, optimization, sense and avoid, vision-based and force field based methods are investigated in this study.

This paper discusses the algorithms of vision systems for automatic UAV (Unmanned Aerial Vehicle) landing. The basic algorithms for finding helipad and adapting the UAV control system to an autonomous UAV landing are presented. The authors consider algorithms to solve navigation problems, building block diagrams of automatic UAV control. Also, the developed algorithms make it possible to implement an automatic landing system for UAVs using technical vision systems, using various camera parameters, algorithms allowing research, as part of the autonomous navigation of unmanned aerial vehicles. The article discusses the developed algorithm that allows you to highlight certain characteristic points for visual navigation. Also, a search system for characteristic points has been developed that allows automatic UAV landing. These algorithms may be useful for the autonomous landing UAV systems. Algorithms can also be used to track a trajectory of the UAV system.

Unmanned aerial vehicles (UAVs) have drawn increased research interest in recent years, leading to a vast number of applications, such as, terrain exploration, disaster assistance and industrial inspection. Unlike UAV navigation in outdoor environments that rely on GPS (Global Positioning System) for localization, indoor navigation cannot rely on GPS due to the poor quality or lack of signal. Although some reviewing papers particularly summarized indoor navigation strategies (e.g., Visual-based Navigation) or their specific sub-components (e.g., localization and path planning) in detail, there still lacks a comprehensive survey for the complete navigation strategies that cover different technologies. This paper proposes a taxonomy which firstly classifies the navigation strategies into Mapless and Map-based ones based on map usage and then, respectively categorizes the Mapless navigation into Integrated, Direct and Indirect approaches via common characteristics. The Map-based navigation is then split into Known Map/Spaces and Map-building via prior knowledge. In order to analyze these navigation strategies, this paper uses three evaluation metrics (Path Length, Deviation Rate and Exploration Efficiency) according to the common purposes of navigation to show how well they can perform. Furthermore, three representative strategies were selected and 120 flying experiments conducted in two reality-like simulated indoor environments to show their performances against the evaluation metrics proposed in this paper, i.e., the ratio of Successful Flight, the Mean time of Successful Flight, the Mean Length of Successful Flight, the Mean time of Flight, and the Mean Length of Flight. In comparison to the CNN-based Supervised Learning (directly maps visual observations to UAV controls) and the Frontier-based navigation (necessitates continuous global map generation), the experiments show that the CNN-based Distance Estimation for navigation trades off the ratio of Successful Flight and the required time and path length. Moreover, this paper identifies the current challenges and opportunities which will drive UAV navigation research in GPS-denied environments.

Accurate trajectory prediction is essential for ensuring flight safety, autonomous navigation, and efficient control of unmanned aerial vehicles (UAVs). However, existing neural network approaches often focus on a single platform, limiting their generalizability across UAVs with different dynamics. This study evaluates three neural architectures for spatiotemporal trajectory prediction using high-frequency flight data. Two UAV configurations, such as a fixed-wing (Cessna) and a multirotor (Phantom), were simulated in XPlane 11 under identical environmental conditions. Flight data of latitude, longitude, and altitude sampled at 80 Hz were preprocessed using a sliding-window segmentation method to preserve temporal dependencies. Backpropagation Neural Network (BPNN), Recurrent Neural Network (RNN), and Long Short-Term Memory (LSTM) models were trained using standardised preprocessing and tuned hyperparameters. Results show that recurrent models outperform the feedforward BPNN in capturing temporal patterns. For the Cessna dataset, the RNN achieved balanced accuracy with a mean-squared error (MSE) of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5.5 \times 10^{-8}$</tex>, while the LSTM obtained the lowest MSE (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.7 \times 10^{-8}$</tex>). In the more dynamic Phantom dataset, the BPNN remained competitive (MSE <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\approx 1.8 \times 10^{-7}$</tex>), whereas RNN and LSTM maintained superior altitude prediction. These results highlight the suitability of lightweight recurrent architectures for real-time UAV trajectory prediction.

Template matching in real-time is a fundamental issue in many applications in computer vision such as tracking, stereo vision and autonomous navigation. The goal of this paper is to present a system for automatic image sequence processing for autonomous aerial navigation research. Two main experiments were employed in this work, the first with clipping images of a geo-referenced aerial image, to simulate the unmanned aerial vehicle (UAV) navigation, and the second employing the video frames obtained from a camera fixed to a helicopter in a low level flight, simulating the vision system of an UAV. The image features used in recognition task were obtained by the SIFT algorithm. The recognition system consists on a video frame processing to estimate the UAV displacement. Promising results were obtained.

This study explores the application of Particle Swarm Optimization (PSO) in Unmanned Aerial Vehicle (UAV) path planning within a simulated three-dimensional environment. UAVs, increasingly prevalent across various sectors, demand efficient navigation solutions that account for dynamic and unpredictable elements. Traditional pathfinding algorithms often fall short in complex scenarios, hence the shift towards PSO, a bio-inspired algorithm recognized for its adaptability and robustness. We developed a Python-based framework to simulate the UAV path planning scenario. The PSO algorithm was tasked to navigate a UAV from a starting point to a predetermined destination while avoiding spherical obstacles. The environment was set within a 3D grid with a series of waypoints, marking the UAV's trajectory, generated by the PSO to ensure obstacle avoidance and path optimization. The PSO parameters were meticulously tuned to balance the exploration and exploitation of the search space, with an emphasis on computational efficiency. A cost function penalizing proximity to obstacles guided the PSO in real-time decision-making, resulting in a collision-free and optimized path. The UAV's trajectory was visualized in both 2D and 3D perspectives, with the analysis focusing on the path's smoothness, length, and adherence to spatial constraints. The results affirm the PSO's effectiveness in UAV path planning, successfully avoiding obstacles and minimizing path length. The findings highlight PSO's potential for practical UAV applications, emphasizing the importance of parameter optimization. This research contributes to the advancement of autonomous UAV navigation, indicating PSO as a viable solution for real-world path planning challenges.

Unmanned aerial vehicles (UAVs), also known as drones, have gained considerable interest among academics in recent years, which significantly increases the demand for autonomous navigation systems for UAVs. In this study, autonomous navigation is addressed as a two-stage problem. The first stage is the obstacle detection phase, during which the UAVs leverage their onboard sensors to perceive the surroundings and detect obstacles. The second stage is the obstacle avoidance phase, where the UAVs make decisions to continue operations without colliding with anything while approaching the target. This study proposes a self-supervised learning-based depth prediction framework as the criterion for obstacle proximity detection. The framework consists of two subnetworks: the depth and pose estimation modules. The former predicts depth values, while the latter evaluates the object’s translation and rotation matrix. The framework leverages the subnetworks’ outputs to reconstruct the projection relationships of pixels between adjacent frames to optimize the training process. To address the challenge of UAVs operating in large-scale environments, a modified loss function that combines photometric loss and edge-aware smoothness is presented to increase the depth prediction module’s accuracy. This study also presents a deep reinforcement learning-based obstacle avoidance system that takes depth maps as inputs to generate evasive maneuvers. The system works independently from priori maps and models while it learns the collision-free trajectory from interacting with the environment. This study deploys value-based algorithms to learn the nonlinear mapping between the continuous inputs and discrete output motion commands. Furthermore, the proposed system is validated on different simulation scenarios. It is capable of completing the obstacle avoidance task with relatively optimal trajectories in varied environments, demonstrating its superior effectiveness and transferability. Keywords: unmanned aerial vehicles, urban airspace, obstacle detection and avoidance, self-supervised learning, deep reinforcement learning.

Unmanned Aerial Vehicles (UAVs) based delivery is thriving. In this paper, we model autonomous navigation of UAV in large-scale unknown complex environment as a discrete-time continuous control problem and solve it using deep reinforcement learning. Without path planning or map construction, our method enables UAVs to navigate from arbitrary departure places to destinations using only sensory information of local environment and GPS signal. We argue the navigation task is a partially observable Markov decision process (POMDP) and extant recurrent deterministic policy gradient algorithm is less efficient. Consequently, we derive a faster policy learning algorithm for POMDP based on actor-critic architecture. To validate our ideas, we simulate five virtual environments and a virtual UAV flying at a fixed altitude with constant speed. Cognition of local environment is achieved by measuring distances from UAV to obstacles in multiple directions. Simulation results demonstrate the effectiveness of our method.

This paper presents a software framework which combines reactive collision avoidance control approach with path planning techniques for the purpose of safe navigation of multiple Unmanned Aerial Vehicles (UAVs) operating in unknown environments. The system proposed leverages advantages of using a fast local sense-and-react type control which guarantees real-time execution with computationally demanding path planning algorithms which generate globally optimal plans. A number of probabilistic path planning algorithms based on Probabilistic Roadmaps and Rapidly-Exploring Random Trees have been integrated. Additionally, the system uses a reactive controller based on Optimal Reciprocal Collision Avoidance (ORCA) for path execution and fast sense-and-avoid behavior. During the mission execution a 3D map representation of the environment is build incrementally and used for path planning. A prototype implementation on a small scale quad-rotor platform has been developed. The UAV used in the experiments was equipped with a structured-light depth sensor to obtain information about the environment in form of occupancy grid map. The system has been tested in a number of simulated missions as well as in real flights and the results of the evaluations are presented.

Abstract. The Simultaneous Localization and Mapping (SLAM) technology is fundamental to the autonomous navigation of Unmanned Aerial Vehicles (UAVs) and holds significant value for the realization of UAV-based bridge inspections. However, conventional SLAM methods for UAV face challenges related to low continuity and weak reliability across different scenes, making it difficult to meet the requirements for comprehensive bridge localization and mapping. To address the limitations of existing UAV-based SLAM approaches, we propose a seamless SLAM system that integrates IMU, LiDAR, and RTK. In open scenes (such as the top and sides of a bridge), high-precision absolute localization is achieved by fusing IMU and RTK through an iterative error-state Kalman filter (IESKF). In occluded environments (such as the underside of a bridge), an IMU/LiDAR odometry is used to recursively estimate the UAV’s pose. In cross-scene situations (when the UAV passes through a bridge arch), the quality of sensor data is evaluated based on an interactive multi-model (IMM), and an adaptive switching mechanism is employed between two localization modes—IMU/RTK mode and IMU/LiDAR mode—ensuring smooth and seamless multi-source fusion localization even in the presence of sensor signal fluctuations. To validate the effectiveness of our method, extensive tests were conducted on several real-world bridge scenarios. The results show that our method can achieve centimetre-level cross-scene localization accuracy in bridge inspection applications, which indicates its feasibility and effectiveness.