Autonomous navigation of UAV in large-scale unknown complex environment with deep reinforcement learning

The path planning of Unmanned Aerial Vehicle (UAV) is a critical component of rescue operation. As impacted by the continuity of the task space and the high dynamics of the aircraft, conventional approaches cannot find the optimal control strategy. Accordingly, in this study, a deep reinforcement learning (DRL)-based UAV path planning method is proposed, enabling the UAV to complete the path planning in a 3D continuous environment. The deep deterministic policy gradient (DDPG) algorithm is employed to enable UAV to autonomously make decisions. Besides, to avoid obstacles, the concepts of connected area and threat function are proposed and adopted in the reward shaping. Lastly, an environment with static obstacles is built, and the agent is trained using the proposed method. As has been proved by the experiments, the proposed algorithm can fit a range of scenarios.

Unmanned aerial vehicle (UAV) has been widely deployed in efficient data collection for Internet of Things (IoT). UAV can not only act as a relay, but also as an energy source to provide information and energy transmission for ground sensor nodes (SNs). This paper studies the efficient multi-UAV-assisted data collection problem in wireless powered IoT. Specifically multiple UAVs wirelessly charge SNs using radio frequency (RF) energy transfer, and the SNs then use the harvested energy to upload the updates of the sensed information to the UAVs, thus improving the freshness of collected data and extending the service time of the SNs. The problem is modeled as a partially observed Markov decision process (POMDP) with a large observation and action space, where each UAV acts as an intelligent agent to learn the environment and make decisions independently. The value-decomposition network (VDN) algorithm is employed to find the optimal strategy in the multi-agent deep reinforcement learning framework. Simulation results validate the effectiveness of the proposed data collection approach compared to two baseline policies.

Unmanned Aerial Vehicles (UAVs) have a major issue in autonomous navigation, particularly in dynamic or GPS-denied environments. Conventional rule-based or GPS-dependent strategies lack the flexibility and resilience to support real-time decision-making, especially in difficult terrain. We propose a novel deep learning-based framework, DeepFusion-NavNet, that couples reinforcement learning with semantic segmentation to perform end-to-end autonomous UAV navigation in an attempt to overcome these limitations. It employs a Deep Q-Network (DQN) to learn the optimal trajectory of the UAV based on policies learned once spatial features are extracted from aerial images by a U-Net-based semantic segmentation model. The proposed method was experimented with in both urban and forest environments utilizing the AirSim simulation tool. Relative to existing deep reinforcement learning baselines, experimental validation proves that DeepFusion-NavNet results in a navigation success rate of 97.5%, reduces collision incidents by 91.2%, and improves route efficiency by 22.8%. These results indicate that DeepFusion-NavNet significantly enhances UAV safety and autonomy under challenging operating conditions.

An important challenge for air–ground unmanned systems achieving autonomy is navigation, which is essential for them to accomplish various tasks in unknown environments. This paper proposes an end-to-end framework for solving air–ground unmanned system navigation using deep reinforcement learning (DRL) while optimizing by using a priori information from search-based path planning methods, which we call search-based optimizing DRL (SO-DRL) for the air–ground unmanned system. SO-DRL enables agents, i.e., an unmanned aerial vehicle (UAV) or an unmanned ground vehicle (UGV) to move to a given target in a completely unknown environment using only Lidar, without additional mapping or global planning. Our framework is equipped with Deep Deterministic Policy Gradient (DDPG), an actor–critic-based reinforcement learning algorithm, to input the agents’ state and laser scan measurements into the network and map them to continuous motion control. SO-DRL draws on current excellent search-based algorithms to demonstrate path planning and calculate rewards for its behavior. The demonstrated strategies are replayed in an experienced pool along with the autonomously trained strategies according to their priority. We use a multi-stage training approach based on course learning to train SO-DRL on the 3D simulator Gazebo and verify the robustness and success of the algorithm using new test environments for path planning in unknown environments. The experimental results show that SO-DRL can achieve faster algorithm convergence and a higher success rate. We piggybacked SO-DRL directly onto a real air–ground unmanned system, and SO-DRL can guide a UAV or UGV for navigation without adjusting any networks.

Energy-aware control for multiple unmanned aerial vehicles (UAVs) is one of the major research interests in UAV based networking. Yet few existing works have focused on how the network should react around the timing when the UAV lineup is changed. In this work, we study proactive self-remedy of energy-constrained UAV networks when one or more UAVs are short of energy and about to quit for charging. We target at an energy-aware optimal UAV control policy which proactively relocates the UAVs when any UAV is about to quit the network, rather than passively dispatches the remaining UAVs after the quit. Specifically, a deep reinforcement learning (DRL)-based self remedy approach, named SREC-DRL, is proposed to maximize the accumulated user satisfaction scores for a certain period within which at least one UAV will quit the network. To handle the continuous state and action space in the problem, the state-of-the-art algorithm of the actor-critic DRL, i.e., deep deterministic policy gradient (DDPG), is applied with better convergence stability. Numerical results demonstrate that compared with the passive reaction method, the proposed SREC-DRL approach shows a 12.12% gain in accumulative user satisfaction score during the remedy period.

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

This paper proposes a method for planning three-dimensional path for unmanned aerial vehicle (UAV) in complex airspace based on interfered fluid dynamical system (IFDS) and deep reinforcement learning. Firstly, the model of unmanned aerial vehicle under various constraints and the mathematical expression of threat zone are established. Secondly, in order to solve the problems of slow calculation speed and difficult to make the global optimal solution existed at present, an intelligent 3D path planning method on the basis of IFDS is proposed, and deep reinforcement learning is used to solve the coefficient of IFDS. The simulation results show that the path planned by the proposed method can avoid the threat zone effectively, meanwhile, the path is smooth, suitable and fuel saving for UAV.

In recent years, unmanned aerial vehicles (UAVs) have been widely used in wireless communication, attracting intensive attentions. UAVs can not only serve as relays, but also serve as aerial base station for ground users (GUs). However, limited energy means that they cannot work for long and cover a limited area of services. In this paper, we investigate 2D UAV trajectory design and power allocation in order to maximize the UAV's service time and downlink throughput. Based on deep reinforcement learning, we propose a deep deterministic policy gradient (DDPG) algorithm for trajectory design and power allocation (TDPA) to solve the energy efficient and communication service quality problem. The simulation results show that TDPA can extend the service time of UAV, improve the communication service quality, and realize the maximization of downlink throughput, which are significantly improved compared with existing methods.

Unmanned aerial vehicle (UAV) technology is a rapidly growing field with tremendous opportunities for research and applications. To achieve true autonomy for UAVs in the absence of remote control, external navigation aids like global navigation satellite systems and radar systems, a minimum energy trajectory planning that considers obstacle avoidance and stability control will be the key. Although this can be formulated as a constrained optimization problem, due to the complicated non-linear relationships between UAV trajectory and thrust control, it is almost impossible to be solved analytically. While deep reinforcement learning is known for its ability to provide model free optimization for complex system through learning, its state space, actions and reward functions must be designed carefully. This paper presents our vision of different layers of autonomy in a UAV system, and our effort in generating and tracking the trajectory both using deep reinforcement learning (DRL). The experimental results show that compared to conventional approaches, the learned trajectory will need 20% less control thrust and 18% less time to reach the target. Furthermore, using the control policy learning by DRL, the UAV will achieve 58.14% less position error and 21.77% less system power.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

This paper is concerned with the problem of using deep reinforcement learning methods to enable safe navigation of UAVs in unknown large-scale environments containing a large number of obstacles. The ability of UAV to navigate autonomously is a precondition for performing disaster rescue and logistics deliveries. In this paper, the perception-constrained navigation of UAV is modeled as a partially observable Markov decision process (POMDP), and a fast recurrent stochastic valued gradient algorithm based on the actot-critic framework is designed for online solution. Compared with traditional SLAM-based and reactive obstacle avoidance-based approaches, the proposed navigation method can map the raw sensing information into navigation signals, and the learned policy deployed to the UAV can avoid the memory occupation and computational consumption required for real-time map building. Through experiments in a newly designed simulation environment, it is demonstrated that the proposed algorithm can enable the UAV to navigate safely in unknown cluttered large-scale environments and outperform the state-of-the-art DRL algorithm in terms of performance.

Cooperative robots became a very important research topic due to their capability to accomplish tasks fast and more efficiently. In order for cooperative robots to accomplish their tasks, they need to be able to plan their paths in the environment without any collision. Path planning for cooperative robots with deep reinforcement learning is a new research topic in robotics and artificial intelligence. Path planning via deep reinforcement learning is an end to end method. The robot directly receives the data from the sensor (usually high dimensional images) and generates an optimal policy that plans a safe path to the target. In this paper, we first present an overview of reinforcement learning, deep learning, and deep reinforcement learning. Then, we introduce methods used for path planning. Finally, we discuss the challenges of deep reinforcement learning path planning for cooperative robots.

Unmanned Aerial Vehicle (UAV)-assisted communication has drawn increasing attention recently. In this paper, we investigate 3D UAV trajectory design and band allocation problem considering both the UAV's energy consumption and the fairness among the ground users (GUs). Specifically, we first formulate the energy consumption model of a quad-rotor UAV as a function of the UAV's 3D movement. Then, based on the fairness and the total throughput, the fair throughput is defined and maximized within limited energy. We propose a deep reinforcement learning (DRL)-based algorithm, named as EEFC-TDBA (energy-efficient fair communication through trajectory design and band allocation) that chooses the state-of-the-art DRL algorithm, deep deterministic policy gradient (DDPG), as its basis. EEFC-TDBA allows the UAV to: 1) adjust the flight speed and direction so as to enhance the energy efficiency and reach the destination before the energy is exhausted; and 2) allocate frequency band to achieve fair communication service. Simulation results are provided to demonstrate that EEFC-TDBA outperforms the baseline methods in terms of the fairness, the total throughput, as well as the minimum throughput.