Optimization of high-speed fixed-wing UAV penetration strategy based on deep reinforcement learning

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

Design complexities of trending UAVs and the operational harsh environments necessitates Control Law formulation utilizing intelligent techniques that are both robust, model-free and adaptable. In this research, an intelligent control architecture for an Unmanned Aerial Vehicle (UAV) having an unconventional inverted V-tail design, is presented. Due to unique design of the vehicle strong roll and yaw coupling exists, making the control of vehicle challenging. To handle UAV’s inherent control complexities, while keeping them computationally acceptable, a variant of distinct Deep Reinforcement Learning (DRL) algorithm, namely Deep Deterministic Policy Gradient (DDPG) is proposed. Conventional DDPG algorithm after being modified in its learning architecture becomes capable of intelligently handling the continuous state and control space domains besides controlling the platform in its entire flight regime. This stabilization and tracking controller for the UAV achieves the goal of an optimal flight path and exhibits satisfactory performance. The paper illustrates the application of modified DDPG algorithm towards the design, while the performance of the resulting controller is assessed in simulation using dynamic model of the vehicle. Nonlinear simulations were then performed to analyze UAV performance under different environmental and launch conditions. The effectiveness of the proposed strategy is further demonstrated by comparing the results with the linear controller for the same UAV whose feedback loop gains are optimized by employing technique of optimal control theory and another DRL method named Proximal Policy Optimization (PPO). Results indicate the significance of the proposed control architecture and its inherent capability to adapt dynamically to the changing environment, thereby making it of significance utility to airborne applications.

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

Energy-aware control for multiple unmanned aerial vehicles (UAVs) is one of the major research interests in UAV based networking. Yet few existing works have focused on how the network should react around the timing when the UAV lineup is changed. In this work, we study proactive self-remedy of energy-constrained UAV networks when one or more UAVs are short of energy and about to quit for charging. We target at an energy-aware optimal UAV control policy which proactively relocates the UAVs when any UAV is about to quit the network, rather than passively dispatches the remaining UAVs after the quit. Specifically, a deep reinforcement learning (DRL)-based self remedy approach, named SREC-DRL, is proposed to maximize the accumulated user satisfaction scores for a certain period within which at least one UAV will quit the network. To handle the continuous state and action space in the problem, the state-of-the-art algorithm of the actor-critic DRL, i.e., deep deterministic policy gradient (DDPG), is applied with better convergence stability. Numerical results demonstrate that compared with the passive reaction method, the proposed SREC-DRL approach shows a 12.12% gain in accumulative user satisfaction score during the remedy period.

: Unmanned aerial vehicles (UAVs) are critical components of the future naval forces. UAV control and monitoring with autonomous operation will become an absolute necessity and adaptive cooperation of vehicles is the only practical alternative. The objective of this project is to develop and evaluate new methodologies for cooperative (formation) control of multiple unmanned air vehicles. The goal is to have multiple UAVs working together as a group. Instead of separately assigning distinct tasks to each vehicle, the operator would assign tasks to the UAV group, which then determines the best way to accomplish each task, freeing the operator to maintain surveillance over the entire operation. In this project we investigated Path Tracking and obstacle avoidance of UAVs using fuzzy logic method. Algorithms for close formation control of multi-UAVs are developed and simulated. We also investigated fault-tolerant control of single UAVs by neuro-adaptive method. Detailed description of this method is provided in this document. The project has supported S graduate students with 9 technical papers published.

As a complicated multi-input, multi-output, and time-varying nonlinear system, flight control system of unmanned aerial vehicle (UAV), which determines the whole system’s performance directly, is crucial for the simulation training system design. This chapter mainly focuses on parameter identification of flight control system based on the modeling of UAVs and a specific controller design for the pendulum-like oscillation in micro aerial vehicle (MAV). A predator–prey particle swarm optimization (PSO) algorithm for identifying parameters of UAV flight control system is presented, with the aim of reducing the workload of the designers during the process of designing complicated UAV control system. Besides, a software environment for UAV controller design was developed based on the UAV model and the proposed method. Then a specific kind of controller design involving the pendulum-like oscillation in the hover and stare state for a MAV in the presence of external disturbances is discussed in detail, since the pendulum-like oscillation caused by uncertainty and external disturbances badly jeopardize the performance of hover and stare, resulting in blurred images or even MAV’s overturning. A novel type of PSO-based linear-quadratic regulator (LQR) controller for stabilizing the pendulum-like oscillation is developed, which can enhance the MAV’s performance efficiently.

With the development of Internet of Things (IoT) technology, wireless sensor networks (WSNs) composed of multiple sensor nodes (SNs) have been able to provide a large number of high-quality services, such as continuous environmental monitoring, automatic control, and transmission of data for intelligent decision-making. SNs in WSNs are typically energy limited and often exist in large numbers, which are widely distributed in hard-to-reach areas. Hence, it is difficult to collect data from SNs without infrastructure support such as base stations (BSs). The low cost and high maneuverability of unmanned aerial vehicles (UAVs) provide an efficient solution for data acquisition in WSNs but with the challenge of control system of the UAV. In this paper, we proposed a deep reinforcement learning based UAV control system to intelligently solve the data harvesting problem in WSNs. Our optimization target is to jointly optimize the trajectory of the UAV and the bandwidth resources of the airborne base station to maximize the data acquisition success rate when the UAV is limited in energy. The objective function and constraints of this problem are highly non-convex. To solve this problem, we first modeled the movement of the UAV and the bandwidth allocation of the airborne base station in each time slot as a Markov decision process (MDP). Then we designed a deep reinforcement learning architecture to solve the above MDP. Compared with the existing data harvesting algorithms, the proposed method has a great improvement in data acquisition success rate.

Unmanned aerial vehicles (UAVs) are increasingly deployed in missions that require high autonomy and reliable decision-making; however, many operational concepts still assume access to GNSS and stable communication with a human operator. In contested environments, this assumption may no longer hold because GNSS degradation, radio-frequency interference, and intentional jamming can disrupt positioning and communication, thereby reducing mission effectiveness and safety. Recent surveys show that operation in GNSS-denied environments remains a major challenge and often requires alternative perception, localization, and control strategies. In response, this article investigates a reinforcement learning (RL)-based decision-support system for the autonomous control of a quadrotor UAV in a three-dimensional simulated environment. Rather than following pre-programmed waypoints, the UAV learns a control policy through interaction with the environment and reward-driven adaptation. The proposed system is designed for mission execution under uncertainty, limited external guidance, and partial observability. Two policy-gradient approaches are implemented and compared: classical REINFORCE and Proximal Policy Optimization (PPO) with an Actor-Critic architecture. The study presents the simulation environment, state and action representation, reward formulation, staged training procedure, and comparative evaluation. The results indicate that, within the considered unseen test scenario, the PPO-based configuration achieved higher mission effectiveness than REINFORCE in the final unseen test scenario, supporting the practical relevance of structured deep reinforcement learning for UAV operation in GPS-denied and communication-constrained environments.

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

The increasing usage of Unmanned Aerial Vehicles (UAVs) in diverse applications necessitates the development of effective flight control systems. A major difficulty, nevertheless, is maintaining robust flight control under complex and dynamic environmental conditions, such as obstacle and non-flying zones, wind disturbances, and sensor noise. Traditional control techniques fail to achieve required flight control in these environments. To address this point, an adaptive control strategy is proposed based on a Deep Reinforcement Learning (DRL) model to enhance the flight performance of quadcopters. The learning and adaptation of the DRL-based control strategy are implemented in real-time through continuous interaction with the environment. This is to improve flight control and achieve consistent UAV performance under varying conditions. Proximal policy optimization with a reward function is used to minimize positional errors, ensure collision-free flight paths, and reduce energy consumption. The developed DRL model for quadcopters is trained in a simulated environment and then tested in three complex environmental scenarios, including urban, forest and mountain terrains. Experimental results demonstrate remarkable improvements in UAV flight performance. In the training phase, the reported training reward increased from 10 to 110 and the train loss is dropped from 0.85 to 0.05, which indicated successful model learning. Also, during system verification, rewards increased from 12 to 115 and UAV flight path deviations were decreased from 0.5 to 0.08m. The proposed controller outperforms conventional approaches in urban environments by lowering average trajectory deviations to 0.2m from 0.35m for MPC and 0.6m for PID. Also, the developed DRL-based controller outperformed the PID and MPC controllers, with path deviations of 0.18m in mountains and 0.12m in forests. In addition, fewer collision rates with obstacles are achieved with the model, 3% in forest, 1.8% in urban areas, and 4.5% in mountains. Furthermore, the consumed energy is reduced to 950J as compared to 1200J for PID and 1050J for MPC. The results show the strength of deploying the proposed controller in following the intended UAV flight path with high precision, effectively avoiding detected conflicts and minimizing consumed energy by the UAV.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.
\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. 
\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. 
\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. 
\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

Hybrid modeling and control issues provide a flexible powerful tool for Unmanned Aerial Vehicles (UAVs) as complex control platforms. This paper aims to briefly review recent results in this area. Firstly, existing works on hybrid modeling and control of a single UAV are reviewed. Then multiple UAVs control in a synchronously cooperative task planning, which gives much more maneuverability, has been surveyed in the hybrid framework. Formation control of UAVs is a typical cooperative strategy with considerable applications in multi-agent aerial robotics area and can leverage limited abilities of single UAVs into complicated tasks of group missions. There are a few works in this field which we briefly reviewed their results and explored the possible future directions of these research activities.

In this article simultaneous arrival of multiple unmanned aerial vehicle (UAV) at a common point was studied in a fixed network hierarchy is used. By considering the model of fixed wing UAV the decentralized based control algorithm is developed for the formation control of all UAV’s. The tree based networked geometry is developed for the formation control of multiple UAV’s with respect to the leader. Whereas, UAV’s agent or (leader / follower) are autonomously connected with each other. Formation of flight via decentralized control structure which contains two parts, one is to achieve the arrival of expected meet-point for the formation geometry and other is to maintain the synchronization of all UAV’s. Moreover, the leader is supposed and able to regulate the speed or velocity of UAV’s. Simulated results showed that arrival of multiple UAV’s at a common point for the formation is achieved successfully and the above designed control algorithm has good robustness and flexible.

This paper investigates a new scenario of spectrum sharing between unmanned\naerial vehicle (UAV) and terrestrial wireless communication, in which a\ncognitive/secondary UAV transmitter communicates with a ground secondary\nreceiver (SR), in the presence of a number of primary terrestrial communication\nlinks that operate over the same frequency band. We exploit the UAV's mobility\nin three-dimensional (3D) space to improve its cognitive communication\nperformance while controlling the co-channel interference at the primary\nreceivers (PRs), such that the received interference power at each PR is below\na prescribed threshold termed as interference temperature (IT). First, we\nconsider the quasi-stationary UAV scenario, where the UAV is placed at a static\nlocation during each communication period of interest. In this case, we jointly\noptimize the UAV's 3D placement and power control to maximize the SR's\nachievable rate, subject to the UAV's altitude and transmit power constraints,\nas well as a set of IT constraints at the PRs to protect their communications.\nNext, we consider the mobile UAV scenario, in which the UAV is dispatched to\nfly from an initial location to a final location within a given task period. We\npropose an efficient algorithm to maximize the SR's average achievable rate\nover this period by jointly optimizing the UAV's 3D trajectory and power\ncontrol, subject to the additional constraints on UAV's maximum flying speed\nand initial/final locations. Finally, numerical results are provided to\nevaluate the performance of the proposed designs for different scenarios, as\ncompared to various benchmark schemes. It is shown that in the quasi-stationary\nscenario the UAV should be placed at its minimum altitude while in the mobile\nscenario the UAV should adjust its altitude along with horizontal trajectory,\nso as to maximize the SR's achievable rate in both scenarios.\n

Instruction sender authentication of UAV (Unmanned Aerial Vehicle) control is critical for the secure operation of UAV systems. By exploring user touch action characteristics of the user interaction with the UAV control mobile terminal, this paper proposes a novel instruction sender continuous authentication framework for UAV control systems. In particular, we first obtain the touch screen data in real time from the interaction with the mobile device screen of a user when he controls the UAV. Then we create a polynomial to fit the touch trajectory on the screen of the user, and use the least square method to obtain the optimal estimate of the polynomial coefficients. Finally, we mark the grid area covered by the polynomial to extract the user’s touch screen operation action fingerprint(OAF), and employ the fuzzy extractor method to realize the identity authentication of the sender with certain error tolerance. Extensive experiments are conducted to illustrate the authentication performance of the proposed authentication framework in terms of false acceptance rate, false rejection rate and equal-error rate.