Lightweight Obstacle Avoidance for Fixed-Wing UAVs Using Entropy-Aware PPO

<div>In this dissertation, methods for real-time trajectory generation and autonomous obstacle avoidance for fixed-wing and quad-rotor unmanned aerial vehicles (UAV) are studied. A key challenge for such trajectory generation is the high computation time required to plan a new path to safely maneuver around obstacles instantaneously. Therefore, methods for rapid generation of obstacle avoidance trajectory are explored. The high computation time is a result of the computationally intensive algorithms used to generate trajectories for real-time object avoidance. Recent studies have shown that custom solvers have been developed that are able to solve the problem with a lower computation time however these designs are limited to small sized problems or are proprietary. Additionally, for a swarm problem, which is an area of high interest, as the number of agents increases the problem size increases and in turn creates further computational challenges. A solution to these challenges will allow for UAVs to be used in autonomous missions robust to environmental uncertainties.</div><div><br></div><div>In this study, a trajectory generation problem posed as an optimal control problem is solved using a sequential convex programming approach; a nonlinear programming algorithm, for which custom solver is used. First, a method for feasible trajectory generation for fast-paced obstacle-rich environments is presented for the case of fixed-wing UAVs. Next, a problem of trajectory generation for fixed-wing and quad-rotor UAVs is defined such that starting from an initial state a UAV moves forward along the direction of flight while avoiding obstacles and remaining close to a reference path. The problem is solved within the framework of finite-horizon model predictive control. Finally, the problem of trajectory generation is extended to a swarm of quad-rotors where each UAV in a swarm has a reference path to fly along. Utilizing a centralized approach, a swarm scenario with moving targets is studied in two different cases in an attempt to lower the solution time; the first, solve the entire swarm problem at once, and the second, solve iteratively for a UAV in the swarm while considering trajectories of other UAVs as fixed.</div><div><br></div><div>Results show that a feasible trajectory for a fixed-wing UAV can be obtained within tens of milliseconds. Moreover, the obtained feasible trajectories can be used as initial guesses to the optimal solvers to speed up the solution of optimal trajectories. The methods explored demonstrated the ability for rapid feasible trajectory generation allowing for safe obstacle avoidance, which may be used in the case an optimal trajectory solution is not available. A comparative study between a dynamic and a kinematic model shows that the dynamic model provides better trajectories including aggressive trajectories around obstacles compared to the kinematic counterpart for fixed-wing UAVs, despite having approximately the same computational demands. Whereas, for the case of quad-rotor UAVs, the kinematic model takes almost half the solution time than with a reduced dynamic model, despite having approximately the similar range of values for the cost function. When extended to a swarm, solving the problem for each UAV is four to seven times computationally cheaper than solving the swarm as a whole. With the improved computation time for trajectory generation for a swarm of quad-rotors using centralized approach, the problem is now reasonably scalable, which opens up the possibility to increase the number of agents in a swarm using high-end computing machines for real-time applications. Overall, a custom solver jointly with a sequential convex programming approach solves an optimization problem in a low computation time.</div>

Path planning and obstacle avoidance are pivotal for intelligent unmanned aerial vehicle (UAV) systems in various domains, such as postdisaster rescue, target detection, and wildlife conservation. Currently, reinforcement learning (RL) has become increasingly popular in UAV decision-making. However, the RL approaches confront the challenges of partial observation and large state space when searching for random targets through continuous actions. This paper proposes a representation enhancement-based proximal policy optimization (RE-PPO) framework to address these issues. The representation enhancement (RE) module consists of observation memory improvement (OMI) and dynamic relative position-attitude reshaping (DRPAR). OMI reduces collision under partially observable conditions by separately extracting perception features and state features through an embedding network and feeding the extracted features to a gated recurrent unit (GRU) to enhance observation memory. DRPAR compresses the state space when modeling continuous actions by transforming movement trajectories of different episodes from an absolute coordinate system into different local coordinate systems to utilize similarity. In addition, three step-wise reward functions are formulated to avoid sparsity and facilitate model convergence. We evaluate the proposed method in three 3D scenarios to demonstrate its effectiveness. Compared to other methods, our method achieves a faster convergence during training and demonstrates a higher success rate and a lower rate of timeout and collision during inference. Our method can significantly enhance the autonomy and intelligence of UAV systems under partially observable conditions and provide a reasonable solution for UAV decision-making under uncertainties.

A new consensus theory-based method for formation control and obstacle avoidance of UAVs

Existing resource-intensive obstacle avoidance techniques are hard to be applied on the small-size Unmanned Aerial Vehicles (UAVs) that have limited sensing and computation capacity. Therefore, it is necessary to develop an obstacle recognition and avoidance scheme which works under resource-constrained environments. To this backdrop, this paper first presents an obstacle recognition model based on monocular vision feature points. Afterwards, an obstacle recognition algorithm is put forward, whose computational complexity is low. Then, an obstacle avoidance method is proposed to regulate the obstacle-avoidance path of a UAV until it arrives its destination. To evaluate the effectiveness of the presented algorithms, we design and implement a simulation platform on Objective Modular Network Testbed in C++ (OMNeT++), and conduct a series of experiments. Experimental results show that the proposed model and algorithms can effectively guide a micro UAV to its destination using only an embedded processor and 0.5kg extra load. Even in poor communication conditions, the UAV can independently avoid obstacles and reach the destination only by acquiring the destination coordinates from the ground station.

Aiming at the problem of multi-objective path planning (MOPP) and dynamic obstacle avoidance of manipulator in dynamic environment, this paper proposes a solution based on hierarchical reinforcement learning (HRL) framework. Traditional path planning methods have some problems in dynamic scenes, such as poor real-time performance, difficult to balance multi-objective conflicts and insufficient adaptability to environmental changes. Therefore, this paper designs a two-tier architecture including global path planning layer and local obstacle avoidance layer, which is trained by Proximal policy optimization (PPO) and Soft Actor-Critic (SAC) algorithms respectively, and achieves collaborative optimization through an efficient information exchange mechanism between layers. At the same time, a multi-objective reward function based on dynamic weight adjustment strategy is introduced. Fuzzy logic is used to adaptively balance the relationship among path length, obstacle avoidance safety and energy consumption according to environmental complexity. Combined with Long Short-Term Memory (LSTM), the trajectory of obstacles is predicted, and the potential field method is further introduced to modify the obstacle avoidance reward, which improves the real-time response ability and robustness of the algorithm in dynamic environment. The experimental results show that the HRL-SAC-PPO method proposed in this paper shows superior performance in both static and dynamic scenarios. In the static scene, the success rate of this method reaches 100%, the average path length is shortened to 2.13m, no collision occurs, and the energy consumption is reduced to 1.12 kJ, which shows a good multi-objective optimization effect. In the dynamic scene, the trajectory error of obstacles predicted by LSTM is only 4.2%, and the safe distance between the robot arm and obstacles is improved by 35%, which significantly enhances the reliability of obstacle avoidance. In addition, the average decision delay of this method is only 11.3ms, and the peak delay is 23ms, which is much lower than that of the contrast algorithm, showing stronger real-time response ability. The ablation experiment further verified the key role of LSTM trajectory prediction, dynamic weight adjustment and layered structure on the overall performance. In the welding task verification of the real UR5 manipulator, the success rate of the system in dynamic environment is 95.3%, the average path length is 3.41m, and the maximum joint acceleration is only 0.87 radian/s², which is far below the safety threshold, indicating that the algorithm has good stability and obstacle avoidance ability in practical application. The comprehensive performance comparison shows that this method performs well in different industrial scenarios, and has stronger environmental adaptability and comprehensive path planning ability.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.
\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. 
\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. 
\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. 
\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

Aiming at the obstacle avoidance control problem of small quadrotor, a method of quadrotor obstacle avoidance based on reinforcement learning is proposed. The proposed method can make training converge quickly and has good environmental robustness. The proposed methods include: (1) a framework adopts perception module and decision module to improve the generalization ability of the obstacle avoidance model; (2) An Actor-Critic framework-based Proximal Policy Optimization (PPO) algorithm to provide quadrotor with policy-based decision-making capabilities; The experimental simulation results show that the strategy-based framework converges quickly and has a high success rate, the training time is much lower than that of the value-based framework. The monocular vision observation ability is limited, which leads to deviations between local observation and global state, So LSTM layer is usually added to increase model performance. Policy -based decision can have a good obstacle avoidance effect without adding the LSTM layer, and have good generalization ability after short relearning after changing.

Optimization of high-speed fixed-wing UAV penetration strategy based on deep reinforcement learning

With the development and widely use of unmanned aerial vehicles (UAVs) in recent years, the development of efficient path planning methods for automation has become crucial. Obstacle avoidance and path planning are the key components of UAV path planning. This article provides an overview of obstacle avoidance and path planning techniques for UAVs based on the artificial potential field method (APF method). This article begins with the explaining the principles of artificial potential field on this basis discusses its advantages and limitations. The article then summarizes the improvement strategies proposed by previous researchers to address issues like local minimum values and unreachable targets, such as introducing a new repulsive potential energy function, combining APF with other planning methods, and utilizing flow functions. Furthermore, it presents examples of the application and the performance of usage of these techniques in both static and dynamic environments. Based on this, the prospects and developing trend of UAV obstacle avoidance methods based on artificial potential field are foresee, such as combined with DRL and deep learning.

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

The adoption of novel technologies such as Unmanned Aerial Vehicles (UAVs) in Search and Rescue (SAR) operations remains limited. As a result, their full potential is not yet realized. Although UAVs have been deployed on an ad hoc basis, typically under manual control by dedicated operators, assisted and fully autonomous configurations remain largely unexplored. In this study, three SAR frameworks are systematically evaluated within a unified benchmarking framework: conventional ground missions, UAV-assisted missions, and fully autonomous UAV operations. As the key performance indicator, the target localization time was quantified and used as the means of comparison amongst frameworks. The conventional and assisted frameworks were experimentally tested through physical hardware in a controlled outdoor setting, wherein simulated callouts occurred via rescue teams. The autonomous swarm framework was simulated in the form of a multi-agent Reinforcement Learning (RL) method via the use of the Proximal Policy Optimization (PPO) algorithm. This enabled the optimization of the decentralized cooperative actions that could occur for efficient exploration of a partially observed three-dimensional environment. Our results demonstrated that the autonomous swarm significantly outperformed the conventional and assisted approaches in terms of speed and coverage. Finally, a detailed depiction of the framework’s integration into an operational system is provided.

Distributed multi-agent collaborative decision-making technology is the key to general artificial intelligence. This paper takes the self-developed Unity3D collaborative combat environment as the test scenario, setting a task that requires heterogeneous unmanned aerial vehicles (UAVs) to perform a distributed decision-making and complete cooperation task. Aiming at the problem of the traditional proximal policy optimization (PPO) algorithm’s poor performance in the field of complex multi-agent collaboration scenarios based on the distributed training framework Ray, the Critic network in the PPO algorithm is improved to learn a centralized value function, and the muti-agent proximal policy optimization (MAPPO) algorithm is proposed. At the same time, the inheritance training method based on course learning is adopted to improve the generalization performance of the algorithm. In the experiment, MAPPO can obtain the highest average accumulate reward compared with other algorithms and can complete the task goal with the fewest steps after convergence, which fully demonstrates that the MAPPO algorithm outperforms the state-of-the-art.

The use of unmanned aerial vehicle (UAV) has been recognized by the majority of people, and is regarded as a reliable tool to complete some tasks. As for the obstacle avoidance of coaxial rotor UAV (CR-UAV), a flight pattern map (Flight Situation Awareness Map: FSAM) that can sense the flight environment and an integrated control method (Improved Artificial Potential Field: IAPF) based on the common artificial potential field method with appropriate improvements to achieve obstacle avoidance (IAPF). Firstly, we construct the FSAM, which can map the environment information around the UAV on the FSAM. Then, based on the FSAM, the IAPF functions are established to achieve the obstacles avoidance. Cause the artificial potential field (APF) has a characteristic that cannot avoid: the problem of local minima, a rotating potential field is put forward to ensure that the CR-UAV has only one potential equilibrium point at the target point in the environment, and it will improve the ability of CR-UAV to avoid complex obstacles. At the end of this study, through data analysis, the performance of obstacle avoidance and the attitude stability of CR-UAV are good, the simulation results confirm that the approaches proposed in this paper can address the obstacles avoidance successfully.KeywordsCR-UAVImproved Artificial Potential Field (IAPF)Flight Situation Awareness Map (FSAM)Obstacles avoidance

Design complexities of trending UAVs and the operational harsh environments necessitates Control Law formulation utilizing intelligent techniques that are both robust, model-free and adaptable. In this research, an intelligent control architecture for an Unmanned Aerial Vehicle (UAV) having an unconventional inverted V-tail design, is presented. Due to unique design of the vehicle strong roll and yaw coupling exists, making the control of vehicle challenging. To handle UAV’s inherent control complexities, while keeping them computationally acceptable, a variant of distinct Deep Reinforcement Learning (DRL) algorithm, namely Deep Deterministic Policy Gradient (DDPG) is proposed. Conventional DDPG algorithm after being modified in its learning architecture becomes capable of intelligently handling the continuous state and control space domains besides controlling the platform in its entire flight regime. This stabilization and tracking controller for the UAV achieves the goal of an optimal flight path and exhibits satisfactory performance. The paper illustrates the application of modified DDPG algorithm towards the design, while the performance of the resulting controller is assessed in simulation using dynamic model of the vehicle. Nonlinear simulations were then performed to analyze UAV performance under different environmental and launch conditions. The effectiveness of the proposed strategy is further demonstrated by comparing the results with the linear controller for the same UAV whose feedback loop gains are optimized by employing technique of optimal control theory and another DRL method named Proximal Policy Optimization (PPO). Results indicate the significance of the proposed control architecture and its inherent capability to adapt dynamically to the changing environment, thereby making it of significance utility to airborne applications.

In order to solve multiple unmanned aerial vehicle (UAV) dynamic collision avoidance, a cooperative obstacle avoidance algorithm considering UAV’s kinematic constraints has been developed. In the proposed algorithm, the useful information of UAVs is screened out by a Heartbeat information filtering mechanism and fused by the user datagram protocol (UDP) communication method, which improves communication performance among UAVs. In addition, the velocity obstacle (VO) method combined with cubic uniform B-spline curve is used to avoid obstacles and generate smooth paths, which can be applied to practical scenes. Finally, dynamic and static obstacle avoidance simulations are carried out to verify the effectiveness of the proposed algorithm.