Reinforcement-Learning-Based AAV 3-D Target Tracking and Digital-Twin-Assisted Collision Avoidance With Integrated Sensing and Communication

With the development of Internet of Things (IoT) technology, wireless sensor networks (WSNs) composed of multiple sensor nodes (SNs) have been able to provide a large number of high-quality services, such as continuous environmental monitoring, automatic control, and transmission of data for intelligent decision-making. SNs in WSNs are typically energy limited and often exist in large numbers, which are widely distributed in hard-to-reach areas. Hence, it is difficult to collect data from SNs without infrastructure support such as base stations (BSs). The low cost and high maneuverability of unmanned aerial vehicles (UAVs) provide an efficient solution for data acquisition in WSNs but with the challenge of control system of the UAV. In this paper, we proposed a deep reinforcement learning based UAV control system to intelligently solve the data harvesting problem in WSNs. Our optimization target is to jointly optimize the trajectory of the UAV and the bandwidth resources of the airborne base station to maximize the data acquisition success rate when the UAV is limited in energy. The objective function and constraints of this problem are highly non-convex. To solve this problem, we first modeled the movement of the UAV and the bandwidth allocation of the airborne base station in each time slot as a Markov decision process (MDP). Then we designed a deep reinforcement learning architecture to solve the above MDP. Compared with the existing data harvesting algorithms, the proposed method has a great improvement in data acquisition success rate.

A new consensus theory-based method for formation control and obstacle avoidance of UAVs

Unmanned Aerial Vehicles (UAVs), or drones, have recently become a favorable solution for fast parcel delivery due to their maneuverability and advances in navigation technologies. With the limitation of battery capacity and payload of drones, it is crucial to consider both efficiency and cost while conducting the tasks. Meanwhile, UAVs should not collide with each other while traveling to customers. In this article, we propose a UAV parcel delivery system involving deep reinforcement learning (DRL) approach for collision avoidance and a genetic algorithm for route optimization. Specifically, a delivery center generates near-optimal routes, loading UAV with parcels according to demands. Each UAV takes charge of delivering packages in compliance with the assigned route while avoiding collision with each other. We utilize DRL to achieve collision avoidance without having prior knowledge about the trajectories of other UAVs. Additionally, we adopt a genetic algorithm to obtain the lowest energy cost path for each UAV. To find such an optimized path, we solve a capacitated vehicle routing problem (CVRP) with a modified cost function and extra constraints. Realistic simulations using a physics engine and software-in-the-loop (SITL) are conducted to evaluate the feasibility of the proposed methods.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.
\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. 
\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. 
\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. 
\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

This paper studies the unmanned aerial vehicle (UAV)-enabled integrated sensing and communication (ISAC), in which UAVs are dispatched as aerial dual-functional access points (APs) that can exploit the UAV maneuver control and strong line-of-sight (LoS) air-to-ground (A2G) links for efficient communication and sensing. In particular, we consider that one UAV-AP, equipped with a vertically placed uniform linear array (ULA), sends combined information and sensing signals to communicate with multiple users and at the same time sense potential targets at interested areas on the ground. Under this setup, we consider two scenarios with quasi-stationary and fully mobile UAVs, in which the UAV is deployed at an optimizable location over the whole ISAC mission period and can fly over different locations during the ISAC mission period, respectively. For the two scenarios, our objective is to jointly design the UAV maneuver (deployment location or flight trajectory) and the transmit beamforming, for maximizing the weighted sum-rate throughput of communication users, while ensuring the sensing beampattern gain requirements, subject to the transmit power and flight constraints. However, due to the ULA consideration at the UAV, the two formulated problems are highly non-convex and very difficult to be optimally solved, as the UAV's location/trajectory variables are involved on the exponent parts of each entry in the steering vectors, and are closely coupled with the transmit beamforming vectors. To tackle this issue, we propose efficient algorithms to find their suboptimal but high-quality solutions, by using various techniques from convex and non-convex optimization. Finally, numerical results are provided to validate the superiority of our proposed designs as compared to various benchmark schemes with heuristic maneuver designs. It is shown that the joint maneuver and transmit beamforming design efficiently balances the inherent tradeoff between sensing and communication with regards to different beampattern gain thresholds.

U NMANNED Aerial Vehicles (UAVs) hold good promise for autonomously carrying out complex civilian and military operations. However, many of these missions require them to fly at low altitudes, making them vulnerable to collision with both stationary as well as moving obstacles. Hence, it is vital that UAVs are equipped with autonomous capability to sense and avoid collisions, especially for the pop-up threats. When such a threat is sensed and a collision is predictedwithin a short time ahead, theUAV should be able to react and maneuver away quickly so that the collision is avoided. An algorithmwhich can assure such amaneuver is called a “reactive collision avoidance algorithm.” Since the available reaction time in such a scenario is usually is small and UAVs are usually limited by computational resources, such an algorithm should also be computationally very efficient (it should preferably be noniterative). It is also required that whilemaneuvering away, it should notmaneuver toomuch away from the obstacle either. This is both to avoid collision from other nearby obstacles and not to compromise on the overall mission objective. There are various attempts in the literature to develop algorithms for collision avoidance purpose, many of which are inspired from global path planning algorithms. The artificial potential field method is such an approach where the motion of the vehicle is guided under the influence of a potential field. The potential field (which is essentially a cost function) is designed in such a way that obstacles have repulsive fields while the destination has an attractive field. The safe path of the UAV is then found by optimizing the carefully selected cost function. To tune this basic philosophy for reactive collision avoidance, a model predictive control-based algorithm has been proposed in the literature. This algorithm essentially assures path following under safe conditions (i.e. if no collision is predicted in the near future) and invokes the potential field function when new collisions are sensed. However, in potential field based techniques the associated optimization process is typically done in an iterative manner. Because of this they are usually computationally intensive and hence are not suitable for reactive collision avoidance of airborne UAVs in general. A promising algorithm in collision avoidance and global path planning is the philosophy of rapidly-exploring random tree (RRT) [1], which has also been used for reactive collision avoidance. However, there are many concerns about the RRT approach, which can largely be attributed to the random nature of the algorithm. For example, the path predicted by RRT is usually a sting of connected straight lines that does not reflect the path followed by a vehicle with nonholonomic constraints. More important, it is a probabilistic approach and hence there is no guarantee of finding a feasible path within a limited finite time. Other graph search algorithms such as best-first search are also implemented for reactive collision avoidance [2]. However, this is not systematic approach and could result in the algorithm searching far too many nodes under some conditions. Moreover, precomputing motion primitives and saving them in a lookup table is infeasible for UAVs, which are usually resource-limited. An interesting perspective to collision avoidance problem is the minimum effort guidance [3], where an optimal control-based approach has been proposed after applying the collision cone philosophy to detect collisions. This method is computationally nonintensive as a closed form solution has been proposed. Even though this is an interesting idea, by minimizing the lateral acceleration, perhaps it imposes unwanted extra constraint on the problem formulation as reactive collision avoidance problems do not necessarily have to be carried out with minimum lateral acceleration. More important, one can observe that this formulation only assures position guarantee and no constraint is imposed on the velocity vector. Hence, even though it guides the vehicle to a carefully selected target point on the safety boundary (we call it the “aiming point”), it causes the vehicle to maneuver until this point. This can be risky as the vehicle may enter the safety ball before reaching the aiming point. Even though the collision cone based aiming point philosophy is a very good idea, the authors of this Note strongly believe that instead of only position guarantee, rather the velocity vector should be aligned towards the aiming point as soon as a collision is detected (which will automatically lead to position guarantee as well). Towards this objective, two new nonlinear guidance laws are proposed in this Note, which are named as nonlinear geometric guidance (NGG) and differential geometric guidance (DGG). These guidance laws are inspired by the philosophy of “aiming point guidance” (APG) [4], which has been proposed in missile guidance literature. It turns out that the APG is a simplified case of the NGG where the associated since function is replaced by its linear approximation (hence, for a systematic discussion, it is renamed as linear geometric guidance (LGG) in this Note). Both of the guidance algorithms proposed in this Note quickly align the velocity vector of the UAValong the aiming point within a part of the available time-togo, which ensures quick reaction and hence safety of the vehicle. The main feature of this philosophy is that they effect high maneuvering at the beginning, causing the velocity vector of the UAV to align with the aiming point direction quickly and then settling along it. Therefore there is no need to maneuver all the way until the aiming point is reached and hence the chance of the UAV entering into the safety ball is minimized. Using the point of closest approach (PCA) [5], the proposed NGG andDGGalgorithms have also been extended for collision avoidance with moving obstacles in both cooperative as well as ignorant scenarios. Mathematical correlations between the guidance laws have also been established, which show that the NGG and DGG are exactly correlated to each other with appropriate gain selections, while the LGG is an approximation of DGG. A “sphere-tracking algorithm” is also proposed in this Note where the UAV is guided to track the surface of the safety spherewhenever a brief violation of the safety boundary occurs after reaching the aiming point because of the location of the next aiming point (which may include the target in Presented as Paper 2010-8315 at the AIAA Guidance, Navigation and Control, Toronto, 2–5August 2010; received 26May 2010; revision received 4 October 2010; accepted for publication 6 October 2010. Copyright © 2010 by Radhakant Padhi. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission. Copies of this paper may be made for personal or internal use, on condition that the copier pay the $10.00 per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923; include the code 0731-5090/11 and $10.00 in correspondence with the CCC. ∗Former Project Assistant, Department of Aerospace Engineering; anushamujumdar87@gmail.com. Associate Professor, Department of Aerospace Engineering; padhi@ aero.iisc.ernet.in. JOURNAL OF GUIDANCE, CONTROL, AND DYNAMICS Vol. 34, No. 1, January–February 2011

The high speed attitude maneuver of Unmanned Aerial Vehicle (UAV) always causes large motion between adjacent frames of the video stream produced from the camera fixed on the UAV body, which will severely disrupt the performance of image object tracking process. To solve this problem, this paper proposes a method that using a gyroscope fixed on the camera to measure the angular velocity of camera, and then the object position's substantial change in the video stream is predicted. We accomplished the object tracking based on template matching. Experimental result shows that the object tracking algorithm's performance is improved in its efficiency and robustness with embedded gyroscope information.

Due to the high maneuverability of unmanned aerial vehicle (UAV), a cluster of UAVs is considered used to collect sensing data from the sensors that distributed randomly in an area without the terrestrial infrastructure. The cluster members work as relays to forward the sensing data from sensors to the cluster head. For the reason that the relay deployment impacts the transmission rate and coverage area directly, we are going to optimize the deployment of the UAV relays, aiming to maximize the total capacity of the network. The problem of multi-relay deployment is intractable for two reasons. On one hand, because of the interactional and coupled relationship among the UAV relays, when the deployment of any given relay changes, the deployment optimization of other relays will be affected. On the other hand, on account of that the exact positions of the sensors are unknown, the deployment optimization of the UAV relays cannot be completed directly because of lacking parameters. In order to tackle the coupled relationship among the UAV relays, the problem of multi-relay deployment is modeled as a local interaction game. We prove that the multi-relay deployment game is an exact potential game that has at least one Nash equilibrium (NE) point. Then, the better reply-based relay deployment approach, which is an online learning approach that does not demand the information of the exact positions of sensors, is proposed to search the NE point. The simulation results show that the network capacity is significantly enhanced with the proposed relays deployment approach.

Wireless communication is an essential technology to unlock the full potential of unmanned aerial vehicles (UAVs) in numerous applications and has thus received unprecedented attention recently. Although technologies such as direct link, WiFi, and satellite communications are still useful in some remote scenarios where cellular services are unavailable, it is believed that exploiting the thriving 5G and beyond cellular networks to support UAV communications is the most promising and cost-effective approach, especially when the number of UAVs grows dramatically. On the one hand, to guarantee safe and efficient flight operations of multiple UAVs, it is of paramount importance to provide secure and ultra-reliable communication links between the UAVs and their ground pilots or control stations for conveying command and control signals, especially in beyond-visual-line-of-sight (BVLOS) scenarios. On the other hand, because of advances in communication equipment miniaturization as well as UAV manufacturing, mounting compact and lightweight base stations (BSs) or relays on UAVs becomes increasingly feasible. This has led to two promising research paradigms for UAV communications, namely, UAV-assisted cellular communications and cellular-connected UAVs, where UAVs are integrated into cellular networks as aerial communication platforms and aerial users, respectively. As such, integrating UAVs into cellular networks is believed to be a win-win technology for both UAV-related industries and cellular network operators, which not only creates plenty of new business opportunities but also benefits the communication performance of 3-D wireless networks. In addition, UAV related sensing and computing are also helpful for achieving efficient and reliable communication (e.g., in avoiding coverage holes) as well as smart UAV coordination, positioning, and trajectory design. However, 5G and beyond wireless networks with UAVs significantly differs from traditional communication systems, because of the high altitude and high maneuverability of UAVs, the unique UAV-ground channels, the diversified quality of service (QoS) requirements for downlink command and control (C&C) and uplink mission-related data transmission, the stringent constraints imposed by the size, weight, and power (SWAP) limitations of UAVs, as well as the new design degrees of freedom enabled by joint UAV mobility control and communication resource allocation.

Unmanned Aerial Vehicle (UAV) has received wide interest in recent years because of their promising potentials for several applications that include border surveillance, intrusion, detection, emergent item delivery, wildlife monitoring, rescue operations etc. The growing number of entrances of drones into airspace has emerged a significant interest in the collision and obstacle avoidance in autonomous operation. One of major challenges for safer deployment of UAVs is collision avoidance in order to make UAVs autonomous and self-sustainable. In today’s technological era, it is possible for UAVs to avoid obstacles and prevent collisions while navigation by implementing cutting edge computer vision algorithms and visual tracking systems. But it is challenging due to the limited payload in UAV that only permits vision based sensors such as camera, inertial measurement unit such for control and obstacle detection. This paper aims to summarize existing solutions of collision avoidance strategies used in the navigation of UAV. Collision avoidance strategies based on geometry, optimization, sense and avoid, vision-based and force field based methods are investigated in this study.

The research focuses on methods and software tools for decision support for the organization of distributed production of unmanned aerial vehicles (UAVs) based on digital twin and multi-agent systems technologies. The purpose of the work is to provide a critical review of existing approaches to the organization of distributed production of UAVs, as complex technical objects, and to substantiate an approach to building a multi-agent decision support system for the synthesis of a rational production structure based on digital twins of participants in production cooperation. The tasks include: classifying unmanned aerial vehicles according to the main features that determine their production specifics; analysing modern scientific publications in the field of distributed production informatization, multi-agent systems, digital twins and intelligent decision support; identifying the limitations of existing solutions; and proposing an approach to the organization of distributed production of UAVs using digital twins in the form of intelligent agents. The applied methods are based on system and classification analysis, generalization of modern scientific approaches to the organization of production, decision-making theory, the concept of multi-agent systems, digital twin technologies, production logistics and artificial intelligence. The following results were obtained: UAVs were classified by mass, purpose and level of autonomy, which made it possible to determine the specifics of their production and life cycle management. An analysis of scientific publications was performed, which showed the active development of areas related to the use of multi-agent systems, digital twins and artificial intelligence to address complex tasks in UAV life cycle management. It was established, in particular, that the works of recent years consider the issues of cooperative UAV production, the use of deep learning algorithms for decision-making in dynamic environments and the integration of digital twins into relevant production systems. At the same time, a number of limitations on the implementation of existing solutions were identified: insufficient formalization of business processes between production participants, weak integration of digital twins into production logistics, and the lack of self-organization mechanisms. An approach to solving the problem of effective organization of distributed UAV production is proposed. The architecture of a multi-agent decision support system for distributed UAV production is developed, which includes modules for coordination, self-organization, and the evaluation of the effectiveness of production configuration options, considering the initial risk tolerance indicator. Conclusions. The scientific novelty of the results obtained is as follows: an approach to solving the problem of effective organization of distributed UAV production is proposed, which, unlike existing ones, is based on the representation of production participants as digital twins in the form of intelligent agents, this enables the generation of multiple options and the subsequent selection of the most rational configuration for organizing distributed UAV production in terms of business process efficiency and the reduction of production risks. The research results create a basis for further integration of multi-agent technologies into real-world manufacturing platforms and the expansion of their application to other types of complex machinery.

The number of Unmanned Aerial Vehicles (UAVs) flying around is increasing year after year due to their popularity. This new scenario considerably increases the risk of collision between UAVs, which makes the development of protocols to avoid this issue an emerging topic. This work focuses on the development of a collision avoidance protocol between UAVs performing planned missions. Our proposal is based on the periodic submission of state information, and future position estimates of the UAV, allowing the detection of collision risks. Collision avoidance is achieved by forcing UAVs to stop when they are flying critically near each other, and establishing the priority of the UAV passing by the critical area first; this decision is based on a per-UAV unique identifier. We prove that our solution is robust enough to avoid collisions in all the situations tested; in addition, the overhead introduced by our protocol in terms of additional UAV flight time is low.

Unmanned aerial vehicles (UAV) have been widely used in many fields. UAV formation flight can complete more assignments than single UAV. Collision avoidance is an important part of UAV formation flight. In this paper, we put forward a three-dimensional method of dynamic collision avoidance for UAV formation flight based on the theory of velocity obstacle. The collection of all possible accelerations of the UAV can be obtained by the input of UAV. Considering that the relative velocity between the UAV and obstacle must be out of the collision cone, we can obtain the collection of feasible accelerations next moment. Then we choose the acceleration which is able to let the UAV closest to the target next moment in the collection as the real acceleration. The maximum and minimum velocity should also be set up. Each UAV can determine the best acceleration next time for itself. In addition to formation keeping, this paper also presents how to split the formation on the basis of the direction of each UAV. Numerical simulation is used to test the performance of this method at the end of this paper.

Energy-aware control for multiple unmanned aerial vehicles (UAVs) is one of the major research interests in UAV based networking. Yet few existing works have focused on how the network should react around the timing when the UAV lineup is changed. In this work, we study proactive self-remedy of energy-constrained UAV networks when one or more UAVs are short of energy and about to quit for charging. We target at an energy-aware optimal UAV control policy which proactively relocates the UAVs when any UAV is about to quit the network, rather than passively dispatches the remaining UAVs after the quit. Specifically, a deep reinforcement learning (DRL)-based self remedy approach, named SREC-DRL, is proposed to maximize the accumulated user satisfaction scores for a certain period within which at least one UAV will quit the network. To handle the continuous state and action space in the problem, the state-of-the-art algorithm of the actor-critic DRL, i.e., deep deterministic policy gradient (DDPG), is applied with better convergence stability. Numerical results demonstrate that compared with the passive reaction method, the proposed SREC-DRL approach shows a 12.12% gain in accumulative user satisfaction score during the remedy period.

Autonomous Landing of Fixed Wing Unmanned Aerial Vehicle with Reactive Collision Avoidance