UAV p­­­arcel Delivery System with Deep Reinforcement Learning Based Collision Avoidance and Route Optimization

This paper studies the integration of data collection and offloading for maritime Internet of Things (IoT) systems with multiple unmanned aerial vehicles (UAVs). In the considered multi-UAV maritime IoT system, the UAVs act as the aerial base stations to complete the missions of data collection from buoys and data offloading to the offshore base station (OBS). In this case, the UAVs need to adaptively select the mission mode between data collection and data offloading according to the network resources and mission requirements. In this paper, we aimed to minimize the completion time of data collection and offloading missions for all UAVs by jointly optimizing the UAV trajectories, mission mode selection, transmit power of buoys, and association relationships between the UAVs and buoy/OBS. In order to solve the mixed-integer non-convex minimization problem, we first designed a multi-agent deep reinforcement learning algorithm based on a hybrid discrete and continuous action space to preliminarily obtain the UAV trajectories, mission mode selection, and the transmit power of buoys. Then, we propose an algorithm based on the stable marriage problem to determine the buoy–UAV and UAV–OBS association relationships. Finally, the simulation results show that the proposed algorithms can effectively shorten the total mission completion time of data collection and offloading for the multi-UAV-assisted maritime IoT system.

The paper investigates the security of an intelligent reflecting surface (IRS) assisted unmanned aerial vehicle (UAV) network, where a base station (BS) transmits confidential information to the ground user (GU) via IRS-assisted UAV. The study explores using UAVs with IRS to enhance the security and reliability of wireless communication systems, particularly in the presence of eavesdroppers and friendly jammers. The beamforming at IRS-assisted UAV and UAV trajectory are jointly formulated as a non-convex optimization problem, which is solved by the deep reinforcement learning (DRL) algorithm to maximize the sum secrecy rate of GU with the aid of jammer. We proposed a dual-DDPG (D3PG) algorithm that utilizes the deep deterministic policy gradient (DDPG) structure to effectively address these dual non-convex problems of the UAV-trajectory optimization and the UAV-IRS beamforming optimization. The proposed algorithm's effectiveness and robustness are demonstrated through simulation results, with the IRS significantly enhancing the sum secrecy rate. Extensive simulations show that the proposed DRL-based D3PG scheme outperforms the traditional optimization schemes and ordinary DQN schemes.

With the rapid development of mobile edge computing (MEC) and unmanned aerial vehicle (UAV) communication networks, UAV-assisted edge computing has emerged as a promising paradigm for low-latency and energy-efficient computation. However, the time-varying nature of air-to-ground channels and the coupling between UAV trajectories and computation offloading decisions significantly increase system complexity. To address these challenges, this paper proposes an Adaptive UAV Edge Intelligence Framework (AUEIF) for joint UAV computation offloading and trajectory optimization under dynamic channels. Specifically, a dynamic graph-based system model is constructed to characterize the spatio-temporal correlation between UAV motion and channel variations. A hierarchical reinforcement learning-based optimization framework is developed, in which a high-level actor–critic module is responsible for generating coarse-grained UAV flight trajectories, while a low-level deep Q-network performs fine-grained optimization of task offloading ratios and computational resource allocation in real time. In addition, an adaptive channel prediction module leveraging long short-term memory (LSTM) networks is integrated to model temporal channel state transitions and to assist policy learning and updates. Extensive simulation results demonstrate that the proposed AUEIF achieves significant improvements in end-to-end latency, energy efficiency, and overall system stability compared with conventional deep reinforcement learning approaches and heuristic-based schemes while exhibiting strong robustness against dynamic and fluctuating wireless channel conditions.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

Due to flexible scheduling, and better transmission channel, unmanned aerial vehicle (UAV) can improve transmit performance of Internet of Things (IoT). In this paper, we propose an UAV-enabled multichannel IoT based on intelligent fog computing, where UAV as a relay forwards IoT's information to the data center under the control of fog computing base station in the case of terrestrial channel fading. The IoT's throughput are maximized by jointly optimizing subcarrier, power of IoT, and UAV, and UAV trajectory, subject to the constraints of information causality, maximum transmit power, and maximum UAV speed. The subcarriers are dynamically allocated according to their channel gains, the water filling algorithm is adopted to optimize the power for UAV, and IoT by fixing UAV trajectory, and the optimal UAV trajectory is achieved with successive convex approximation under the fixed power allocation. Then a jointly iterative optimization on subcarrier, power, and trajectory is presented to get the optimal solution. In addition, we propose a fairness optimization scheme to maximize the minimum transmit rate of IoT nodes. The simulations indicate the IoT with mobile UAV, and dynamic subcarrier allocation may achieve better transmit performance, and the fairness optimization can decrease the rate difference of nodes effectively.

Unmanned aerial vehicle (UAV) technology is a rapidly growing field with tremendous opportunities for research and applications. To achieve true autonomy for UAVs in the absence of remote control, external navigation aids like global navigation satellite systems and radar systems, a minimum energy trajectory planning that considers obstacle avoidance and stability control will be the key. Although this can be formulated as a constrained optimization problem, due to the complicated non-linear relationships between UAV trajectory and thrust control, it is almost impossible to be solved analytically. While deep reinforcement learning is known for its ability to provide model free optimization for complex system through learning, its state space, actions and reward functions must be designed carefully. This paper presents our vision of different layers of autonomy in a UAV system, and our effort in generating and tracking the trajectory both using deep reinforcement learning (DRL). The experimental results show that compared to conventional approaches, the learned trajectory will need 20% less control thrust and 18% less time to reach the target. Furthermore, using the control policy learning by DRL, the UAV will achieve 58.14% less position error and 21.77% less system power.

In recent years, the utilization of unmanned aerial vehicles (UAVs) has surged in a variety of applications, including weather monitoring, emergency search and rescue operations, and smart agriculture. In these applications, UAVs are instrumental in executing data collection tasks. However, due to the scarcity of resources such as battery capacity, the duration of work is limited, necessitating the optimization of UAV trajectories and resource allocation. In this paper, we introduce a hierarchical task scheduling and resource allocation scheme for UAV data collection tasks, which incorporates the deep reinforcement learning (DRL) technology into a two-layer training framework. The continuous actions, including UAV trajectories, UAV communication power, and UAV CPU main frequency, are first optimized in the lower layer, while the upper one concentrates on the discrete action space for the target allocation. In particular, the well-trained lower-layer network is integrated into the upper layer, facilitating rapid global reward feedback between the two-layer network and accelerating the training process of the upper-layer network. Experimental results demonstrate that the proposed approach achieves superior performance than the baseline methods in terms of the amount of collected data as well as UAV energy consumption.

Unmanned Aerial Vehicle (UAV)-assisted communication has drawn increasing attention recently. In this paper, we investigate 3D UAV trajectory design and band allocation problem considering both the UAV's energy consumption and the fairness among the ground users (GUs). Specifically, we first formulate the energy consumption model of a quad-rotor UAV as a function of the UAV's 3D movement. Then, based on the fairness and the total throughput, the fair throughput is defined and maximized within limited energy. We propose a deep reinforcement learning (DRL)-based algorithm, named as EEFC-TDBA (energy-efficient fair communication through trajectory design and band allocation) that chooses the state-of-the-art DRL algorithm, deep deterministic policy gradient (DDPG), as its basis. EEFC-TDBA allows the UAV to: 1) adjust the flight speed and direction so as to enhance the energy efficiency and reach the destination before the energy is exhausted; and 2) allocate frequency band to achieve fair communication service. Simulation results are provided to demonstrate that EEFC-TDBA outperforms the baseline methods in terms of the fairness, the total throughput, as well as the minimum throughput.

Caching the contents of unmanned aerial vehicles (UAVs) could significantly improve the content fetching performance of request users (RUs). In this paper, we study UAV trajectory design, content fetching, power allocation, and content placement problems in multi-UAV-aided networks, where multiple UAVs can transmit contents to the assigned RUs. To minimize the energy consumption of the system, we develop a constrained optimization problem that simultaneously designs UAV trajectory, power allocation, content fetching, and content placement. Since the original minimization problem is a mixed-integer nonlinear programming (MINLP) problem that is difficult to solve, the optimization problem was first transformed into a semi-Markov decision process (SMDP). Next, we developed a new technique to solve the joint optimization problem: option-based hierarchical deep reinforcement learning (OHDRL). We define UAV trajectory planning and power allocation as the low-level action space and content placement and content fetching as the high-level option space. Stochastic optimization can be handled using this strategy, where the agent makes high-level option selections, and the action is carried out at a low level based on the chosen option’s policy. When comparing the proposed approach to the current technique, the numerical results show that it can produce more consistent learning performance and reduced energy consumption.

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.

U NMANNED Aerial Vehicles (UAVs) hold good promise for autonomously carrying out complex civilian and military operations. However, many of these missions require them to fly at low altitudes, making them vulnerable to collision with both stationary as well as moving obstacles. Hence, it is vital that UAVs are equipped with autonomous capability to sense and avoid collisions, especially for the pop-up threats. When such a threat is sensed and a collision is predictedwithin a short time ahead, theUAV should be able to react and maneuver away quickly so that the collision is avoided. An algorithmwhich can assure such amaneuver is called a “reactive collision avoidance algorithm.” Since the available reaction time in such a scenario is usually is small and UAVs are usually limited by computational resources, such an algorithm should also be computationally very efficient (it should preferably be noniterative). It is also required that whilemaneuvering away, it should notmaneuver toomuch away from the obstacle either. This is both to avoid collision from other nearby obstacles and not to compromise on the overall mission objective. There are various attempts in the literature to develop algorithms for collision avoidance purpose, many of which are inspired from global path planning algorithms. The artificial potential field method is such an approach where the motion of the vehicle is guided under the influence of a potential field. The potential field (which is essentially a cost function) is designed in such a way that obstacles have repulsive fields while the destination has an attractive field. The safe path of the UAV is then found by optimizing the carefully selected cost function. To tune this basic philosophy for reactive collision avoidance, a model predictive control-based algorithm has been proposed in the literature. This algorithm essentially assures path following under safe conditions (i.e. if no collision is predicted in the near future) and invokes the potential field function when new collisions are sensed. However, in potential field based techniques the associated optimization process is typically done in an iterative manner. Because of this they are usually computationally intensive and hence are not suitable for reactive collision avoidance of airborne UAVs in general. A promising algorithm in collision avoidance and global path planning is the philosophy of rapidly-exploring random tree (RRT) [1], which has also been used for reactive collision avoidance. However, there are many concerns about the RRT approach, which can largely be attributed to the random nature of the algorithm. For example, the path predicted by RRT is usually a sting of connected straight lines that does not reflect the path followed by a vehicle with nonholonomic constraints. More important, it is a probabilistic approach and hence there is no guarantee of finding a feasible path within a limited finite time. Other graph search algorithms such as best-first search are also implemented for reactive collision avoidance [2]. However, this is not systematic approach and could result in the algorithm searching far too many nodes under some conditions. Moreover, precomputing motion primitives and saving them in a lookup table is infeasible for UAVs, which are usually resource-limited. An interesting perspective to collision avoidance problem is the minimum effort guidance [3], where an optimal control-based approach has been proposed after applying the collision cone philosophy to detect collisions. This method is computationally nonintensive as a closed form solution has been proposed. Even though this is an interesting idea, by minimizing the lateral acceleration, perhaps it imposes unwanted extra constraint on the problem formulation as reactive collision avoidance problems do not necessarily have to be carried out with minimum lateral acceleration. More important, one can observe that this formulation only assures position guarantee and no constraint is imposed on the velocity vector. Hence, even though it guides the vehicle to a carefully selected target point on the safety boundary (we call it the “aiming point”), it causes the vehicle to maneuver until this point. This can be risky as the vehicle may enter the safety ball before reaching the aiming point. Even though the collision cone based aiming point philosophy is a very good idea, the authors of this Note strongly believe that instead of only position guarantee, rather the velocity vector should be aligned towards the aiming point as soon as a collision is detected (which will automatically lead to position guarantee as well). Towards this objective, two new nonlinear guidance laws are proposed in this Note, which are named as nonlinear geometric guidance (NGG) and differential geometric guidance (DGG). These guidance laws are inspired by the philosophy of “aiming point guidance” (APG) [4], which has been proposed in missile guidance literature. It turns out that the APG is a simplified case of the NGG where the associated since function is replaced by its linear approximation (hence, for a systematic discussion, it is renamed as linear geometric guidance (LGG) in this Note). Both of the guidance algorithms proposed in this Note quickly align the velocity vector of the UAValong the aiming point within a part of the available time-togo, which ensures quick reaction and hence safety of the vehicle. The main feature of this philosophy is that they effect high maneuvering at the beginning, causing the velocity vector of the UAV to align with the aiming point direction quickly and then settling along it. Therefore there is no need to maneuver all the way until the aiming point is reached and hence the chance of the UAV entering into the safety ball is minimized. Using the point of closest approach (PCA) [5], the proposed NGG andDGGalgorithms have also been extended for collision avoidance with moving obstacles in both cooperative as well as ignorant scenarios. Mathematical correlations between the guidance laws have also been established, which show that the NGG and DGG are exactly correlated to each other with appropriate gain selections, while the LGG is an approximation of DGG. A “sphere-tracking algorithm” is also proposed in this Note where the UAV is guided to track the surface of the safety spherewhenever a brief violation of the safety boundary occurs after reaching the aiming point because of the location of the next aiming point (which may include the target in Presented as Paper 2010-8315 at the AIAA Guidance, Navigation and Control, Toronto, 2–5August 2010; received 26May 2010; revision received 4 October 2010; accepted for publication 6 October 2010. Copyright © 2010 by Radhakant Padhi. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission. Copies of this paper may be made for personal or internal use, on condition that the copier pay the $10.00 per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923; include the code 0731-5090/11 and $10.00 in correspondence with the CCC. ∗Former Project Assistant, Department of Aerospace Engineering; anushamujumdar87@gmail.com. Associate Professor, Department of Aerospace Engineering; padhi@ aero.iisc.ernet.in. JOURNAL OF GUIDANCE, CONTROL, AND DYNAMICS Vol. 34, No. 1, January–February 2011

Unmanned aerial vehicle (UAV)-enabled vehicular communications in the sixth generation (6G) are characterized by line-of-sight (LoS) and dynamically varying channel conditions. However, the presence of obstacles in the LoS path leads to shadowed fading environments. In UAV-assisted cellular vehicle-to-everything (C-V2X) communication, vehicle and UAV mobility and shadowing adversely impact latency and throughput. Moreover, 6G vehicular communications comprise data-intensive applications such as augmented reality, mixed reality, virtual reality, intelligent transportation, and autonomous vehicles. Since vehicles’ sensors generate immense amount of data, the latency in processing these applications also increases, particularly when the data are not independently identically distributed (non-i.i.d.). Furthermore, when the sensors’ data are heterogeneous in size and distribution, the incoming packets demand substantial computing resources, energy efficiency at the UAV servers and intelligent mechanisms to queue the incoming packets. Due to the limited battery power and coverage range of UAV, the quality of service (QoS) requirements such as coverage rate, UAV flying time, and fairness of vehicle selection are adversely impacted. Controlling the UAV trajectory so that it serves a maximum number of vehicles while maximizing battery power usage is a potential solution to enhance QoS. This paper investigates the system performance and communication disruption between vehicles and UAV due to Doppler effect in the orthogonal time–frequency space (OTFS) modulated channel. Moreover, a low-complexity UAV trajectory prediction and vehicle selection method is proposed using federated learning, which exploits related information from past trajectories. The weighted total energy consumption of a UAV is minimized by jointly optimizing the transmission window (), transmit power and UAV trajectory considering Doppler spread. The simulation results reveal that the weighted total energy consumption of the OTFS-based system decreases up to 10% when combined with federated learning to locally process the sensor data at the vehicles and communicate the processed local models to the UAV. The weighted total energy consumption of the proposed federated learning algorithm decreases by 10–15% compared with convex optimization, heuristic, and meta-heuristic algorithms.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.
\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. 
\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. 
\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. 
\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

This article presents a cooperative multiagent deep reinforcement learning (MADRL) approach for unmanned aerial vehicle (UAV)-aided mobile edge computing (MEC) networks. An UAV with computing capability can provide task offlaoding services to ground Internet of Things devices (IDs). With partial observation of the entire network state, the UAV and the IDs individually determine their MEC strategies, i.e., UAV trajectory, resource allocation, and task offloading policy. This requires joint optimization of decision-making process and coordination strategies among the UAV and the IDs. To address this difficulty, the proposed cooperative MADRL approach computes two types of action variables, namely, message action and solution action, each of which is generated by dedicated actor neural networks (NNs). As a result, each agent can automatically encapsulate its coordination messages to enhance the MEC performance in the decentralized manner. The proposed actor structure is designed based on graph attention networks such that operations are possible regardless of the number of IDs. A scalable training algorithm is also proposed to train a group of NNs for arbitrary network configurations. Numerical results demonstrate the superiority of the proposed cooperative MADRL approach over conventional methods.

Next-generation wireless network should complete integrated sensing, communication, and computation (ISCC) functionalities. To unlock the mutual gains of ISCC, non-orthogonal multiple access (NOMA) and unmanned aerial vehicle (UAV) are appealing techniques that can be invoked. In this work, we design a NOMA-aided UAV ISCC network, where UAV is dispatched as a multi-functional aerial node to perform edge computing for users, whilst radiating integrated sensing and communication beam to sense targets and further offload to a fog node. The objective is to maximize the computation throughput by jointly optimizing UAV trajectory planning, beamforming design and computing resource allocation while satisfying sensing quality, power consumption as well as ultra-reliable low latency communications (URLLC) requirements. We then propose a deep reinforcement learning (DRL)-based algorithm named SAC-TPBD to dynamically modify the UAV trajectory and transmission beamformer according to real-time user queue backlogs and URLLC constraint deviations, thereby realizing URLLC awareness. Evaluations demonstrate that compared to the state-of-the-art methods based on convex approximation techniques, the proposed SAC-TPBD achieves comparable computation performance and beampattern gains with higher implementation efficiency, and the queue backlog can be significantly reduced thanks to the URLLC awareness. Moreover, our approach outperforms baseline DRL schemes in terms of convergence and training stability.