Deep Reinforcement Learning-Based Resource Allocation in Cooperative UAV-Assisted Wireless Networks

Integrating autonomous unmanned aerial vehicles (UAVs) with fifth-generation (5G) networks presents a significant challenge due to network interference. UAVs’ high altitude and propagation conditions increase vulnerability to interference from neighbouring 5G base stations (gNBs) in the downlink direction. This paper proposes a novel deep reinforcement learning algorithm, powered by AI, to address interference through power control. By formulating and solving a signal-to-interference-and-noise ratio (SINR) optimization problem using the deep Q-learning (DQL) algorithm, interference is effectively mitigated, and link performance is improved. Performance comparison with existing interference mitigation schemes, such as fixed power allocation (FPA), tabular Q-learning, particle swarm optimization, and game theory demonstrates the superiority of the DQL algorithm, where it outperforms the next best method by 41.66% and converges to an optimal solution faster. It is also observed that, at higher speeds, the UAV sees only a 10.52% decrease in performance, which means the algorithm is able to perform effectively at high speeds. The proposed solution effectively integrates UAVs with 5G networks, mitigates interference, and enhances link performance, offering a significant advancement in this field.

Recent developments in unmanned aerial vehicle (UAV) technology have given UAVs more processing and storage resources, paving the way for the concept of edge computing-enabled UAV networks. In this paper, we propose a cooperative multi-agent reinforcement learning-based computation offloading framework for a UAV swarm. Flying UAVs with missions can offload part of their tasks to neighboring UAVs or to fixed edge servers at terrestrial base stations. This approach reduces the total energy consumption of all devices during a core mission. Our framework helps UAVs form stable sequences of offloading decisions under uncertainties in a dynamic environment. This study demonstrates the superiority of the proposed deep Q-learning (DQN) algorithm to the existing Q-learning, heuristic, and random decision-making algorithms.

In the last few years, a considerable amount of research effort has been put into creating networks of Unmanned Aerial Vehicles (UAVs). Flying Ad Hoc Networks (FANETs) and Internet of Drones (IoDs) are the result of such efforts. FANETs assume that the member UAV nodes belong to different users. However, in UAV networks that would find their applications in areas such as agriculture, environment monitoring, crowd surveillance and natural disaster situations, all UAVs would belong to the same organization, creating a private UAV network. The number of UAVs used in private UAV networks is relatively small compared to that of FANETs due to the cost. If the packet routing approaches designed for FANETs are used in such networks, the quality of video transmitted through the network will suffer. This is because, while there can be many neighbors for each UAV in a FANET, the movements of UAVs in a private UAV network may cause some UAVs not to have any neighbors. To overcome this problem, a UAV in such a network needs to be controlled differently and an appropriate method of selecting the next hop node is required. The new approach presented in this paper uses mathematical analysis to find the optimum next hop node in a private UAV network by reducing a global optimization problem to a local optimization problem at each UAV node along the path from the source to the destination. Once the best next hop node is found, the UAVs can be controlled accordingly.

This paper proposes a new methodology to address the image quality problem encountered as the use of an unmanned aerial vehicle (UAV) in the field of bridge inspection increased. When inspecting a bridge, the image obtained from the UAV was degraded by various interference factors such as vibration, wind, and motion of UAV. Image quality degradation such as blur, noise, and low-resolution is a major obstacle in utilizing bridge inspection technology based on UAV. In particular, in the field of bridge inspection where damages must be accurately and quickly detected based on data obtained from UAV, these quality issues weaken the advantage of using UAVs by requiring re-take of images through re-flighting. Therefore, in this study, image quality assessment (IQA) based on local blur map (LBM) and image quality enhancement (IQE) using the variational Dirichlet (VD) kernel estimation were proposed as a solution to address the quality issues. First, image data was collected by setting different camera parameters for each bridge member. Second, a blur map was generated through discrete wavelet transform (DWT) and a new quality metric to measure the degree of blurriness was proposed. Third, for low-quality images with a large degree of blurriness, the blind kernel estimation and blind image deconvolution were performed to enhance the quality of images. In the validation tests, the proposed quality metric was applied to material image sets of bridge pier and deck taken from UAV, and its results were compared with those of other quality metrics based on singular value decomposition (SVD), sum of gray-intensity variance (SGV) and high-frequency multiscale fusion and sort transform (HiFST) methods. It was validated that the proposed IQA metric showed better classification performance on UAV images for bridge inspection through comparison with the classification results by human perception. In addition, by performing IQE, on average, 26% of blur was reduced, and the images with enhanced quality showed better damage detection performance through the deep learning model (i.e., mask and region-based convolutional neural network).

Introduction Autonomous unmanned aerial vehicles (UAVs) are considered with increasing interest in a diverse set of civil and commercial applications, such as environmental and natural disaster monitoring, border surveillance, emergency assistance, search and rescue missions, delivery of goods, and construction [208, 132, 362, 265, 434, 110, 280, 163]. Use of single or multiple UAVs as communication relays or aerial base stations for network provisioning in emergency situations and for public safety communications has been of particular interest due to UAVs fast deployment and large coverage capabilities [331, 193, 206, 429]. The deployed aerial platform size and the number of necessary platforms are key factors in determining the on-board communication system in addition to the coverage and capacity requirements [73]. Small-scale multi-rotors are of particular interest in practice due to their ease of deployment, low acquisition and maintenance costs, high maneuverability, and ability to hover. Research and development of small-scale UAVs focused initially on control issues, such as flight stability, maneuverability, and robustness, followed by designing autonomous vehicles capable of waypoint flights with minimal user intervention. Considering the limited flight time and payload capacity of these vehicles and the area sizes the aforementioned applications might span, the interest is shifting more and more toward collaborative multiple UAV systems for efficient and successful missions. The number of UAVs might be from tens to hundreds traveling over tens of meters to kilometers for different applications (see Figure 3.1 for an illustration). Wireless communication is an essential factor not only for network provisioning with UAVs but also for successful deployment of systems consisting of multiple small-scale UAVs. It is highly likely that high-performance links and connectivity in three-dimensional space will be required for several applications (e.g., monitoring certain areas) with data delivery meeting a certain quality of service demands [35].The question arises as to which wireless technology should be employed for aerial networks that consist of air-to-ground and air-to-air links, and data needs to be delivered regardless of significant height and orientation differences. Taking into account the quality of service to be satisfied over diverse links and the high node mobility, it is not yet clear as to whether networking protocols developed for ground networks are readily deployable on UAV networks.

Deploying wireless routers on the ground and unmanned aerial vehicles (UAVs) in the air is believed to be a fast and efficient approach to providing the emergency communication service to disaster areas. The network lifetime is restricted to the lifetime of mesh networks of routers that are scattered across a complex disaster environment. We consider the problem of multi-UAVs relaying and moving with the goal of maximizing the network lifetime. UAVs must learn where to move in order to relay messages from the routers efficiently. However, under the dynamics of the router traffic, that is, the uncertainty of the environment, it is challenging for UAVs to find a movement mechanism that maximizes the network lifetime. By embracing the on-policy reinforcement learning algorithm Sarsa, we are able to demonstrate a greedy movement policy. Specifically, we study the trajectory planning of multiple UAVs in a dense wireless router mesh networks (WMNs) on the ground. UAVs can learn the unknown environment by a little movement of UAVs in each step, in which communication connections between routers with UAVs are retained. A Q-table of movement actions and environment states is formed after multiple attempts of movements. Simulation results show the proposal effectiveness comparing with other methods.

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.

In modern conditions unmanned aerial vehicles (UAVs) generate new classes of threats, including their use for terrorist purposes. A feature of modern UAVs is the ability to perform sudden maneuvers and to keep the same position in the point in space. For the description of the UAV movement with various types of maneuver it is used a rectangular coordinate system. We use the model in the form of stochastic dynamic system with random structure in the discrete time in which the change type UAV movement occurs at random times. When a UAV emits a sign, its location can be determined by wireless sensor networks (WSN) using the TDOA method. On the basis of a mathematical apparatus of the mixed Markov processes for in discrete time optimal and quasi-optimal adaptive algorithms for filtration of UAV movement parameters based on the TDOA-measurement, sensor networks are synthesized. Devices that realize these algorithms are multichannel and belong to the class of devices with feedback between channels. At the same time, in a quasi-optimal algorithm, a sequential procedure of the arriving measurements from sensors of a sensor network is realized, which allows to avoid the inversion of large-dimensional matrices. An analysis of the quasi-optimal adaptive algorithm is performed using statistical modeling. On the intervals of hovering and of the UAV movements without maneuver, the developed algorithm allows to increase significantly the accuracy of the estimation of the UAV coordinates, and also to recognize various types of its movement with high probability level.

Wireless communication with unmanned aerial vehicle (UAV) has attracted much attention due to its flexibility and low-cost in providing wireless connectivity. In this article, we investigate the spectrum sharing between UAV and terrestrial wireless communication. Two scenarios are considered in this work. Scenario 1: the UAV is energy sufficient to transmit the required data. In this case, our objective is to minimize the time that is needed to complete data transmission. Scenario 2: the battery of the UAV is energy insufficient. In this case, the goal becomes maximizing the size of data that can be transmitted. Different scenarios require different placement of the UAV. When the UAV flies towards the primary user (PU), the sensing result will be more accurate and more opportunities can be utilized to transmit data. However, the interference to primary receiver (PR) and the propulsion energy due to UAV's movement will be increased. In this article, the sensing performance, the power allocation and the UAV's position are jointly optimized to satisfy different requirements of the UAV system. To efficiently utilize the licensed spectrum, a hybrid transmission mode is proposed for the UAV system, in which sensing and power control are jointly employed to protect the PR. Then, efficient iterative algorithms are proposed to solve the optimization problems. Simulations results show that proposed designs can significantly reduce the time that is required to complete data transmission in energy sufficient scenario and enhance the size of data that can be transferred in energy insufficient scenario.

In this paper, we investigate the coverage performance of a reference user equipment (UE) in a finite network of multiple three-dimensional (3D) mobile unmanned aerial vehicles (UAVs). Specifically, following the general theory on mobile ad hoc networks (MANETs), we propose a mixed mobility (MM) model which characterizes the movement process of a UAV in the 3D cylindrical region. In this paper, we invoke the random waypoint mobility (RWPM) and uniform mobility (UM) models to represent the movement of a UAV in vertical and spatial directions, respectively. By employing this MM model, we analyze the coverage probability of a reference UE in a finite network of multiple UAVs under the uniform UAV association and closest UAV association policies in one snapshot. According to the former policy, the UE connects to a random UAV while in the latter policy it connects to the closest UAV. Furthermore, all the non-serving UAVs act as the interferers to UE. To facilitate coverage probability analysis, we determine the serving and interfering UAVs-to-UE distance distributions under two association policies for Nakagami- ${m}$ fading. We quantify the coverage performance of UE for various system and mobility parameters to exemplify the closest UAV association policy. Moreover, the analytical results are verified through simulations.

To archive diverse kinds of goal, multiple unmanned aerial vehicles (UAVs) need to cooperate each other. Therefore, the importance of a ground station becomes more important. This paper proposes a framework of the ground station that controls multiple and hetero UAVs. The framework contains multiple UAV control modules and UI interfaces and has the flexibility for adding diverse kinds of UAVs and embedded devices such as commercial UAVs, Arduino-based UAVs, cameras and storage devices.

Driven by the recent advancement in unmanned aerial vehicle (UAV) technology, this paper proposes a new wireless network architecture of coordinate multipoint (CoMP) in the sky to harness both the benefits of interference mitigation via CoMP and high mobility of UAVs. Specifically, we consider uplink communications in a multi-UAV enabled multi-user system, where each UAV forwards its received signals from all ground users to a central processor (CP) for joint decoding. Moreover, we consider the case where the users may move on the ground, thus the UAVs need to adjust their locations in accordance with the user locations over time to maximize the network throughput. Utilizing random matrix theory, we first characterize, in closed form, a set of approximated upper and lower bounds of the user's achievable rate in each time epoch under the practical Rician fading channel model, which is shown to be very tight, both analytically and numerically. UAV placement and movement over different epochs are then optimized based on the derived bounds to maximize the minimum of user average achievable rates over all epochs for both cases of full information (of current and future epochs) and current information on the user's movement. Interestingly, it is shown that the optimized location of each UAV at any particular epoch is the weighted average of the ground user locations at the current epoch as well as its own location at the previous and/or next epoch. Finally, simulation results are provided to validate and compare the performance of the proposed UAV placement and movement designs under different practical application scenarios.

Unmanned aerial vehicles (UAVs) have a significant potential for sensing applications in further cellular networks due to their extensive coverage and flexible deployment. In this paper, we consider a multi-cell cellular network with a cellular-connected UAV, which senses data with onboard sensors and uploads sensory data to the ground base stations (BSs). To evaluate the freshness of sensory data, we employ the concept of age of information (AoI), which is defined as the time elapsed since the latest successful transmission of sensory data. A lower AoI implies fresher sensory data, which may lead to the increase of UAV operation time. To balance such tradeoff, we aim to minimize the weighted sum of operation time and total AoI for the UAV by jointly optimizing transmission scheduling, BS association, as well as UAV trajectory. The problem is formulated as a mixed-integer nonlinear programming (MINLP) problem, which is difficult to solve due to the time-varying propagation channels. To this end, we first characterize the average communication performance with statistic channel information, and then develop a search algorithm to obtain the optimal solution via employing the optimal structure as well as convex optimization techniques, while a low-complexity Double Graph based Algorithm (DGA) is developed to obtain a suboptimal solution. Then, by taking into account the site-specific performance and making fast decisions online, we propose a Deep reinforcement Learning Algorithm (DLA). Compared to DGA, DLA can adapt to the specific local environment and obtain a solution more rapidly once the training process is completed. Simulation results show that the proposed algorithms outperform the benchmarks about 30%, and achieve flexible tradeoff between operation time and AoI of UAV sensing, which is not available by considering just one objective.

Currently, the problem of ensuring safety of flights of unmanned aerial vehicles (UAVs) over the territory of transport infrastructure facilities (TIF), primarily airports, remains topical.In one of the previous works, the author together with the co-author proposed a method for increasing safety of movement of unmanned aerial vehicles and the system for controlling the routes of unmanned aerial vehicles (hereinafter – route control system, RCS) that implements it, which makes it possible to improve safety and security of UAV traffic at transport infrastructure facilities by limiting UAV traffic area strictly to a dedicated air corridor (DAC). The development of this system creates the prerequisites for removing the existing restrictions on the use of unmanned aerial vehicles at transport infrastructure facilities.For practical implementation of the proposed system, it is relevant to develop a method of placing RCS at transport infrastructure facilities. This condition can be justified by the fact that RCS, as a rule, will be located under conditions of dense infrastructural development, including dangerous technical elements of technical equipment, a collision of a UAV with which can lead to an emergency situation (ES); besides, the movement of air/ground vehicles will be carried out in the immediate vicinity of RCS along transport routes/corridors, and employees, passengers and visitors of TIF will move along the pedestrian paths.The objective of this study – to develop a methodology for placing systems controlling routes of unmanned aerial vehicles at transport infrastructure facilities.The study conducted with well-known scientific methods, including the basic routing problem posed by Dantzig and Ramser, modelling, analysis, and synthesis, made it possible to develop a method for placing RCS for unmanned aerial vehicles at transport infrastructure facilities. The practical application of the proposed methodology makes it possible to build routes for movement of UAV at TIF, to form a network of dedicated air corridors for UAV at TIF operated in relationship with the system, determine the optimal location of the RCS elements at TIF.

Information acquisition and rescue operation management can be facilitated by the use of unmanned aerial vehicles (UAVs) equipped with cameras and sensors that are capable of aerial photography and acquire important information from disaster-stricken areas. UAV technology is a relatively low-cost, flexible, and safe solution in missions that are difficult for humans or other systems. UAV usage in pre-and post-disaster assessment, response, and management is an active area of research. There are two important technical aspects of UAV applications in disaster situations. The first is the limited communication range. Information acquired and photography produced by UAVs are only transmitted in real-time to the ground control station (GCS) within its communication range. Remote real-time monitoring of distant areas beyond the UAV's communication range is not possible using a single UAV. It is only possible through communication among several UAVs. The second is the relatively short flight time due to limited battery capacity. UAV replacement is required in order to carry out a persistent monitoring mission. In this paper, we propose a new remote real-time monitoring system of a distant disaster area using multiple UAVs. Wireless communications between multiple UAVs constitutes a UAV network in our proposed system. Moreover, the proposed system replaces the UAVs in-flight with standby UAVs according to the remaining energy levels. We compare the performance of the proposed system with an existing UAV system. Our simulation results show that the proposed system can achieve efficient remote real-time monitoring.