DQN-Based Proactive Trajectory Planning of UAVs in Multi-Access Edge Computing

End-to-end network slicing in unmanned aerial vehicle (UAV) networks has attracted significant attention for delivering differentiated services tailored to the quality of service (QoS) requirements of user terminals (UTs). In a sliced UAV network, the dual mobility of both UTs and UAVs, combined with the three-layer association of UT-UAV-network slice (NS), makes handoff a critical and challenging problem for maintaining the communication performance of UTs. This paper proposes a sequential handoff process where a central controller makes handoff decisions for each UT sequentially according to the UT queue. This process is modeled as a Markov decision process (MDP). In the MDP, the state space includes NS available resources, UTs' QoS requirements, and UT queue information. Additionally, the reward function accounts for UTs' QoS and handoff overhead. The double deep Q-network (DDQN) technique is then employed to design a handoff algorithm. This algorithm achieves the optimization goal of ensuring UTs' QoS requirements while avoiding frequent handoffs by maximizing the total reward of the MDP. Additionally, a state-dependent exploration technique with adaptive intensity adjustment, called noisy network (NoisyNet), is utilized to drive the DDQN for effective exploration, thereby enhancing the performance of the handoff algorithm. Finally, the runtime complexity of the proposed NoisyNet-DDQN-based sequential handoff (NoisyNet-DDQN-SH) algorithm is analyzed. The simulation results show that the NoisyNet-DDQN-SH algorithm converges well. Moreover, it exhibits strong robustness. Additionally, the NoisyNet technique effectively enhances the algorithm's performance, making the NoisyNet-DDQN-SH algorithm superior to the vanilla DDQN (with an ϵ-greedy strategy), greedy, maximum signal-to-interference-plus-noise ratio (MaxSINR), and random algorithms.

Wireless communication is an essential technology to unlock the full potential of unmanned aerial vehicles (UAVs) in numerous applications and has thus received unprecedented attention recently. Although technologies such as direct link, WiFi, and satellite communications are still useful in some remote scenarios where cellular services are unavailable, it is believed that exploiting the thriving 5G and beyond cellular networks to support UAV communications is the most promising and cost-effective approach, especially when the number of UAVs grows dramatically. On the one hand, to guarantee safe and efficient flight operations of multiple UAVs, it is of paramount importance to provide secure and ultra-reliable communication links between the UAVs and their ground pilots or control stations for conveying command and control signals, especially in beyond-visual-line-of-sight (BVLOS) scenarios. On the other hand, because of advances in communication equipment miniaturization as well as UAV manufacturing, mounting compact and lightweight base stations (BSs) or relays on UAVs becomes increasingly feasible. This has led to two promising research paradigms for UAV communications, namely, UAV-assisted cellular communications and cellular-connected UAVs, where UAVs are integrated into cellular networks as aerial communication platforms and aerial users, respectively. As such, integrating UAVs into cellular networks is believed to be a win-win technology for both UAV-related industries and cellular network operators, which not only creates plenty of new business opportunities but also benefits the communication performance of 3-D wireless networks. In addition, UAV related sensing and computing are also helpful for achieving efficient and reliable communication (e.g., in avoiding coverage holes) as well as smart UAV coordination, positioning, and trajectory design. However, 5G and beyond wireless networks with UAVs significantly differs from traditional communication systems, because of the high altitude and high maneuverability of UAVs, the unique UAV-ground channels, the diversified quality of service (QoS) requirements for downlink command and control (C&C) and uplink mission-related data transmission, the stringent constraints imposed by the size, weight, and power (SWAP) limitations of UAVs, as well as the new design degrees of freedom enabled by joint UAV mobility control and communication resource allocation.

Joint modulation-coding scheme (MCS) adaptation and resource block (RB) allocation is an effective approach to guarantee different quality of service (QoS) requirements of all UEs under dynamic network environments. In this paper, we consider a fifth generation (5 G) cellular network with time-varying wireless channels, in which the BS serves multiple user equipments (UEs) under limited available RBs. We aim to minimize the total RB consumption subject to the rigorous constraints of each UE's QoS requirement. To attain this objective, this paper puts forth an online learning technique, referred to as integrated Deep Reinforcement learning and stable Matching (DeepRM), in the sense that the MCS adaptation and RB allocation decisions are conducted without acquiring the real-time channel quality indicator (CQI) feedback. DeepRM is a closed-loop framework, where the output of deep reinforcement learning (DRL) is imported into the stable matching to guide optimal RB allocation whilst the output of stable matching is fed into the DRL framework to assist efficient MCS decision-making. Specifically, in DeepRM, we first develop a powerful DRL algorithm, termed as Action-and-Reward Branching Deep Q-network (ARBDQ), by incorporating the action branch architecture into conventional DRL and modifying the traditional deep neural network training mechanism, to perform judicious MCS decisions on different links in parallel. Then, a new many-to-one stable matching algorithm, called adaptive deferred acceptance, is exploited to dynamically adjust the RB quota of each UE in a computationally efficient fashion. Simulation results demonstrate that compared with ACO-HM, OLLA-ADA, and ARBDQ-Random algorithms, DeepRM induces much less RB consumption while guaranteeing the QoS requirements of all UEs in various network scenarios. Furthermore, under miscellaneous QoS requirement, number of UEs, and CQI reporting period setups, DeepRM is more robust than other baselines.

This paper proposes a method for the trajectory planning of an unmanned aerial vehicle (UAV) acting as an aerial base station of a communication network. The UAV base station provides wireless communication for moving ground vehicles with different quality-of-service (QoS) requirements. The ground vehicles follow locally planned trajectories, which are communicated to the aerial base station. The UAV possesses a network estimator which determines the channel properties between the UAV and each ground vehicle. Whenever the QoS deceeds a threshold the UAV trajectory is changed based on this estimate by a trajectory planning unit in a way to fulfil all individual QoS requirements of the ground vehicles if possible. In contrast to literature, this approach considers the communication relay between moving vehicles with different QoS requirements. The performance of the method is demonstrated in a simulation study.

Multi-access edge computing (MEC) is an emerging computing paradigm that brings services from the centralized cloud to nearby network edge to improve users' Quality of Experience (QoE). As massive services with dynamic Quality of Service (QoS) are available in MEC, it becomes challenging for users to find reliable services that satisfy their needs. Therefore, service recommendation technology is urgently needed in MEC. Although existing service recommendation methods work well on recommending popular services that users might be interested in, they fail to recommend services with reliable QoS in the MEC environment. To tackle this issue, an accurate and reliable service recommendation (ARSR) approach based on bilateral perception is proposed, which aims to proactively recommend reliable services by perceiving both users' service demands and multi-QoS of candidate services. ARSR consists of three main steps. First, a user's service demand is estimated by a context-aware service demand prediction method based on an improved online deep learning model. Then, multiple QoS attributes of candidate services are forecasted by a multidimensional contexts-aware QoS prediction method based on an improved multi-task deep neural network. Finally, the optimal service is recommended to the user based on the predicted QoS. Extensive experiments have been carried out to verify the proposed approach and to prove its performance superiority.

In the current unmanned aircraft systems (UASs) for sensing services, unmanned aerial vehicles (UAVs) transmit their sensory data to terrestrial mobile devices over the unlicensed spectrum. However, the interference from surrounding terminals is uncontrollable due to the opportunistic channel access. In this paper, we consider a cellular Internet of UAVs to guarantee the Quality-of-Service (QoS), where the sensory data can be transmitted to the mobile devices either by UAV-to-Device (U2D) communications over cellular networks, or directly through the base station (BS). Since UAVs' sensing and transmission may influence their trajectories, we study the trajectory design problem for UAVs in consideration of their sensing and transmission. This is a Markov decision problem (MDP) with a large state-action space, and thus, we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space, and then propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a higher total utility than policy gradient algorithm and single-agent algorithm.

Vehicular networks are envisioned to deliver data transmission services ubiquitously, especially in the upcoming autonomous driving era. Accordingly, the high data traffic load poses a heavy burden to the terrestrial network infrastructure. Unmanned Aerial Vehicles (UAVs) show enormous potential to assist vehicular networks in providing services. In a dense UAV-assisted vehicular network with a large number of users, spectrum sharing is leveraged for alleviating the spectrum scarcity. However, the increasing data traffic still leads to the UAV energy consumption problem. This paper considers a specific network scenario where a UAV transmits its cached content files to vehicular users over UAV-to-vehicle (U2V) links while vehicle-to-vehicle (V2V) links reuse the U2V spectrum for safety-critical message exchanges. To improve the UAV's energy efficiency while guaranteeing users' quality of service (QoS), we jointly optimize content placement, spectrum allocation, co-channel link pairing, and power control, which are the key factors affecting energy efficiency and QoS. The joint optimization problem is formulated as a mixed-integer nonlinear programming (MINLP) problem, which is solved by combining Hungarian and DDQN. We perform system performance evaluations, demonstrating that our approach can not only improve the UAV's energy efficiency while satisfying the users' QoS requirements but also increase the timeliness of making decisions.

A survey on UAV-assisted wireless communications: Recent advances and future trends

A Framework to Facilitate Network Operators to Provide Better Quality of Service in Mobile Networks

Unmanned aerial vehicles (UAVs) are playing an important role in serving ground users as base stations (BSs) during temporary events, or after terrestrial BSs fail. UAV deployment is a fundamental and vital problem which directly affects the coverage capacity and user experience. Especially, while deploying a UAV backbone network, the cost and the quality of service (QoS) of the ground users as well as the connectivity among UAVs need to be considered jointly. In this paper, the deployment problem is modeled as minimizing the amount of deployed UAVs while guaranteeing the coverage capacity and connectivity among UAVs. Further, a novel UAV deployment scheme is proposed where a heuristic algorithm is adopted to delete locations from the initially defined candidate UAV locations iteratively, and UAVs are deployed at the remaining locations finally. The simulation results show that our UAV deployment scheme can achieve minimum deployed UAVs while ensuring QoS requirements and a robust network among UAVs.

In recent years, using the Unmanned Aerial Vehicle (UAV) as a Base Stations (BS) to cover users in wireless networks has increased dramatically. One of the main goals of integrating UAVs into wireless networks is to deploy UAVs in such a way that user expectations are met with the fewest number of UAVs. To achieve this aim, the coverage area of each UAV should include as many users as possible. Furthermore, the resources assigned to the backhaul links for such UAV deployments must fulfill the requirements of users served by each UAV. In this paper the goal is to position the least number of UAVs in a 3D position to cover cellular network users. To provide appropriate quality of service, we consider a maximum path loss allowed for the network. The path loss of potential links is affected by the propagation environment and might vary depending on network structure. To reflect this uncertainty, path loss is expressed as a random variable with a probability distribution based on environmental characteristics. As a result, we're dealing with an optimization problem with uncertain information. We use stochastic programming to work with uncertain information and formulate the UAV positioning and data rate assignment problem. The implementation results of our proposed mixed-binary linear mathematical model and Monte Carlo simulation in various scenarios show its optimum performance in different dimensions.

Unmanned Aerial vehicles (UAVs) are widely used as network processors in mobile networks, but more recently, UAVs have been used in Mobile Edge Computing as mobile servers. However, there are significant challenges to use UAVs in complex environments with obstacles and cooperation between UAVs. We introduce a new multi-UAV Mobile Edge Computing platform, which aims to provide better Quality-of-Service and path planning based on reinforcement learning to address these issues. The contributions of our work include: 1) optimizing the quality of service for mobile edge computing and path planning in the same reinforcement learning framework; 2) using a sigmoid-like function to depict the terminal users’ demand to ensure a higher quality of service; 3) applying synthetic considerations of the terminal users’ demand, risk and geometric distance in reinforcement learning reward matrix to ensure the quality of service, risk avoidance, and the cost-savings. Simulations have shown the effectiveness and feasibility of our platform, which can help advance related researches. The source code can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bczhangbczhang</uri> .

A third‐generation partnership project long‐term evolution system uses the concept of a 2‐tier heterogeneous network, where low‐power and short‐range femtocells are laid under macrocells to fulfill the quality of service (QoS) requirements of users and to boost overall network throughput. However, cochannel deployment gives rise to the problem of cross‐tier interference that significantly deteriorates the performance of the cellular network. The coordinated multipoint (CoMP) transmission scheme is proposed as a promising solution to alleviate cross‐tier interference by doing cooperation among the neighbor base stations. In this paper, we propose a QoS priority‐based coordinated scheduling and hybrid spectrum access (QoS‐CSaHSA) scheme for downlink CoMP transmission in 2‐tier networks. The proposed QoS‐CSaHSA scheme dynamically reduces the femtocells' power requirements by considering the neighbor cell interference, and it also balances the macrocell load by switching the femtocells to HA mode. Moreover, it can reduce unnecessary muting of the base stations because it only enables the coordinated scheduling CoMP when there is no other possibility to reduce the cross‐tier interference. System‐level simulation results prove the validity of the proposed QoS‐CSaHSA scheme in a heterogeneous network environment. Compared with the existing schemes that operate the femtocells in a closed‐subscriber group without considering users' QoS requirements, the proposed scheme almost doubles the cell‐edge user throughput and reduces the packet loss rate and call‐blocking probability.

With high mobility and low cost, unmanned aerial vehicles (UAVs) are widely used in wireless communication systems. Especially in emergencies, UAVs can be used as aerial base stations (BSs) to provide wireless communication services for ground users. Aiming to reduce cost, we prefer to minimizing the number of UAVs needed to serve all users. Compared with the existing works, we take the constraints of required quality of service (QoS) and the service ability of each UAV into consideration. To solve the formulated mixed-integer programming problem, we propose a three-step method. First, to ensure each UAV can serve more users, the maximum service radius of UAVs is derived according to users’ QoS requirement. Second, we propose an artificial bee colony (ABC) algorithm based clustering method to cluster users into different groups in the horizontal direction. Third, we adjust the positions of UAVs to obtained a better communication performance of the wireless communication system. Finally, the simulation results are presented to demonstrate the superiority of the proposed method.KeywordsWireless communicationUnmanned aerial vehiclesAerial base stationsThree-dimensional deployment

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.