Enabling Efficient Scheduling in Large-Scale UAV-Assisted Mobile-Edge Computing via Hierarchical Reinforcement Learning

With the growing demands of mobile IoT applications, unmanned aerial vehicles (UAVs) have attracted increasing attention for their flexibility and mobility, and are being widely employed as aerial edge servers in mobile edge computing (MEC) systems to extend coverage and improve adaptability in dynamic environments. This paper jointly optimizes task offloading, resource allocation, and UAV trajectory planning in a multi-UAV-assisted MEC system, aiming to minimize total energy consumption and delay. To cope with environmental dynamics, reinforcement learning (RL) is employed. Specifically, we propose an Option mechanism based on hierarchical RL, enabling agents to leverage knowledge from peers before decisionmaking, which enhances exploration efficiency and accelerates convergence. Additionally, mean field theory is integrated with neural networks to approximate the collective influence of numerous agents, reducing interaction complexity in large-scale systems. Experimental results verify the effectiveness and scalability of the proposed framework.

Unmanned Aerial Vehicle (UAV)-enabled mobile edge computing has been proposed as an efficient task-offloading solution for user equipments (UEs). Nevertheless, the presence of heterogeneous UAVs makes centralized navigation policies impractical. Decentralized navigation policies also face significant challenges in knowledge sharing among heterogeneous UAVs. To address this, we present the soft hierarchical deep reinforcement learning network (SHDRLN) and dual-end federated reinforcement learning (DFRL) as a decentralized navigation policy solution. It enhances overall task-offloading energy efficiency for UAVs while facilitating knowledge sharing. Specifically, SHDRLN, a hierarchical DRL network based on maximum entropy learning, reduces policy differences among UAVs by abstracting atomic actions into generic skills. Simultaneously, it maximizes the average efficiency of all UAVs, optimizing coverage for UEs and minimizing task-offloading waiting time. DFRL, a federated learning (FL) algorithm, aggregates policy knowledge at the cloud server and filters it at the UAV end, enabling adaptive learning of navigation policy knowledge suitable for the UAV's performance parameters. Extensive simulations demonstrate that the proposed solution not only outperforms other baseline algorithms in overall energy efficiency but also achieves more stable navigation policy learning under different levels of heterogeneity of different UAV performance parameters.

In this paper, we propose a secure unmanned aerial vehicle (UAV) mobile edge computing (MEC) system where multiple ground users offload large computing tasks to a nearby legitimate UAV in the presence of multiple eavesdropping UAVs with imperfect locations. To enhance security, jamming signals are transmitted from both the full-duplex legitimate UAV and non-offloading ground users. For this system, we design a low-complexity iterative algorithm to maximize the minimum secrecy capacity subject to latency, minimum offloading and total power constraints. Specifically, we jointly optimize the UAV location, users' transmit power, UAV jamming power, offloading ratio, UAV computing capacity, and offloading user association. Numerical results show that our proposed algorithm significantly outperforms baseline strategies over a wide range of UAV self-interference (SI) efficiencies, locations and packet sizes of ground users. Furthermore, we show that there exists a fundamental tradeoff between the security and latency of UAV-enabled MEC systems which depends on the UAV SI efficiency and total UAV power constraints.

Unmanned Aerial vehicles (UAVs) are widely used as network processors in mobile networks, but more recently, UAVs have been used in Mobile Edge Computing as mobile servers. However, there are significant challenges to use UAVs in complex environments with obstacles and cooperation between UAVs. We introduce a new multi-UAV Mobile Edge Computing platform, which aims to provide better Quality-of-Service and path planning based on reinforcement learning to address these issues. The contributions of our work include: 1) optimizing the quality of service for mobile edge computing and path planning in the same reinforcement learning framework; 2) using a sigmoid-like function to depict the terminal users’ demand to ensure a higher quality of service; 3) applying synthetic considerations of the terminal users’ demand, risk and geometric distance in reinforcement learning reward matrix to ensure the quality of service, risk avoidance, and the cost-savings. Simulations have shown the effectiveness and feasibility of our platform, which can help advance related researches. The source code can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bczhangbczhang</uri> .

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

This paper studies the trajectory control and task offloading (TCTO) problem in an unmanned aerial vehicle (UAV)-assisted mobile edge computing system, where a UAV flies along a planned trajectory to collect computation tasks from smart devices (SDs). We consider a scenario that SDs are not directly connected by the base station (BS) and the UAV has two roles to play: MEC server or wireless relay. The UAV makes task offloading decisions online, in which the collected tasks can be executed locally on the UAV or offloaded to the BS for remote processing. The TCTO problem involves multi-objective optimization as its objectives are to minimize the task delay and the UAV's energy consumption, and maximize the number of tasks collected by the UAV, simultaneously. This problem is challenging because the three objectives conflict with each other. The existing reinforcement learning (RL) algorithms, either single-objective RLs or single-policy multi-objective RLs, cannot well address the problem since they cannot output multiple policies for various preferences (i.e. weights) across objectives in a single run. An evolutionary multi-objective RL (EMORL) algorithm is applied to address the TCTO problem. We improve the multi-task multi-objective proximal policy optimization of the original EMORL by retaining all new learning tasks in the offspring population, which can preserve promissing learning tasks. The simulation results demonstrate that the proposed algorithm can obtain more excellent non-dominated policies by striking a balance between the three objectives regarding policy quality, compared with two evolutionary algorithms, two multi-policy RL algorithms, and the original EMORL.

Accurate and robust state estimation is critical for the heterogeneous agent systems, particularly when considering the challenges posed by Unmanned Aerial Vehicles (UAVs) operating in perceptually-degraded environments where access to Global Navigation Satellite System (GNSS) signals is lost. We can, however, actively increase the amount of optimal localization available to UAV by augmenting them with a small number of more expensive, but less resource-constrained, heterogeneous agents. In this paper, we propose a novel detection, localization, and tracking framework for UAV based on LiDAR. First, we present an innovative approach that integrates range image projection and Depth Cluster of LiDAR point clouds with UAV technology. Subsequently, we devise a multidimensional feature probability detection and tracking evaluation function, enabling the detection, estimation, and active tracking of UAV movement. Finally, we conduct comprehensive experiments using heterogeneous agent systems to assess the effectiveness and robustness of the developed framework. The experiments reveal a minimum 20% reduction in running time and an average localization accuracy error of 1.98[Formula: see text]cm.

This article establishes a new multiunmanned aerial vehicle (multi-UAV)-enabled mobile edge computing (MEC) system, where a number of unmanned aerial vehicles (UAVs) are deployed as flying edge clouds for large-scale mobile users. In this system, we need to optimize the deployment of UAVs, by considering their number and locations. At the same time, to provide good services for all mobile users, it is necessary to optimize task scheduling. Specifically, for each mobile user, we need to determine whether its task is executed locally or on a UAV (i.e., offloading decision), and how many resources should be allocated (i.e., resource allocation). This article presents a two-layer optimization method for jointly optimizing the deployment of UAVs and task scheduling, with the aim of minimizing system energy consumption. By analyzing this system, we obtain the following property: the number of UAVs should be as small as possible under the condition that all tasks can be completed. Based on this property, in the upper layer, we propose a differential evolution algorithm with an elimination operator to optimize the deployment of UAVs, in which each individual represents a UAV's location and the entire population represents an entire deployment of UAVs. During the evolution, we first determine the maximum number of UAVs. Subsequently, the elimination operator gradually reduces the number of UAVs until at least one task cannot be executed under delay constraints. This process achieves an adaptive adjustment of the number of UAVs. In the lower layer, based on the given deployment of UAVs, we transform the task scheduling into a 0-1 integer programming problem. Due to the large-scale characteristic of this 0-1 integer programming problem, we propose an efficient greedy algorithm to obtain the near-optimal solution with much less time. The effectiveness of the proposed two-layer optimization method and the established multi-UAV-enabled MEC system is demonstrated on ten instances with up to 1000 mobile users.

Joint energy and throughput optimization for MEC-enabled multi-UAV IoRT networks

This chapter studies mobile edge computing (MEC) networks assisted by unmanned aerial vehicles (UAVs). According to the application scenarios, we consider three roles for UAVs in MEC networks: exploiting MEC computing capabilities, serving as a computing server, and serving as a relay for computation offloading. Furthermore, the details for resource allocation and optimization are presented in the three scenarios of UAV-assisted MEC networks. In addition, we focus on the situation in which a UAV not only functions as an MEC server to inspect turbines on a wind farm, but also performs task computation. To facilitate wide applications of UAV-assisted MEC in practice, this chapter highlights the main implementation issues of UAV-assisted MEC, including optimal UAV deployment, wind models, and joint trajectory–computation performance optimization.

Due to the exponential expansion of the Internet of Things (IoT) technology, the increasing amount of data generated by numerous IoT devices has become a significant challenge for communication and computing networks. On the other hand, the concept of Mobile Edge Computing (MEC) presents a viable solution as it brings computing, storage, and networking capabilities in close proximity to the user, enabling the hosting of applications that require significant computation power and low latency right at the network’s edge. In addition, the maneuverability, cost-effectiveness, and ease of deployment make Unmanned Aerial Vehicles (UAVs) highly versatile wireless platforms. Capitalizing on the benefits of MEC and UAVs, this paper proposes a UAV-assisted MEC system that offloads services to reduce the computation burden in IoT. The objective is to maximize the task completion rate, considering factors such as user transmit power, task offloading rate, and UAV trajectory variables. To tackle this optimization problem, this paper devises a method based on the Deep Deterministic Policy Gradient (DDPG), an algorithm for continuous action spaces in deep reinforcement learning. The numerical simulations show the effectiveness of the proposed approach in comparison to other benchmarks, i.e., deep Q-network (DQN) and full offloading methods. In a simulation, the proposed DDPG method can achieve a fully optimized system with a 100% task success rate, while the DQN-based method achieves a lower task success rate of 95.93% and the full offloading method only obtains a task success rate of 82.73%.

Unmanned Aerial Vehicles (UAVs) have traditionally served as network processors in mobile networks, and more recently, as mobile servers in Mobile Edge Computing (MEC). However, deploying UAVs in dynamic and obstacle-rich environments while ensuring efficient coordination among multiple UAVs presents significant challenges. To tackle these challenges, we propose a unified reinforcement learning framework for a multi-UAV MEC platform that enhances Quality-of-Service (QoS) and optimizes path planning. Specifically, our contributions include: (1) Integrating QoS optimization and UAV path planning into a unified reinforcement learning framework; (2) Modeling terminal users' demands with a sigmoid-like function to enhance service quality; (3) Incorporating terminal users' demand, risk factors, and geometric distance into the reinforcement learning reward matrix for balancing service quality, risk mitigation, and cost efficiency. Experimental results demonstrate the framework’s effectiveness: simulations in 3D environments show a 20–35% reduction in collision incidents compared to greedy algorithms and a 15–25% improvement in demand fulfillment rates under dynamic user distributions. Further analysis highlights the impact of sigmoid parameter tuning on risk-QoS trade-offs, validating the adaptability of the proposed approach in complex scenarios.

Mobile edge computing (MEC) has emerged as a key enabler for low-latency and computation-intensive applications. However, conventional terrestrial MEC networks face challenges in coverage, scalability, and dynamic environments. To address these limitations, this paper explores unmanned aerial vehicle (UAV)-assisted MEC, leveraging the mobility and flexibility of UAVs to enhance edge computing performance. However, challenges such as dynamic mobile user association, multi-UAV collaborative trajectory design, and limited communication and computation resources remain significant hurdles. We then formulate a joint optimization problem for user association, trajectory design, and resource allocation for the multi-UAV collaborative MEC network, aiming to minimize system cost considering computation task demands and UAV resource constraints. The problem, modeled as a Mixed Integer NonLinear Program (MINLP), is reformulated as a decentralized partially observable Markov decision process (Dec-POMDP) to capture complex system dynamics. Afterwards, we propose a multi-agent generative-enhanced deep reinforcement learning method that integrates generative diffusion model (GDM) with multi-agent deep reinforcement learning (MADRL) to effectively handle high-dimensional action spaces and enable collaborative multi-UAV decision-making. Extensive simulations using a real-world mobility dataset demonstrate that the proposed method consistently outperforms three baselines in terms of time-average system cost, task completion delay, task satisfactory ratio, and energy consumption ratio when varying the number of UAVs.

Collaborative computation offloading and wireless charging scheduling in multi-UAV-assisted MEC networks: A TD3-based approach

Two-timescale hierarchical DRL for resource allocation in UAV-assisted 6G edge networks