Decentralized Navigation With Heterogeneous Federated Reinforcement Learning for UAV-Enabled Mobile Edge Computing

Due to the advanced capabilities of the Internet of Vehicles (IoV) components such as vehicles, Roadside Units (RSUs) and smart devices as well as the increasing amount of data generated, Federated Learning (FL) becomes a promising tool given that it enables privacy-preserving machine learning that can be implemented in the IoV. However, the performance of the FL suffers from the failure of communication links and missing nodes, especially when continuous exchanges of model parameters are required. Therefore, we propose the use of Unmanned Aerial Vehicles (UAVs) as wireless relays to facilitate the communications between the IoV components and the FL server and thus improving the accuracy of the FL. However, a single UAV may not have sufficient resources to provide services for all iterations of the FL process. In this paper, we present a joint auction-coalition formation framework to solve the allocation of UAV coalitions to groups of IoV components. Specifically, the coalition formation game is formulated to maximize the sum of individual profits of the UAVs. The joint auction-coalition formation algorithm is proposed to achieve a stable partition of UAV coalitions in which an auction scheme is applied to solve the allocation of UAV coalitions. The auction scheme is designed to take into account the preferences of IoV components over heterogeneous UAVs. The simulation results show that the grand coalition, where all UAVs join a single coalition, is not always stable due to the profit-maximizing behavior of the UAVs. In addition, we show that as the cooperation cost of the UAVs increases, the UAVs prefer to support the IoV components independently and not to form any coalition.

Due to the high maneuverability and flexibility, unmanned aerial vehicles (UAVs) have been considered as a promising paradigm to assist mobile edge computing (MEC) in many scenarios including disaster rescue and field operation. Most existing research focuses on the study of trajectory and computation-offloading scheduling for UAV-assisted MEC in stationary environments, and could face challenges in dynamic environments where the locations of UAVs and mobile devices (MDs) vary significantly. Some latest research attempts to develop scheduling policies for dynamic environments by means of reinforcement learning (RL). However, as these need to explore in high-dimensional state and action space, they may fail to cover in large-scale networks where multiple UAVs serve numerous MDs. To address this challenge, we leverage the idea of “divide-and-conquer” and propose HT3O, a scalable scheduling approach for large-scale UAV-assisted MEC. First, HT3O is built with neural networks via deep RL to obtain real-time scheduling policies for MEC in dynamic environments. More importantly, to make HT3O more scalable, we decompose the scheduling problem into two-layered subproblems and optimize them alternately via hierarchical RL. This not only substantially reduces the complexity of each subproblem, but also improves the convergence efficiency. Experimental results show that HT3O can achieve promising performance improvements over state-of-the-art approaches.

With the development of Unmanned Aerial Vehicles (UAVs) in recent years, UAV-mounted Mobile Edge Computing (MEC) systems are widely used for broadband connectivity and emergency communications in areas without internet. However, the onboard energy of UAVs is limited, and how to provide long-term computing services for ground User Equipments (UEs) still faces significant challenges. In this paper, UAVs are used as mobile base stations to offer MEC services for UEs. Unlike existing solutions that mainly solve UAV path planning from convex optimization methods, we propose a Multi-Agent Path planning (MAP) scheme based on deep reinforcement learning. We aim to minimize the total energy consumption of UAVs while completing the offloading tasks of UEs. At the same time, fairness is taken into account to ensure UE offloading balance and UAV load balance. To this end, we establish the optimization problem and design the state space, action space, and reward function for modeling each UAV through Deep Neural Networks (DNNs). Extensive simulation experiments show that the proposed scheme outperforms other benchmark schemes.

Joint energy and throughput optimization for MEC-enabled multi-UAV IoRT networks

Unmanned Aerial vehicles (UAVs) are widely used as network processors in mobile networks, but more recently, UAVs have been used in Mobile Edge Computing as mobile servers. However, there are significant challenges to use UAVs in complex environments with obstacles and cooperation between UAVs. We introduce a new multi-UAV Mobile Edge Computing platform, which aims to provide better Quality-of-Service and path planning based on reinforcement learning to address these issues. The contributions of our work include: 1) optimizing the quality of service for mobile edge computing and path planning in the same reinforcement learning framework; 2) using a sigmoid-like function to depict the terminal users’ demand to ensure a higher quality of service; 3) applying synthetic considerations of the terminal users’ demand, risk and geometric distance in reinforcement learning reward matrix to ensure the quality of service, risk avoidance, and the cost-savings. Simulations have shown the effectiveness and feasibility of our platform, which can help advance related researches. The source code can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bczhangbczhang</uri> .

This paper studies the trajectory control and task offloading (TCTO) problem in an unmanned aerial vehicle (UAV)-assisted mobile edge computing system, where a UAV flies along a planned trajectory to collect computation tasks from smart devices (SDs). We consider a scenario that SDs are not directly connected by the base station (BS) and the UAV has two roles to play: MEC server or wireless relay. The UAV makes task offloading decisions online, in which the collected tasks can be executed locally on the UAV or offloaded to the BS for remote processing. The TCTO problem involves multi-objective optimization as its objectives are to minimize the task delay and the UAV's energy consumption, and maximize the number of tasks collected by the UAV, simultaneously. This problem is challenging because the three objectives conflict with each other. The existing reinforcement learning (RL) algorithms, either single-objective RLs or single-policy multi-objective RLs, cannot well address the problem since they cannot output multiple policies for various preferences (i.e. weights) across objectives in a single run. An evolutionary multi-objective RL (EMORL) algorithm is applied to address the TCTO problem. We improve the multi-task multi-objective proximal policy optimization of the original EMORL by retaining all new learning tasks in the offspring population, which can preserve promissing learning tasks. The simulation results demonstrate that the proposed algorithm can obtain more excellent non-dominated policies by striking a balance between the three objectives regarding policy quality, compared with two evolutionary algorithms, two multi-policy RL algorithms, and the original EMORL.

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

Nowadays, in an Unmanned Aerial Vehicle (UAV)- enabled Integrated Sensing and Communication (ISAC) System, a UAV can provide on-demand communication services to ground terminals (GTs) by cooperating with infrastructure networks. Furthermore, with radio frequency (RF) sensing capabilities enabled by the ISAC signal, UAVs can track the locations of GTs and thereby provide better communication services. However, due to the high mobility and SWAP (size, weight, and power) constraints of UAVs, as well as the complex working environment, it is difficult to achieve high-accuracy GT location sensing. As a result, UAVs cannot continuously support highdata-rate communication services for GTs. To address the above issues, we adopt Federated Learning (FL) between UAVs and a central server to improve the accuracy of GT location sensing, and jointly optimize UAV trajectory and resource allocation to ensure the Quality of Service(QoS) of communication services. FL is employed to perform distributed learning for the maximum likelihood estimation (MLE) of GT locations. We design a gametheoretic method to enhance UAV operations under the FL framework. This method ensures that each UAV contributes effectively to location sensing. On the other hand, the formulated joint trajectory and resource optimization problem is complex and subject to multiple non-convex constraints. We apply the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method to solve the problem and obtain a sub-optimal solution. Numerical results validate that, with our proposed solution, UAVs can deliver communication services with enhanced QoS in terms of higher data rates, while also operating more energy-efficiently in the considered system compared to benchmark solutions.

In recent years, low-latency tasks such as real-time communication, virtual reality, and augmented reality have higher latency requirements. Since users have relatively weak computing power, the past practice was to offload low-latency tasks to base stations (BSs) with more powerful computing power for execution, which can reduce computing latency in real-time communication. Duetothe inflexible deployment of BSs and the uneven distribution of computing resources, users often find it difficult to meet the needs of low-latency tasks in places that lack BSs, so unmanned aerial vehicles (UAVs) are needed to alleviate this problem. UAVs can serve as communication relays between users and BSs and provide computing services directly to users. However, when there are few BSs, UAVs may experience load imbalance, that is, some UAVs are overloaded while other UAVs are idle. To solve this problem, we propose a UAV collaborative three-layer mobile edge computing (MEC) system model and describe the problem as a mixed integer nonlinear programming (MINLP) problem. Then, we propose an extended genetic algorithm for UAV collaboration in three-layer MEC task allocation and optimization (EGAC-3MEC) where the three-layer structure includes users, UAVs, and BSs. The proposed EGAC- 3MEC achieves reasonable allocation of tasks and resources through offloading and resource allocation decisions, reduces system delay and energy consumption as much as possible, and prioritizes the delay of high-priority users. Experimental results show that the algorithm in this paper converges quickly when various parameters change, achieves better performance than the baseline algorithms when the user task size is different, and moreover, the necessity of UAV collaboration is proven.

This chapter studies mobile edge computing (MEC) networks assisted by unmanned aerial vehicles (UAVs). According to the application scenarios, we consider three roles for UAVs in MEC networks: exploiting MEC computing capabilities, serving as a computing server, and serving as a relay for computation offloading. Furthermore, the details for resource allocation and optimization are presented in the three scenarios of UAV-assisted MEC networks. In addition, we focus on the situation in which a UAV not only functions as an MEC server to inspect turbines on a wind farm, but also performs task computation. To facilitate wide applications of UAV-assisted MEC in practice, this chapter highlights the main implementation issues of UAV-assisted MEC, including optimal UAV deployment, wind models, and joint trajectory–computation performance optimization.

Federated learning (FL) is considered a promising machine learning technique that has attracted increasing attention in recent years. Instead of centralizing data in one location for training a global model, FL allows the model training to occur on user devices, such as smartphones, IoT devices, or local servers, thereby respecting data privacy and security. However, implementing FL in wireless communication faces a significant challenge due to the inherent unpredictability and constant fluctuations in channel characteristics. A key challenge in implementing FL over wireless communication lies in optimizing energy efficiency. This holds significant importance, especially considering user devices with restricted power resources. On the other hand, unmanned aerial vehicle (UAV) technologies present a cost-effective solution owing to flexibility and mobility compared to terrestrial base stations. Consequently, the deployment of UAV communication in FL is viewed as a potential approach to deal with the energy efficiency challenge. In this paper, we address the problem of minimizing the total energy consumption of all user equipment (UE) during the training phase of FL over a UAV communication network. Our proposed system facilitates UE to operate concurrently at the same time and frequency, thereby improving bandwidth utilization efficiently. In this paper, we address the problem of minimizing the total energy consumption during the training phase of FL over a UAV communication network. To deal with the proposed nonconvex problem, we propose a novel alternating optimization approach by dividing the problem into two suboptimal problems. We then develop iterative algorithms based on the inner approximation method, yielding at least one locally optimal solution. The numerical results demonstrate the superiority of the proposed algorithm in solving the proposed problem compared to other benchmark algorithms, particularly in determining the optimal trajectory of the UAVs. In addition, we conduct extensive experiments to evaluate how different parameter settings affect performance after implementing the proposed optimization approaches for deploying FL within the UAV communication system. These analyses yield valuable insights into the comparative effectiveness of the proposed optimization algorithms concerning overall energy consumption reduction.

As maritime activities become more frequent, many computationally intensive applications have emerged. Unmanned aerial vehicles (UAVs) can be utilized to provide computing services for maritime networks. In this paper, a multi-UAV assisted mobile edge computing (MEC) network is designed. Each UAV is configured with a nano-server to provide computational offloading services for maritime users (MVs). The goal is to minimize the maximum processing delay of the multi-VAV assisted MEC network. We present an optimization problem for joint task offloading, resource allocation, and flight trajectory of UAVs. The above problem is an mixed non-integer linear programming problem (MINLP), which is transformed into a Markov decision process (MDP). A computational offload algorithm based on deep deterministic policy gradient (DDPG) is proposed to solve this problem. Simulation results reveal that the DDPG algorithm can achieve fast convergence and minimize processing delay compared with baseline algorithms.

Collaborative computation offloading and wireless charging scheduling in multi-UAV-assisted MEC networks: A TD3-based approach

The application of unmanned aerial vehicles (UAV) as base station (BS) is gaining popularity. In this paper, we consider maximization of the overall data rate by intelligent deployment of UAV BS in the downlink of a cellular system. We investigate a reinforcement learning (RL)-aided approach to optimize the position of flying BSs mounted on board UAVs to support a macro BS (MBS). We propose an algorithm to avoid collision between multiple UAVs undergoing exploratory movements and to restrict UAV BSs movement within a predefined area. Q-learning technique is used to optimize UAV BS position, where the reward is equal to sum of user equipment (UE) data rates. We consider a framework where the UAV BSs carry out exploratory movements in the beginning and exploitary movements in later stages to maximize the overall data rate. Our results show that a cellular system with three UAV BSs and one MBS serving 72 UE reaches 69.2% of the best possible data rate, which is identified by brute force search. Finally, the RL algorithm is compared with a K-means algorithm to study the need of accurate UE locations. Our results show that the RL algorithm outperforms the K-means clustering algorithm when the measure of imperfection is higher. The proposed algorithm can be made use of by a practical MBS–UAV BSs–UEs system to provide protection to UAV BSs while maximizing data rate.

The demand for low-latency computing from the Internet of Things (IoT) and emerging applications challenges traditional cloud computing. Mobile Edge Computing (MEC) offers a solution by deploying resources at the network edge, yet terrestrial deployments face limitations. Unmanned Aerial Vehicles (UAVs), leveraging their high mobility and flexibility, provide dynamic computation offloading for User Equipments (UEs), especially in areas with poor infrastructure or network congestion. However, UAV-assisted MEC confronts significant challenges, including time-varying wireless channels and the inherent energy constraints of UAVs. We put forward the Lyapunov-based Deep Deterministic Policy Gradient (LyDDPG), a novel computation offloading algorithm. This algorithm innovatively integrates Lyapunov optimization with the Deep Deterministic Policy Gradient (DDPG) method. Lyapunov optimization transforms the long-term, stochastic energy minimization problem into a series of tractable, per-timeslot deterministic subproblems. Subsequently, DDPG is utilized to solve these subproblems by learning a model-free policy through environmental interaction. This policy maps system states to optimal continuous offloading and resource allocation decisions, aiming to minimize the Lyapunov-derived “drift-plus-penalty” term. The simulation outcomes indicate that, compared to several baseline and leading algorithms, the proposed LyDDPG algorithm reduces the total system energy consumption by at least 16% while simultaneously maintaining low task latency and ensuring system stability.