Reinforcement Learning-Based Optimization of Quality-of-Service and Path Planning for Multi-UAV Mobile Edge Computing

Unmanned Aerial vehicles (UAVs) are widely used as network processors in mobile networks, but more recently, UAVs have been used in Mobile Edge Computing as mobile servers. However, there are significant challenges to use UAVs in complex environments with obstacles and cooperation between UAVs. We introduce a new multi-UAV Mobile Edge Computing platform, which aims to provide better Quality-of-Service and path planning based on reinforcement learning to address these issues. The contributions of our work include: 1) optimizing the quality of service for mobile edge computing and path planning in the same reinforcement learning framework; 2) using a sigmoid-like function to depict the terminal users’ demand to ensure a higher quality of service; 3) applying synthetic considerations of the terminal users’ demand, risk and geometric distance in reinforcement learning reward matrix to ensure the quality of service, risk avoidance, and the cost-savings. Simulations have shown the effectiveness and feasibility of our platform, which can help advance related researches. The source code can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bczhangbczhang</uri> .

This study explores the application of Particle Swarm Optimization (PSO) in Unmanned Aerial Vehicle (UAV) path planning within a simulated three-dimensional environment. UAVs, increasingly prevalent across various sectors, demand efficient navigation solutions that account for dynamic and unpredictable elements. Traditional pathfinding algorithms often fall short in complex scenarios, hence the shift towards PSO, a bio-inspired algorithm recognized for its adaptability and robustness. We developed a Python-based framework to simulate the UAV path planning scenario. The PSO algorithm was tasked to navigate a UAV from a starting point to a predetermined destination while avoiding spherical obstacles. The environment was set within a 3D grid with a series of waypoints, marking the UAV's trajectory, generated by the PSO to ensure obstacle avoidance and path optimization. The PSO parameters were meticulously tuned to balance the exploration and exploitation of the search space, with an emphasis on computational efficiency. A cost function penalizing proximity to obstacles guided the PSO in real-time decision-making, resulting in a collision-free and optimized path. The UAV's trajectory was visualized in both 2D and 3D perspectives, with the analysis focusing on the path's smoothness, length, and adherence to spatial constraints. The results affirm the PSO's effectiveness in UAV path planning, successfully avoiding obstacles and minimizing path length. The findings highlight PSO's potential for practical UAV applications, emphasizing the importance of parameter optimization. This research contributes to the advancement of autonomous UAV navigation, indicating PSO as a viable solution for real-world path planning challenges.

Unmanned Aerial Vehicles (UAVs), particularly quadrotor UAVs, are widely recognized for their cost-effectiveness, operational flexibility, and vertical takeoff and landing capabilities, making them ideal for specific airspace operations and emergency response scenarios. Efficient path planning is critical for UAV mission execution, requiring optimal flight paths that ensure safe obstacle avoidance while addressing challenges such as dynamic environments, energy optimization, and multi-parameter management. Despite advancements in path planning techniques, including the artificial potential field (APF) method and reinforcement learning (RL), issues like local optima and parameter tuning complexity persist, limiting adaptability in dynamic environments. This study proposes a hybrid approach integrating Markov Decision Process (MDP) theory with an improved artificial potential field (IAPF) method to enhance quadrotor UAV path planning in three-dimensional environments. By generating global waypoints based on known obstacle, the method minimizes flight path deviations and improves navigation performance. The results demonstrate significant advancements in trajectory accuracy and adaptability, offering a robust solution for UAV path optimization in complex scenarios.

Joint energy and throughput optimization for MEC-enabled multi-UAV IoRT networks

In this letter, we study an unmanned aerial vehicle (UAV)-mounted mobile edge computing network, where the UAV executes computational tasks offloaded from mobile terminal users (TUs) and the motion of each TU follows a Gauss-Markov random model. To ensure the quality-of-service (QoS) of each TU, the UAV with limited energy dynamically plans its trajectory according to the locations of mobile TUs. Towards this end, we formulate the problem as a Markov decision process, wherein the UAV trajectory and UAV-TU association are modeled as the parameters to be optimized. To maximize the system reward and meet the QoS constraint, we develop a QoS-based action selection policy in the proposed algorithm based on double deep Q-network. Simulations show that the proposed algorithm converges more quickly and achieves a higher sum throughput than conventional algorithms.

This paper studies the trajectory control and task offloading (TCTO) problem in an unmanned aerial vehicle (UAV)-assisted mobile edge computing system, where a UAV flies along a planned trajectory to collect computation tasks from smart devices (SDs). We consider a scenario that SDs are not directly connected by the base station (BS) and the UAV has two roles to play: MEC server or wireless relay. The UAV makes task offloading decisions online, in which the collected tasks can be executed locally on the UAV or offloaded to the BS for remote processing. The TCTO problem involves multi-objective optimization as its objectives are to minimize the task delay and the UAV's energy consumption, and maximize the number of tasks collected by the UAV, simultaneously. This problem is challenging because the three objectives conflict with each other. The existing reinforcement learning (RL) algorithms, either single-objective RLs or single-policy multi-objective RLs, cannot well address the problem since they cannot output multiple policies for various preferences (i.e. weights) across objectives in a single run. An evolutionary multi-objective RL (EMORL) algorithm is applied to address the TCTO problem. We improve the multi-task multi-objective proximal policy optimization of the original EMORL by retaining all new learning tasks in the offspring population, which can preserve promissing learning tasks. The simulation results demonstrate that the proposed algorithm can obtain more excellent non-dominated policies by striking a balance between the three objectives regarding policy quality, compared with two evolutionary algorithms, two multi-policy RL algorithms, and the original EMORL.

With the development of Unmanned Aerial Vehicles (UAVs) in recent years, UAV-mounted Mobile Edge Computing (MEC) systems are widely used for broadband connectivity and emergency communications in areas without internet. However, the onboard energy of UAVs is limited, and how to provide long-term computing services for ground User Equipments (UEs) still faces significant challenges. In this paper, UAVs are used as mobile base stations to offer MEC services for UEs. Unlike existing solutions that mainly solve UAV path planning from convex optimization methods, we propose a Multi-Agent Path planning (MAP) scheme based on deep reinforcement learning. We aim to minimize the total energy consumption of UAVs while completing the offloading tasks of UEs. At the same time, fairness is taken into account to ensure UE offloading balance and UAV load balance. To this end, we establish the optimization problem and design the state space, action space, and reward function for modeling each UAV through Deep Neural Networks (DNNs). Extensive simulation experiments show that the proposed scheme outperforms other benchmark schemes.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.&#13;\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. &#13;\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. &#13;\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. &#13;\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

Recent developments in unmanned aerial vehicles (UAVs) have led to the introduction of a wide variety of innovative applications, especially in the Mobile Edge Computing (MEC) field. UAV swarms are suggested as a promising solution to cope with the issues that may arise when connecting Internet of Things (IoT) applications to a fog platform. We are interested in a crucial aspect of designing a swarm of UAVs in this work, which is the coordination of swarm agents in complicated and unknown environments. Centralized leader–follower formations are one of the most prevalent architectural designs in the literature. In the event of a failed leader, however, the entire mission is canceled. This paper proposes a framework to enable the use of UAVs under different MEC architectures, overcomes the drawbacks of centralized architectures, and improves their overall performance. The most significant contribution of this research is the combination of distributed formation control, online leader election, and collaborative obstacle avoidance. For the initial phase, the optimal path between departure and arrival points is generated, avoiding obstacles and agent collisions. Next, a quaternion-based sliding mode controller is designed for formation control and trajectory tracking. Moreover, in the event of a failed leader, the leader election phase allows agents to select the most qualified leader for the formation. Multiple possible scenarios simulating real-time applications are used to evaluate the framework. The obtained results demonstrate the capability of UAVs to adapt to different MEC architectures under different constraints. Lastly, a comparison is made with existing structures to demonstrate the effectiveness, safety, and durability of the designed framework.

This chapter studies mobile edge computing (MEC) networks assisted by unmanned aerial vehicles (UAVs). According to the application scenarios, we consider three roles for UAVs in MEC networks: exploiting MEC computing capabilities, serving as a computing server, and serving as a relay for computation offloading. Furthermore, the details for resource allocation and optimization are presented in the three scenarios of UAV-assisted MEC networks. In addition, we focus on the situation in which a UAV not only functions as an MEC server to inspect turbines on a wind farm, but also performs task computation. To facilitate wide applications of UAV-assisted MEC in practice, this chapter highlights the main implementation issues of UAV-assisted MEC, including optimal UAV deployment, wind models, and joint trajectory–computation performance optimization.

Unmanned Aerial Vehicle (UAV)-enabled mobile edge computing has been proposed as an efficient task-offloading solution for user equipments (UEs). Nevertheless, the presence of heterogeneous UAVs makes centralized navigation policies impractical. Decentralized navigation policies also face significant challenges in knowledge sharing among heterogeneous UAVs. To address this, we present the soft hierarchical deep reinforcement learning network (SHDRLN) and dual-end federated reinforcement learning (DFRL) as a decentralized navigation policy solution. It enhances overall task-offloading energy efficiency for UAVs while facilitating knowledge sharing. Specifically, SHDRLN, a hierarchical DRL network based on maximum entropy learning, reduces policy differences among UAVs by abstracting atomic actions into generic skills. Simultaneously, it maximizes the average efficiency of all UAVs, optimizing coverage for UEs and minimizing task-offloading waiting time. DFRL, a federated learning (FL) algorithm, aggregates policy knowledge at the cloud server and filters it at the UAV end, enabling adaptive learning of navigation policy knowledge suitable for the UAV's performance parameters. Extensive simulations demonstrate that the proposed solution not only outperforms other baseline algorithms in overall energy efficiency but also achieves more stable navigation policy learning under different levels of heterogeneity of different UAV performance parameters.

Collaborative computation offloading and wireless charging scheduling in multi-UAV-assisted MEC networks: A TD3-based approach

Unmanned aerial vehicles (UAVs) are rapidly developing in major fields but also exposed it to security vulnerabilities. UAVs operate in high airborne network where it encounters many obstacles and uncertainty. And the existing path planning algorithms don’t consider path deviation attacks where the data communication is not secured. But, in multiple unmanned aerial vehicles, path planning is a distinctive optimization problem. And for swarm of unmanned aerial vehicles (UAVs) deployed in an uncertain dynamic environment, optimizing the UAV coverage area is a major research area that it needs to be addressed to support effective and secure data transmission. For optimized path planning in UAV swarm, we proposed a modified particle swarm optimization (MPSO) algorithm which encodes the multidimensional points for instant optimal path. The factors that are considered for the encoding helps to find the global optimum. And the blockchain distributed ledger in UAV nodes will protect the UAV network from security attacks such as denial-of-service and man-in-the-middle attacks. It also supports unrestricted coverage area, where the storage is made active only on demand which is dynamically cost-effective in UAV network. Simulation results shows that the proposed algorithm provides instant optimal path with unrestricted coverage aided with data security.KeywordsSwarm optimizationDenial-of-service attackMan-in-the-middle attackBlockchainData securityUnmanned aerial vehicles

A comprehensive review of computation offloading in UAV-assisted mobile edge computing for IoT applications