Joint Task Partitioning and Resource Allocation in RAV-Enabled Vehicular Edge Computing Based on Deep Reinforcement Learning

5G beyond is an end-edge-cloud orchestrated network that can exploit heterogeneous capabilities of the end devices, edge servers, and the cloud and thus has the potential to enable computation-intensive and delay-sensitive applications via computation offloading. However, in multi user wireless networks, diverse application requirements and the possibility of various radio access modes for communication among devices make it challenging to design an optimal computation offloading scheme. In addition, having access to complete network information that includes variables such as wireless channel state, available bandwidth and computation resources, is a major issue. Deep Reinforcement Learning (DRL) is an emerging technique to address such an issue with limited and less accurate network information. In this paper, we utilize DRL to design an optimal computation offloading and resource allocation strategy for minimizing system energy consumption. We first present a multi-user end-edge-cloud orchestrated network where all devices and base stations have computation capabilities. Then, we formulate the joint computation offloading and resource allocation problem as a Markov Decision Process (MDP) and propose a new DRL algorithm to minimize system energy consumption. Numerical results based on a real-world dataset demonstrate that the proposed DRL-based algorithm significantly outperforms the benchmark policies in terms of system energy consumption. Extensive simulations show that learning rate, discount factor, and number of devices have considerable influence on the performance of the proposed algorithm.

A Novel Lyapunov based Dynamic Resource Allocation for UAVs-assisted Edge Computing

The task offloading and resource allocation strategies aim to provide efficient edge computing services for ground mobile devices in mobile edge computing (MEC) environments. In this paper, we propose a UAV-assisted dynamic deployment and decision-making framework (dDDM) to address the issue of insufficient base station coverage in remote areas or emergency scenarios, which leverages the flexibility and computational capabilities of unmanned aerial vehicle (UAV) to supplement terrestrial cellular networks. We focus on optimizing the UAV trajectory to support and enhance the effectiveness of task offloading and resource allocation strategies under conditions of user mobility and stochastic task arrivals. Therefore, a joint optimization problem is formulated with the objective of minimizing the weighted sum of the total latency of ground mobile devices and the overall energy consumption of UAV. To solve this problem, the original optimization problem is decomposed into two subproblems: UAV trajectory planning, and decision-making for task offloading & computing resource allocation. For the UAV trajectory planning subproblem, we propose an improved particle swarm optimization(PSO) algorithm integrated with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i>-means clustering, i.e., KMPSO algorithm, enabling dynamic UAV deployment in response to user mobility. For the task offloading and resource allocation decision-making, an improved black-winged kite algorithm (IBKA) is developed to determine optimal offloading and allocation strategies. Simulation results demonstrate that the proposed approach outperforms existing benchmark algorithms in terms of both latency and energy consumption.

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

Collaborative computation offloading and wireless charging scheduling in multi-UAV-assisted MEC networks: A TD3-based approach

Computation offloading services provide required computing resources for vehicles with computation-intensive tasks. Past computation offloading research mainly focused on mobile edge computing (MEC) or cloud computing, separately. This paper presents a collaborative approach based on MEC and cloud computing that offloads services to automobiles in vehicular networks. A cloud-MEC collaborative computation offloading problem is formulated through jointly optimizing computation offloading decision and computation resource allocation. Since the problem is non-convex and NP-hard, we propose a collaborative computation offloading and resource allocation optimization (CCORAO) scheme, and design a distributed computation offloading and resource allocation algorithm for CCORAO scheme that achieves the optimal solution. The simulation results show that the proposed algorithm can effectively improve the system utility and computation time, especially for the scenario where the MEC servers fail to meet demands due to insufficient computation resources.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

In the current fifth-generation (5G) and Beyond 5G (B5G) era, the Unmanned Aerial Vehicles (UAVs) have been playing a vital role and attracting interest in different application areas in the military, and civil applications such as communications, disaster management, search and rescue, security, control, agriculture, Internet of things (IoT), etc. In these networks, ultra-heterogeneous IoT devices generate time-sensitive traffic. However, those devices have limited resources to compute tasks. Recently, Mobile Edge Computation Offloading (MECO) has been considered as an encouraging model to enable the computation tasks of IoT devices to be performed by MEC servers and support ultra-low latency IoT applications to ensure Quality of services (QoS). However, terrestrial network failure due to natural and human-made disasters has been increasing, and difficult to provide reliable computation offloading and resource allocation services to IoT networks. Nowadays, UAVs have been promising technology to quickly deploy and recover the system to provide efficient services to edge nodes. The offloading and resource allocation problems in current network technology are complex, and offloading task to edge server is vulnerable to security risks. Hence, we utilize a deep reinforcement learning method to handle a complex problem for computation offloading and resource allocations in a dynamic environment. And also, we explore a blockchain-based multi-UAV-assisted MEC architecture in securing and optimizing the offloading problems.

In this paper, we propose a double-edge-assisted computation offloading and resource allocation scheme tailored for space-air-marine integrated networks (SAMINs). Specifically, we consider a scenario where both unmanned aerial vehicles (UAVs) and a low earth orbit (LEO) satellite are equipped with edge servers, providing computing services for maritime autonomous surface ships (MASSs). Partial computation workloads of MASSs can be offloaded to both UAVs and the LEO satellite, concurrently, for processing via a multi-access approach. To minimize the energy consumption of SAMINs under latency constraints, we formulate an optimization problem and propose energy efficient algorithms to jointly optimize offloading mode, offloading volume, and computing resource allocation of the LEO satellite and the UAVs, respectively. We further exploit an alternating optimization (AO) method and a layered approach to decompose the original problem to attain the optimal solutions. Finally, we conduct simulations to validate the effectiveness and efficiency of the proposed scheme in comparison with benchmark algorithms.

The increasing computational demand for real-time mobile applications has led to the development of mobile edge computing (MEC), with support from unmanned aerial vehicles (UAVs), as a promising paradigm for constructing high-throughput line-of-sight links for ground users and pushing computational resources to network edges. Users can reduce processing latency and the load on their local computers by delegating tasks to the UAV in its role as an edge server. The coverage capacity of a single UAV is, however, very limited. Moreover, it will be easy to intercept the data that is transferred to the unmanned aerial vehicle. Thus, for UAV-assisted mobile edge computing, we proposed a transmission technique based on multi-agent deep reinforcement learning in this study. The recommended approach to maximize UAV deployment first applies the particle swarm optimization algorithm. Then, deep reinforcement learning is utilized to optimize the secure offloading to maximize the system utility and minimize the quantity of information eavesdropping, taking into consideration different user task types with diverse preferences for processing time and residual energy of computing equipment. The results of the simulation demonstrate that, in comparison to the single-agent strategy and the benchmark, the multi-agent approach can optimize offloading more successfully and produce higher system utility.

Unmanned Aerial Vehicles (UAVs) are used in various applications, including crowd management, crime prevention, accident detection, and rescue operations. However, since UAVs perform their tasks independently, some UAV applications are dynamic and geographically distributed, which may require extensive real-time processing capabilities. Thus, processing UAV data locally can be challenging due to their limited computing capabilities. To overcome such limitations, fog and cloud computing can facilitate UAV application development by providing additional resource capacities when needed. Despite this, designing sophisticated and efficient UAV task offloading strategies that collaborate with fog and cloud technologies considering their service latency and energy consumption, is rarely addressed in the literature. Therefore, a collaborative offloading strategy for UAV applications is presented in this work, leveraging fog and cloud computing advantages and capabilities. This approach aims to minimize UAVs’ service latency and energy consumption, as well as provide the required resources and services in real time. In addition, task offloading decisions are formulated using the Mixed-Integer Linear Programming (MILP) model to reduce the energy consumption of the entire UAV-fog-cloud system by optimizing the allocation of computation resources and communication requested by each UAV. The simulation results demonstrate that the proposed strategy can significantly reduce UAV service latency by 15.38%, 35.29%, and 59.26%, as well as decrease overall energy consumption (including processing and networking) by 3.3%, 7.37%, and 12% when compared to alternative standalone strategies (namely UAV, fog, and cloud).

Mobile edge computing (MEC) has emerged as a key enabler for low-latency and computation-intensive applications. However, conventional terrestrial MEC networks face challenges in coverage, scalability, and dynamic environments. To address these limitations, this paper explores unmanned aerial vehicle (UAV)-assisted MEC, leveraging the mobility and flexibility of UAVs to enhance edge computing performance. However, challenges such as dynamic mobile user association, multi-UAV collaborative trajectory design, and limited communication and computation resources remain significant hurdles. We then formulate a joint optimization problem for user association, trajectory design, and resource allocation for the multi-UAV collaborative MEC network, aiming to minimize system cost considering computation task demands and UAV resource constraints. The problem, modeled as a Mixed Integer NonLinear Program (MINLP), is reformulated as a decentralized partially observable Markov decision process (Dec-POMDP) to capture complex system dynamics. Afterwards, we propose a multi-agent generative-enhanced deep reinforcement learning method that integrates generative diffusion model (GDM) with multi-agent deep reinforcement learning (MADRL) to effectively handle high-dimensional action spaces and enable collaborative multi-UAV decision-making. Extensive simulations using a real-world mobility dataset demonstrate that the proposed method consistently outperforms three baselines in terms of time-average system cost, task completion delay, task satisfactory ratio, and energy consumption ratio when varying the number of UAVs.

Towards enhanced threat modelling and analysis using a Markov Decision Process

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

With the rapid development of mobile edge computing (MEC) and unmanned aerial vehicle (UAV) communication networks, UAV-assisted edge computing has emerged as a promising paradigm for low-latency and energy-efficient computation. However, the time-varying nature of air-to-ground channels and the coupling between UAV trajectories and computation offloading decisions significantly increase system complexity. To address these challenges, this paper proposes an Adaptive UAV Edge Intelligence Framework (AUEIF) for joint UAV computation offloading and trajectory optimization under dynamic channels. Specifically, a dynamic graph-based system model is constructed to characterize the spatio-temporal correlation between UAV motion and channel variations. A hierarchical reinforcement learning-based optimization framework is developed, in which a high-level actor–critic module is responsible for generating coarse-grained UAV flight trajectories, while a low-level deep Q-network performs fine-grained optimization of task offloading ratios and computational resource allocation in real time. In addition, an adaptive channel prediction module leveraging long short-term memory (LSTM) networks is integrated to model temporal channel state transitions and to assist policy learning and updates. Extensive simulation results demonstrate that the proposed AUEIF achieves significant improvements in end-to-end latency, energy efficiency, and overall system stability compared with conventional deep reinforcement learning approaches and heuristic-based schemes while exhibiting strong robustness against dynamic and fluctuating wireless channel conditions.