Optimal Task Offloading with Deep Q-Network for Edge-Cloud Computing Environment

Edge-cloud computing is increasingly prevalent for Internet-of-thing (IoT) service provisioning by combining both benefits of edge and cloud computing. In this paper, we aim to improve the user satisfaction and the resource efficiency by service caching and task offloading for edge-cloud computing. We propose a hybrid heuristic method to combine the global search ability of the genetic algorithm (GA) and heuristic local search ability, to improve the number of satisfied requests and the resource utilization. The proposed method encodes the service caching strategies into chromosomes, and evolves the population by GA. Given a caching strategy from a chromosome, our method exploits a dual-stage heuristic method for the task offloading. In the first stage, the dual-stage heuristic method pre-offloads tasks to the cloud, and offloads tasks whose requirements cannot be satisfied by the cloud to edge servers, aiming at satisfying as many tasks’ requirements as possible. The second stage re-offloads tasks from the cloud to edge servers, to get the utmost out of limited edge resources. Experimental results demonstrate the competitive edges of the proposed method over multiple classical and state-of-the-art techniques. Compared with five existing scheduling algorithms, our method achieves 11.3%–23.7% more accepted tasks and 1.86%–18.9% higher resource utilization.

In recent years, there is development in the field of computing devices; Internet of Things (IoT) becomes the latest trend. IoT comprises ubiquitous things that are associated with day-to-day life of individuals like smartphones, smart TV, laptops, and now vehicles too. Internet of Vehicles (IoV) has become the latest area of research used to develop applications in the field of traffic management and road safety. A collaborative approach of cloud and edge computing is termed cloud-edge computing. To manage the enormous amount of IoT devices and the coordination among IoT, the cloud and edge concept of computation offloading is required. In the process of computation offloading, tasks are computationally offloaded to the cloud data center that enhances the resource utilization of the cloud server and minimizes the energy utilization for the tasks. This paper represents the literature review related to various computational offloading schemes in cloud-edge computing proposed as part of the study. The resources comprise of related book chapters and research papers from different publishers of international and national reputes. The study is carried out with the analysis of various computation offloading schemes in cloud-edge computing for the Internet of Vehicles. In addition, computing technologies like cloud computing, edge computing, and computation offloading for the Internet of Vehicles (IoV) were also discussed.KeywordsInternet of Things (IoT)Edge computingCloud computingComputation offloadingInternet of Vehicles (IoV)

The rapid evolution of 6G networks has brought new challenges in the domain of task offloading (TO), particularly within edge computing environments that are heavily reliant on the internet of things (IoT). Traditional TO methods that based on rule-based heuristics or shallow learning techniques fail to adapt efficiently to the dynamic, unpredictable network conditions, resource heterogeneity, and varying task demands. The proliferation of edge computing, the IoT, and 6G networks has introduced new challenges in TO due to dynamic network conditions, resource heterogeneity, and unpredictable task demands. To address these challenges, this work proposes an innovative TO method that integrates deep reinforcement learning (DRL) with heuristic search methods. The combination of DRL and heuristic algorithms enhances adaptability, convergence speed, and decision-making efficiency, making it well-suited for real-time TO in complex and unpredictable environments This paper proposes a novel hybrid TO framework that integrates DRL with genetic algorithms (GA) to address these challenges. The proposed hybrid optimization technique offer promising solutions by leveraging the strengths of individual approaches to balance competing objectives, such as energy consumption, task completion time, and resource utilization. This method explores optimization strategies to enhance TO efficiency in decentralized environments mainly focusing on optimizing energy use while ensuring performance metrics like latency, throughput, and task deadlines are met.

Mobile edge computing (MEC) can effectively overcome the shortcomings of high-latency in mobile cloud computing (MCC) by deploying the cloud resources, e.g., storage and computational capability, to the edge. However, the limited computation capability of the MEC restricts the scalability of offloading. Therefore, the basic requirements of the MEC system are to explore effective offloading decisions and resource allocation methods. To address it, we develop a collaborative computing system composed of local computing (mobile device), MEC (edge cloud) and MCC (central cloud). Based on the proposed collaborative computing system, we design a novel Q-learning based computation offloading (QLCOF) policy to achieve the optimal resource allocation and offloading scheme by prescheduling the computation side for each task from a global perspective. Specifically, we first model the offloading decision process as a Markov decision process (MDP) and design a state loss function (STLF) to measure the quality of experience (QoE). After that, we define the cumulation of STLFs as the system loss function (SYLF) and formulate an SYLF minimization problem. Due to the difficulty to directly solve the formulated problem, we decompose it into multiple subproblems and preferentially optimize the transmission power and computation frequency of the edge cloud by the quasi-convex bisection and polynomial analysis method, respectively. Based on the precalculated offline transmission power and edge cloud computation frequency, we develop a Q-learning based offloading (QLOF) scheme to minimize the SYLF by optimizing offloading decisions. Finally, the numeral results show that the proposed QLOF scheme effectively reduces the SYLF under different parameters.

Unmanned Aerial Vehicles (UAVs) are used in various applications, including crowd management, crime prevention, accident detection, and rescue operations. However, since UAVs perform their tasks independently, some UAV applications are dynamic and geographically distributed, which may require extensive real-time processing capabilities. Thus, processing UAV data locally can be challenging due to their limited computing capabilities. To overcome such limitations, fog and cloud computing can facilitate UAV application development by providing additional resource capacities when needed. Despite this, designing sophisticated and efficient UAV task offloading strategies that collaborate with fog and cloud technologies considering their service latency and energy consumption, is rarely addressed in the literature. Therefore, a collaborative offloading strategy for UAV applications is presented in this work, leveraging fog and cloud computing advantages and capabilities. This approach aims to minimize UAVs’ service latency and energy consumption, as well as provide the required resources and services in real time. In addition, task offloading decisions are formulated using the Mixed-Integer Linear Programming (MILP) model to reduce the energy consumption of the entire UAV-fog-cloud system by optimizing the allocation of computation resources and communication requested by each UAV. The simulation results demonstrate that the proposed strategy can significantly reduce UAV service latency by 15.38%, 35.29%, and 59.26%, as well as decrease overall energy consumption (including processing and networking) by 3.3%, 7.37%, and 12% when compared to alternative standalone strategies (namely UAV, fog, and cloud).

Federated deep reinforcement learning for task offloading and resource allocation in mobile edge computing-assisted vehicular networks

Emerging digital twin (DT) and mobile-edge computing (MEC) are crucial for enabling the rapid development of 6G. However, the existing works ignore the edge collaboration, which can provide the system with additional performance gain. In this article, we study the problem of mobile users (MUs) intelligently offloading tasks to cooperative mobile-edge servers (MESs) with the assistance of DT. Specifically, a DT-assisted task offloading scheme (DTTOS) that consists of the selection of MESs and intelligent task offloading is proposed. Channel state information (CSI) and blockchain are employed to implement the selection of MESs. Then, we present a solution to enable MU&#x2019;s task offloading that is modeled as a Markov decision process (MDP) in an intelligent way. After this, a mathematical optimization model aiming at decreasing power and time overhead is formulated. In view of the complexity, it is decomposed into two suboptimization models and solved by the decision tree algorithm (DTA) and double deep-<inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning (DDQN), respectively. Simulations are conducted to prove the superiority of the proposed scheme in terms of data security assurance and network performance improvement.

Task offloading is crucial for the efficient management of vehicular wireless networks, requiring access to real-time information on network status and available computational resources. However, because vehicles are mobile network nodes with variable speeds and limited computational resources for near-real-time task offloading, task offloading schemes may incur high latency, rendering this goal practically infeasible. To address this issue, this paper presents a trajectory-prediction-based pre-offloading decision (TPPD) algorithm that analyzes historical vehicle trajectories to predict future coordinates, thereby enabling the allocation of computational resources in advance. We first use a Long Short-Term Memory (LSTM) network to predict vehicle trajectories with error correction. Then, based on the task requirements and predicted trajectories, a dynamic resource allocation algorithm using a Double Deep Q-Network (DDQN) is designed to enable the multi-access edge server to minimize task processing delay while ensuring effective utilization of available computational resources. Numerical results verify the effectiveness of the proposed approach, demonstrating that, compared to traditional real-time task offloading strategies, the proposed TPPD algorithm significantly reduces task processing delay while improving resource utilization.

While service-oriented low Earth orbit (LEO) satellite edge computing (LSEC) frameworks enable diverse edge services for user tasks, the heterogeneous distribution of terrestrial users causes substantial imbalance in computational task loads across satellites. This asymmetric workload leads to inefficient resource utilization and degraded edge computing performance. To address these challenges, we propose a joint optimization framework that integrates edge service deployment and task offloading, supported by service popularity analysis and spatiotemporal user-task modeling. The framework employs a two-timescale design: at the large timescale, an improved atomic orbital search (iAOS) heuristic dynamically optimizes service placement, configuration, and resource allocation; at the small timescale, a direction-selective multi-agent double deep Q-network (DS-MDDQN) leverages deep reinforcement learning to route tasks to the most suitable processing nodes. Extensive simulations show that our approach significantly outperforms six representative baselines in both user-perceived performance and system-level efficiency. Replacement studies further verify the effectiveness of each component: iAOS enhances resource utilization and reduces task failure through optimized service deployment, while DS-MDDQN mitigates network dynamics and lowers task completion latency via adaptive task offloading.

This paper presents a hybrid approach based discrete Particle Swarm Optimization (PSO) and chaotic strategies for solving multi-objective task scheduling problem in cloud computing. The main purpose is to allocate the summited tasks to the available resources in the cloud environment with minimum makespan (i.e. schedule length) and processing cost while maximizing resource utilization without violating Service Level Agreement (SLA) among users and cloud providers. The main challenges faced by Particle Swarm Optimization (PSO) when used to solve scheduling problems are premature convergence and trapping into local optimum. This paper presents an enhanced Particle Swarm Optimization algorithm hybridized with Chaotic Map strategies. The proposed approach is called Enhanced Particle Swarm Optimization based Chaotic Strategies (EPSOCHO) algorithm. Our proposed approach suggests two Chaotic Map strategies: sinusoidal iterator and Lorenz attractor to enhanced PSO algorithm in order to get good convergence and diversity for optimizing the task scheduling in cloud computing. The proposed approach is simulated and implemented in Cloudsim simulator. The performance of the proposed approach is compared with the standard PSO algorithm, the improved PSO algorithm with Longest job to fastest processor (LJFP-PSO), and the improved PSO algorithm with minimum completion time (MCT-PSO) using different sizes of tasks and various benchmark datasets. The results clearly demonstrate the efficiency of the proposed approach in terms of makespan, processing cost and resources utilization.

Preemptive scheduling provides an efficient mechanism for the coexistence of enhanced Mobile Broadband (eMBB) and Ultra-Reliable and Low-Latency Communication (URLLC) services in 5G networks. Due to differences in Quality-of-Service (QoS) requirements of the two services, it is necessary to investigate the trade-offs among energy-efficiency, throughput, latency, and reliability, in task offloading scenarios. Existing studies do not fully extend the preemptive scheduling mechanism for uplink task offloading while considering the interplay between all the QoS requirements of eMBB and URLLC User Equipment (UE). We address this gap by considering an edge network with eMBB and URLLC UE and formulate an objective function to improve energy-efficiency, throughput, and task queueing delay of eMBB UE while satisfying the transmission reliability of URLLC users. We also introduce the Twin Delayed Deep Deterministic Policy Gradient (TD3) to optimize task offloading. Simulation results show that the TD3 outperforms a widely used benchmark model in terms of convergence rate and variance. Results also highlight a very crucial aspect in reward shaping that embodies the trade-offs among the optimization variables and constraints.

EDITORS: Energy-aware Dynamic Task Offloading using Deep Reinforcement Transfer Learning in SDN-enabled Edge Nodes

With the ongoing advancement of electric power Internet of Things (IoT), traditional power inspection methods face challenges such as low efficiency and high risk. Unmanned aerial vehicles (UAVs) have emerged as a more efficient solution for inspecting power facilities due to their high maneuverability, excellent line-of-sight communication capabilities, and strong adaptability. However, UAVs typically grapple with limited computational power and energy resources, which constrain their effectiveness in handling computationally intensive and latency-sensitive inspection tasks. In response to this issue, we propose a UAV task offloading strategy based on deep reinforcement learning (DRL), which is designed for power inspection scenarios consisting of mobile edge computing (MEC) servers and multiple UAVs. Firstly, we propose an innovative UAV-Edge server collaborative computing architecture to fully exploit the mobility of UAVs and the high-performance computing capabilities of MEC servers. Secondly, we established a computational model concerning energy consumption and task processing latency in the UAV power inspection system, enhancing our understanding of the trade-offs involved in UAV offloading strategies. Finally, we formalize the task offloading problem as a multi-objective optimization issue and simultaneously model it as a Markov Decision Process (MDP). Subsequently, we proposed a task offloading algorithm based on a Deep Deterministic Policy Gradient (OTDDPG) to obtain the optimal task offloading strategy for UAVs. The simulation results demonstrated that this approach outperforms baseline methods with significant improvements in task processing latency and energy consumption.

Mobile IoT networks face major problems in completing efficient task offloading and allocating resources effectively, as all major actions require ample energy consumption and service delays. This paper proposes a solution in the form of a reinforcement learning-based framework that is capable of dynamically optimizing these activities with the help of Group Relative Policy Optimization (GRPO). The new method outcompetes traditional heuristic-nometric methods by being able to adjust device inactivity, network activity, and available resources. Devices are structured into mobility groups and task offloading is defined as an MDP or Markov Decision Process. An ample amount of offloading strategies can be obtained by introducing declaration of tasks and corresponding computers, network throughput, and queueing delays. Additionally, GRPO aids in the allocation of computing power, network bandwidth, or storage to maximize resource allocation without breaking task deadlines and other resource limitations. Comparing the GRPO-based method to traditional methods via simulation showed increased performance and reduced service delay. The results clearly demonstrate that GRPO helps reduce service delay and consumption of energy significantly in high load with high dynamism conditions. With the inclusion of decision-making processes along with constraint-tolerance and reinforcement learning, this research aids in IoT frameworks that require low energy output with less delay. The recommended method helps build a strong base for efficient task delegation alongside increased performance and scalability, which subsequently enhances AI edge computing and smart management of IoT systems.

Multi-agent deep reinforcement learning for task offloading in group distributed manufacturing systems