Service migration with edge collaboration: Multi-agent deep reinforcement learning approach combined with user preference adaptation

In recent years, research has been conducted on dynamic map systems, which are information and communication platforms that manage vehicle sensor information and run applications. However, there is a concern about scalability in the dynamic map system, which operates on a cloud on the Internet, when the number of vehicles that transmit and receive sensor information increases. Therefore, it is considered that the application load can be distributed by distributing multiple edge servers geographically and having the information managed in the cloud by the edge servers. However, the edge server that controls the vehicle information and the edge server that receives the data do not always match depending on the actual radio wave conditions. Therefore, some applications are difficult to manage, such as intersection collision danger warning and merging arbitration when edge servers are assigned in the same way as base stations that receive data from vehicles. Such an application should receive all vehicle data for the target road area. Therefore, we divide the area on the road where a vehicle travels as “lane section ID” and assign an edge server to each lane section ID on the basis of that area. In addition, we implemented a dynamic map system that connects vehicles and edge servers by linking multiple edge servers. This enables the edge server to aggregate vehicle data without being affected by radio traffic conditions. We evaluated the scalability of the dynamic map system and verified the effectiveness of the load balancing mechanism using multiple edge servers.

The growing demand for low delay vehicular content has put tremendous strain on the backbone network. As a promising alternative, cooperative content caching among different cache nodes can reduce content access delay. However, heterogeneous cache nodes have different communication modes and limited caching capacities. In addition, the high mobility of vehicles renders the more complicated caching environment. Therefore, performing efficient cooperative caching becomes a key issue. In this paper, we propose a cross-tier cooperative caching architecture for all contents, which allows the distributed cache nodes to cooperate. Then, we devise the communication link and content caching model to facilitate timely content delivery. Aiming at minimizing transmission delay and cache cost, an optimization problem is formulated. Furthermore, we use a multi-agent deep reinforcement learning (MADRL) approach to model the decision-making process for caching among heterogeneous cache nodes, where each agent interacts with the environment collectively, receives observations yet a common reward, and learns its own optimal policy. Extensive simulations validate that the MADRL approach can enhance hit ratio while reducing transmission delay and cache cost.

Federated edge cloud (FEC) is an edge cloud environment where multiple edge servers in a single administrative domain collaborate together to provide real-time services. This environment reduces the possibility of violating the quality of service (QoS) requirements of target services by locating delay-sensitive services at nearby edge servers instead of deploying them on the cloud. However, as the number of edge servers increases, the amount of energy consumed by servers and network switches also increases. This creates another challenge for how to schedule delay-sensitive services over FEC, while minimizing the total energy consumption and reducing the QoS violation of a service at the same time. This paper proposes an energy-efficient service scheduling algorithm in FEC. The proposed algorithm is based on an observation that as the number of edge servers along the service path is reduced, the total energy consumption can be minimized. Traditional approaches place services using their maximum traffic requirements to ensure QoS without considering the actual traffic change. In contrast, the proposed algorithm schedules them with actual traffic requirements to increase the number of services co-located in a single server. This maximizes the consolidation of services in a single server and thus minimizes the energy consumption. Moreover, when edge servers are overloaded, the proposed algorithm reconfigures the service path such that service migration overhead and energy consumption are minimized while guaranteeing the QoS requirements of services. The simulation results show that the proposed algorithm improves energy efficiency by up to 21% and lowers the service violation rate by up to 80% against existing approaches.

Convolutional neural network (CNN)-based object detection is a key technology to enable autonomous mobile vision applications on mobile end devices such as smart phones and drones. With the advance of edge computing technology, a prevalent solution is to offload the computation-intensive CNN inference tasks to edge networks for fast and accurate object detection. However, a single edge server may not be powerful enough to ensure fast and accurate object detection due to its resource constraints. In this paper, we propose a multi-edge assisted fast object detection framework, MASS, to further reduce the object detection latency while maintaining the detection accuracy. In MASS, the CNN model is divided into two parts, namely the Head part and the Tail part. The Head part is executed locally and the Tail part is further split into multiple subtasks, which can be offloaded to multiple edge servers for parallel execution. First, we propose a method to select the optimal parallel entry point to separate the Head part from the Tail part. Then, an adaptive subtask generation and offloading strategy is proposed to divide the Tail part into multiple subtasks and offload these subtasks to multiple heterogeneous edge servers. Besides, we also propose a uniformly sampled zero-padding scheme to reduce the communication cost among edge servers when executing these subtasks in parallel. We implement MASS in a testbed with four edge servers and evaluate its performance, where the experimental results show that MASS can reduce object detection latency by up to $64.83%$, while the detection accuracy degradation is less than $3%$.

Edge computing is a promising computing paradigm that satisfies QoS requirements of delay-sensitive applications. In edge computing, server migration control is indispensable for managing client mobility. As a server migration method for edge computing, the method based on Q-learning has been proposed. However, the method assumes that there is only one application client and the number of destination edge servers is limited to one. In this paper, we propose a server migration method using Q-learning that copes with realistic situations where there are multiple application clients and destination edge servers. The contributions of this paper are as follows: 1) we clarify that, under the situation with multiple application clients and multiple destination edge servers, a straightforward server migration method using Q-learning (RL method) does not scale due to state space explosion, and 2) we propose a server migration method using Q-learning (RL-DR method) that reduces the dimensionality of state space by abstracting the numbers of application clients at all locations into a center of the gravity (COG) of application clients. The simulation results show that 1) RL method shows up to 248% worse performance than conventional server migration methods because of state space explosion and 2) RL-DR method achieves up to 38.3% better performance than the conventional methods.

Online Learning to parallel offloading in heterogeneous wireless networks

As the growth of vehicles and the acceleration of urbanization, the urban traffic congestion problem becomes a burning issue in our society. Constructing a software defined Internet of things(SD-IoT) with a proper traffic control scheme is a promising solution for this issue. However, existing traffic control schemes do not make the best of the advances of the multi-agent deep reinforcement learning area. Furthermore, existing traffic congestion solutions based on deep reinforcement learning(DRL) only focus on controlling the signal of traffic lights, while ignore controlling vehicles to cooperate traffic lights. So the effect of urban traffic control is not comprehensive enough. In this article, we propose Modified Proximal Policy Optimization (Modified PPO) algorithm. This algorithm is ideally suited as the traffic control scheme of SD-IoT. We adaptively adjust the clip hyperparameter to limit the bound of the distance between the next policy and the current policy. What's more, based on the collected data of SD-IoT, the proposed algorithm controls traffic lights and vehicles in a global view to advance the performance of urban traffic control. Experimental results under different vehicle numbers show that the proposed method is more competitive and stable than the original algorithm. Our proposed method improves the performance of SD-IoT to relieve traffic congestion.

Mobile edge computing (MEC) is one of the promising solutions to process computational-intensive tasks for the emerging time-critical Internet-of-Things (IoT) use cases, e.g., virtual reality (VR), augmented reality (AR), autonomous vehicle. The latency can be reduced further, when a task is partitioned and computed by multiple edge servers' (ESs) collaboration. However, the state-of-the-art work studies the MEC-enabled offloading based on a static framework, which partitions tasks at either the local user equipment (UE) or the primary ES. The dynamic selection between the two offloading schemes has not been well studied yet. In this paper, we investigate a dynamic offloading framework in a multi-user scenario. Each UE can decide who partitions a task according to the network status, e.g., channel quality and allocated computation resource. Based on the framework, we model the latency to complete a task, and formulate an optimization problem to minimize the average latency among UEs. The problem is solved by jointly optimizing task partitioning and the allocation of the communication and computation resources. The numerical results show that, compared with the static offloading schemes, the proposed algorithm achieves the lower latency in all tested scenarios. Moreover, both mathematical derivation and simulation illustrate that the wireless channel quality difference between a UE and different ESs can be used as an important criterion to determine the right scheme.

Service migration is crucial in mobile edge computing (MEC) to ensure seamless service provision as users move. Although some migration schemes have been proposed, they fail to efficiently support the migration of microservices in a directed acyclic graph (DAG)-based service across different edge servers, resulting in high service latency. This paper focuses on the DAG-based service migration problem and proposes a collaborative microservice migration framework for MEC, aiming to minimize the service migration latency while efficiently distributing the migration workload across edge servers. We divide edge servers into clusters and formulate the DAG-based service migration problem as a two-stage optimization problem. In the first stage, a deep reinforcement learning-based service pre-migration algorithm is developed to identify the optimal cluster of edge servers for hosting the migrated service. In the second stage, a microservice migration algorithm is devised, utilizing topological sorting and network flow techniques to further determine the target edge server for each microservice. Our design addresses the inherent dependencies among microservices within a DAG task and adapts well to dynamic network environments. Experimental results on real-world datasets demonstrate that our approach significantly reduces service migration latency.

Virtual machine (VM) is an important mechanism to multiplex computer resources. The increasing popularity of network computing has renewed research interests in the adaptive and distributed virtual machines. Service migration is a vital technique to construct reconfigurable VMs. By incorporating mobile agent technology, VM systems can improve their resource utilization, load-balancing and fault-tolerance significantly. This work focuses on the decision problem of hybrid mobility for load-balancing in reconfigurable distributed VMs. We tackle this problem from three aspects: migration candidate determination, migration timing and destination server selection. The service migration timing and destination server selection are formulated as two optimization models. We derive the optimal migration policy for distributed and heterogeneous systems based on stochastic optimization theories. Renewal processes are applied to model the dynamics of migration. We solve the agent migration problem by dynamic programming and extend the optimal service migration decision by considering the interplay of the hybrid mobility. Our decision policy is complementary to the existing service and agent migration techniques. Its accuracy is verified by simulations.

In mixed-mode operation, arrivals and departures are allowed to land and depart on the same runway. An appropriate strategy from air traffic controllers for arrivals and departures sequencing would boost the runway through-put significantly. On the other hand, safety is still the most crucial feature in the operation. Therefore, to assist air traffic controllers to make decisions on departures and arrivals with efficient utilization of runway capacity and safe operations, this paper proposed a Multi-agent Deep Reinforcement Learning approach using Multi-agent Deep Deterministic Policy Gradient to train two agents simultaneously: departure agent, and arrival agent. The departure agent makes departure slotting decisions for departures while the arrival agent determines the time delay or spacing decision on the arrival stream. A data-driven simulation environment is developed using Singapore Changi Airport data to support the learning process. Besides, a random sampling technique is also introduced to reduce redundant samples and increase off-policy sample efficiency. Moreover, the impact of different reward functions on runway throughput is also investigated and two specific models, e.g., ‘arrival priority’ and ‘departure priority’, are selected for further analysis in this study. As the result, by comparing the trained models with the ad-hoc model, the proposed approach increases the runway throughput significantly, with the highest 12.8% additional departure releases and overall 5.3% additional departure releases in identical environments while safety separations are maintained.

Strategic bidding of wind farms in medium-to-long-term rolling transactions: A bi-level multi-agent deep reinforcement learning approach

Bio-inspired distributed load frequency control in Islanded Microgrids: A multi-agent deep reinforcement learning approach

Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management

A multi-agent deep reinforcement learning approach for the collaborative optimization of electric distribution networks and independent prosumers after extreme weather events