PerDNN: Offloading Deep Neural Network Computations to Pervasive Edge Servers

The momentous enabling of deep learning (DL)-powered mobile application is posing a soaring demand for computing resources that can hardly be satisfied by mobile devices. In this paper, we employ Edge Computing to deliver DL inference services to mobile users, where Deep Neural Networks (DNNs) are configured on edge servers, processing inference tasks received from mobile devices. A novel method called Adaptive DNN Ensemble (ADE) is proposed to enhance the performance of DL inference services. The core of ADE is the DNN ensemble technique which improves the stability and accuracy of DL inference. Due to the limited computing resources and service response deadline, ADE needs to judiciously determine DNNs to be included in the DNN ensemble, which poses a unique DNN ensemble selection problem. In addition, because DNNs exhibit performance variations for tasks with different features, DNN ensemble selection also aims to reconFigure DNN ensembles according to the feature of admitted tasks. We design an online learning algorithm, Contextual Combinatorial Multi-Armed Bandit (CC-MAB), to learn the DNN performance for tasks with different features. We rigorously prove that the proposed online learning algorithm is able to achieve asymptotic optimality. Experiments are carried out on an edge computing testbed to evaluate our method. Various implementation concerns, including memory usage, time complexity, and DNN switching cost, are considered. The results show that ADE outperforms other benchmarks in terms of inference accuracy and can provide real-time responses.

The emerging edge computing technique provides support for the computation tasks that are delay-sensitive and compute-intensive, such as deep neural network inference, by offloading them from a user-end device to an edge server for fast execution. The increasing offloaded tasks on an edge server are gradually facing the contention of both the network and computation resources. The existing offloading approaches often partition the deep neural network at a place where the amount of data transmission is small to save network resource, but rarely consider the problem caused by computation resource shortage on the edge server. In this paper, we design LoADPart, a deep neural network offloading system. LoADPart can dynamically and jointly analyze both the available network bandwidth and the computation load of the edge server, and make proper decisions of deep neural network partition with a light-weighted algorithm, to minimize the end-to-end inference latency. We implement LoADPart for MindSpore, a deep learning framework supporting edge AI, and compare it with state-of-the-art solutions in the experiments on 6 deep neural networks. The results show that under the variation of server computation load, LoADPart can reduce the end-to-end latency by 14.2% on average and up to 32.3% in some specific cases.

Vehicular applications in smart cities, such as assisted and autonomous driving, require sophisticated data processing, low latency, and high throughput data transmission. Edge Computing is a leading approach designed to meet those application requirements. By deploying Edge servers at the network's edge, close to the vehicles, such applications can be successfully delivered while adhering to low-latency and high-throughput requirements. However, optimal placement of Edge servers is challenging since it necessitates a trade-off between quality of service and deployment cost. Latency can be reduced by placing as many Edge servers as feasible close to the vehicles, however, this results in significant deployment costs. This work addresses the problem of optimal Edge server placement. It solves this problem using integer linear programming, considering the relation between delay and cost, as well as the capacity of Edge servers in realistic road traffic scenarios. The proposed generic methodology is designed to reduce the cost of deploying Edge servers by combining the achievement of the desired latency threshold with workload balancing between Edge servers. We evaluate the efficiency of the proposed solution mathematically and through simulations based on open data from real vehicles traffic on roadways of Bordeaux, France. The obtained results demonstrate that our solution outperforms existing Edge server placement approaches, especially on workload balancing.

Edge Computing is a distributed computing architecture where the computing process takes place near the user’s physical location or at the source where the data originates. By placing the edge servers (ES) closer to the user’s location, the services received are faster and more reliable while benefiting from edge computing. Various Internet of Vehicle (IoV) applications in smart cities including assisted and autonomous driving, real-time accidents monitoring require huge data processing and low-latency communication. Since the user’s devices are resource constrained, an effective approach to address this constraint is to offload their tasks to nearby ES. So, an adaptive placements (to respond continuously changing environment) of these edge computing devices play a major role in the performance of various IoV applications. So, an efficient placement of ES is considered a critical issue in vehicular edge computing (VEC). To address the efficient and cost aware dynamic ES placement problem (CADEP), we developed two greedy algorithms. First, is cost aware and vehicle density based deployment of ES (static) that ensures that each vehicle’s demand is covered by at least by one ES (coverage constraint), called Greedy_static. Second, is based on vehicle density and is dynamic as per changing environment, called Greedy_dynamic which updates ES locations periodically based on change in the environment. To minimize the relocation cost, we formulated an optimization problem and used Hungarian matching to find optimal cost. For various vehicle densities, we found that our algorithms outperform uniform strategies in terms of cost-effectiveness and ES utilization. Further, for dynamic relocation of ES, we have shown that the cost required to relocate ES randomly is more as compared to our proposed algorithm Greedy_dynamic.

As the deep learning models keep growing larger, it becomes more difficult to carry out highly accurate inference on mobile and Internet of things (IoT) devices with acceptable latency. By using edge computing, a popular approach to reducing inference latency is to partition the deep neural network (DNN) model and off-load part of the model inference to the edge server for processing. However, the partition approach still has some limitations, such as data transmission latency and inflexible model deployment. In this research, we propose a novel two-level inference architecture for deep learning applications in mobile edge computing. Instead of partitioning a large DNN model, we use small models to do two levels of inferences on edge devices and edge servers respectively. With our architecture, the on-device first-level inference is conducted in parallel with the feature uploading to the edge server and different types of second-level models can be flexibly deployed in edge servers according to their environments. Experiments demonstrate that our architecture can decrease computing costs and inference latency in complicated edge environments.

A Novel Lyapunov based Dynamic Resource Allocation for UAVs-assisted Edge Computing

Edge computing, which locates edge servers with limited computing and storage resources at the edge of networks, is expected as a novel architecture for low-latency applications. In edge computing, users offload a task to edge servers due to poor computing resources and batteries of mobile devices so that the edge servers execute the offloaded task and return the result of it to users. Users can thus enjoy various applications without depending on the limitations of mobile devices. However, when the edge server’s load is too heavy, a large number of tasks will be offloaded to distant cloud servers. The long distance between users and cloud servers significantly degrades the quality of mobile applications. To prevent this problem, we propose flow splitting and aggregation schemes to improve offload delay to cloud servers in edge computing. This scheme splits TCP connections between users and cloud servers at the edge server, and then aggregates TCP connections between the edge server and cloud servers. We show the effectiveness of our scheme through simulation evaluations.

Mobile edge computing (MEC) is an emerging computing and communication model that brings extremely resourceful servers at a network's edge. This reduces the latency associated with cloud computing as applications executing on mobile devices do not always need to communicate with a cloud server because most of the functionalities provided by the cloud server are available on an edge server. Moreover, with the emergence of 5G really high amount of bandwidth is available to a mobile device, hence communication latency between the edge server and a mobile device is low. The combination of 5G and edge computing has a potential to better enable many interesting time-critical and/or computationally expensive computing and communication use cases, for example, connected cars. The stated combination enables development of computationally expensive applications for mobile devices and sensors nodes as these devices can offload their computational tasks to the edge server. Task offloading is a promising feature of edge computing, therefore here priority-class-based methods are presented to allocated CPU cycles to an offloaded task at the edge server. In one of the presented methods, a mobile device or a senor node assign their task to one of the available priority classes, and then offload the task to the edge server. Afterwards, the edge server assigns CPU cycles to such a task depending upon its priority class. Similarly, another method is presented that assigns priority to each device based on the type of tasks the device is offloading. While assigning CPU cycles to different offloaded tasks, the edge server uses their device priority. The simulation results demonstrate that the presented methods are capable of providing service differentiation.

With the rapid development of computing power and artificial intelligence, IoT devices equipped with ubiquitous sensors are gradually installed with intelligence. People can enjoy many conveniences with intelligent devices, such as face recognition, video understanding, and motion estimation. Currently, deep neural networks are the mainstream technology in intelligent mobile applications. Inspired by DNN model partition schemes, the paradigm of edge computing could be utilized collaboratively to improve the effectiveness of intelligent task execution in IoT devices. However, due to the dynamics of the wireless network environment and the increasing number of IoT devices, a DNN partition policy without adequate consideration would pose a significant challenge to the efficiency of task inference. Moreover, the shortage and high rental cost of edge computing resources make the optimization of DNN-based task execution more difficult. To cope with those situations, we propose a joint method by a self-adaptive DNN partition with cost-effective resource allocation to facilitate collaborative computation between IoT devices and edge servers. Our proposed online algorithm can be proved to ensure the overall rental cost within an upper bound above the optimal solution while guaranteeing the latency for DNN-based task inference. To evaluate the performance of our strategy, we conduct extensive trace-driven illustrative studies and show that the proposed method can achieve sub-optimal results and outperforms other alternative methods.

Edge computing promises a dramatic reduction in the network latency and the traffic volume, where many edge servers are placed at the edge of the Internet. Furthermore, those edge servers cache data to provide services for edge users. The data sharing among those edge servers can effectively shorten the latency to retrieve the data and further reduce the network bandwidth consumption. The key challenge is to construct an efficient data indexing mechanism no matter how the data is cached in the edge network. Although this is essential, it is still an open problem. Moreover, existing methods such as the centralized indexing and the DHT indexing in other fields fail to meet the performance demand of edge computing. This paper presents a COordinate-based INdexing (COIN) mechanism for the data sharing in edge computing. COIN maintains a virtual space where switches and data indexes are associated with their coordinates. Then, COIN distributes data indexes to indexing edge servers based on those coordinates. The COIN is effective because any query request from an edge server can be responded when the data has been stored in the edge network. More importantly, COIN is efficient in both routing path lengths and forwarding table sizes for publishing/querying data indexes. We implement COIN in a P4 prototype. Experimental results show that COIN uses 59% shorter path length and 30% less forwarding table entries to retrieve data indexes compared to using Chord, a well-known DHT solution.

Efficient resource usage in edge computing requires clever allocation of the workload of application components. In this paper, we show that under certain circumstances, the number of superfluous workload reallocations from one edge server to another may grow to a significant proportion of all user tasks—a phenomenon we present as a reallocation storm. We showcase this phenomenon on a city-scale edge server deployment by simulating the allocation of user task workloads in a number of scenarios capturing likely edge computing deployments and usage patterns. The simulations are based on a large real-world data set of city-wide Wi-Fi network connections, with more than 47M connections over ca. 560 access points. We study the occurrence of reallocation storms in three common edge-based reallocation strategies and compare the latency–workload trade-offs related to each strategy. As a result, we find that the superfluous reallocations vanish when the edge server capacity is increased above a certain threshold, unique for each reallocation strategy, peaking at ca. 35% of the peak ES workload. Further, while a reallocation strategy aiming to minimize latency consistently resulted in the worst reallocation storms, the two other strategies, namely a random reallocation strategy and a bottom-up strategy which always chooses the edge server with the lowest workload as a reallocation target, behave nearly identically in terms of latency as well as the reallocation storm in dense edge deployments. Since the random strategy requires much less coordination, we recommend it over the bottom-up one in dense ES deployments. Moreover, we study the conditions associated with reallocation storms. We discover that edge servers with the very highest workloads are best associated with reallocation storms, with other servers around the few busy nodes thus mirroring their workload. Further, we identify circumstances associated with an elevated risk of reallocation storms, such as summertime (ca. 4 times the risk than on average) and on weekends (ca. 1.5 times the risk). Furthermore, mass events such as popular sports games incurred a high risk (nearly 10 times that of the average) of a reallocation storm in a MEC-based scenario.

Many advancements are being made in vehicular networks, such as self-driving, dynamic route scheduling, real-time traffic condition monitoring, and on-board infotainment services. However, these services require high computation power and precision and can be met using mobile edge computing (MEC) mechanisms for vehicular networks. MEC operates through the edge servers available at the roadside, also known as roadside units (RSU). MEC is very useful for vehicular networks because it has extremely low latency and supports operations that require near-real-time access to rapidly changing data. This paper proposes an efficient computational offloading, smart division of tasks, and cooperation among RSUs to increase service performance and decrease the delay in a vehicular network via MEC. The computational delay is further reduced by parallel processing. In the division of tasks, each task is divided into two sub-components which are fed to a deep neural network (DNN) for training. Consequently, this reduces the overall time delay and overhead. We also adopt an efficient routing policy to deliver the results through the shortest path to improve service reliability. The offloading, computing, division, and routing policies are formulated, and a model-based DNN approach is used to obtain an optimal solution. Simulation results prove that our proposed approach is suitable in a dynamic environment. We also compare our results with the existing state-of-the-art, showing that our proposed approach outperforms the existing schemes.

Recent advances in machine learning and deep learning have enabled many existing applications in smart cities, autonomous systems, and wearable devices. These applications often demand scalable real-time cognitive intelligence and on-the-spot decision making. Current computer systems have been customized for a cloud computing paradigm which often does not meet latency constraints and scalability requirements. To address the limitations of the cloud computing paradigm, the general trend is toward shifting the computation next to data producers at the edge. However, the edge computing paradigm is in the very early stages. Many system-level aspects of edge computing, including algorithms mapping and partitioning across edge computing resources (edge server, and edge nodes) are unknown. New research is required to understand and quantify design dimensions for edge computing.This paper presents a novel edge computing infrastructure for distributed real-time video analytics. This paper presents a holistic solution for co-designing application and edge infrastructure, including edge nodes and edge servers, to enable scalable real-time Artificial Intelligence (AI)/Deep Learning (DL) video analytics across many cameras. For experimental results and evaluation, we focus on the case study of object re-identification across many cameras, which is composed of object detection/classification (TinyYOLOv3), feature extraction, local re-identification, and global re-identification kernels. We evaluate the edge system under three different task mapping and resource allocation configurations. The results present that with the edge nodes (video cameras) more than 32, the only scalable solution is to perform detection/classification (TinyYOLOv3), feature extraction, local re-identification on the edge nodes next to cameras, and execute global re-identification on edge server.

Mobile edge computing (MEC) plays a pivotal role in the Internet of Vehicles and the Internet of Things. Edge server deployment is the initial step in establishing edge computing systems, which impact the overall system performance significantly. Besides, the performance of an edge computing system is also contingent upon the type of service deployed on servers, in the case of the same server deployment, different deployment of services will bring different profits. Most current studies concentrate solely on the former aspect, neglecting the optimization of service deployment in MEC system. In this article, we proposed a two-step method KPSOP for edge server and edge service deployment, aiming to reduce time delay, balance load, and improve the profit of MEC system, and KPSOP includes clustering algorithm and heuristic algorithm. We considered the location distribution of base stations, the task requests of vehicle users, the resource limitations of edge servers, etc. First, the edge server deployment was completed with the goal of minimizing time delay and load balancing. Second, the service deployment was completed with the goal of maximizing edge server profit. The experiments were based on real world base station information. The simulation results validate that our algorithm is more stable and converges faster. In addition, compared to other algorithms, it performs better in load balance and increasing profit.

In this paper, we study key theoretical approaches such as modeling and identification of business process-oriented coupling structure problems from the perspective of resource allocation efficiency of computing, network and storage that can be provided by terminal sensing devices and edge servers of edge computing. A particle swarm intelligent optimization algorithm APSO is proposed for multi-terminal business sequencing. Based on the collected resource information of mobile terminals and edge servers, considering the interaction between multiple business activity requests, we study how to efficiently allocate the resources of edge servers, dynamically make the approximate optimal process sequencing for all applications, and make the average completion time of all business activities and the average energy consumption of mobile terminals and and edge servers are made to be approximately optimal. Finally, a series of experiments are conducted to evaluate the performance of APSO. The experimental results show that the proposed particle swarm optimization algorithm has a good advantage in terms of average completion time and average energy consumption in a multi-terminal business scenario with limited edge server resources.