LoADPart: Load-Aware Dynamic Partition of Deep Neural Networks for Edge Offloading

Device-edge collaboration on deep neural network (DNN) inference is a promising approach to efficiently utilizing network resources for supporting Artificial Intelligence of Things (AIoT) applications. In this article, we propose a novel digital twin (DT)-assisted approach to device-edge collaboration on DNN inference that determines whether and when to stop local inference at a device and upload the intermediate results to complete the inference on an edge server. Instead of determining the collaboration for each DNN inference task only upon its generation, multi-step decision making is performed during the on-device inference to adapt to the dynamic computing workload status at the device and the edge server. To enhance the adaptivity, a DT is constructed to evaluate all potential offloading decisions for each DNN inference task, which provides augmented training data for a machine learning-assisted decision-making algorithm. Then, another DT is constructed to estimate the inference status at the device to avoid frequently fetching the status information from the device, thus reducing the signaling overhead. We also derive necessary conditions for optimal offloading decisions to reduce the offloading decision space. Simulation results demonstrate the outstanding performance of our DT-assisted approach in terms of balancing the tradeoff among inference accuracy, delay, and energy consumption.

Many intelligent applications, such as autonomous driving and virtual reality, require running both latency-critical (real-time) and best-effort deep neural network (DNN) inference tasks to achieve both real-time and work-conserving on the GPU. However, commodity GPUs lack efficient preemptive scheduling support, and existing state-of-the-art approaches either have to monopolize GPU or let real-time tasks to wait for best-effort tasks to complete, resulting in low utilization, high latency, or both. This article presents Reef , the first GPU-accelerated DNN inference serving system that achieves low-latency and work-conserving for concurrent real-time and best-effort tasks. Reef accomplishes this by enabling microsecond-scale kernel preemption and controlled concurrent execution in GPU scheduling. Reef is novel in two ways. First, based on the observation that DNN inference kernels are mostly idempotent, Reef devises a reset-based preemption scheme that launches a real-time kernel on the GPU by proactively killing and restoring best-effort kernels at microsecond-scale. Second, since DNN inference kernels have varied parallelism and predictable latency, Reef proposes a dynamic kernel padding mechanism that dynamically pads the real-time kernel with appropriate best-effort kernels to fully utilize the GPU with negligible overhead. Evaluation using a new DNN inference serving benchmark (DISB) with diverse workloads and a real-world trace on both NVIDIA and AMD GPUs shows that Reef only incurs less than 5% overhead in end-to-end latency for real-time tasks but increases the overall throughput by up to 1.53×, compared to scheduling tasks sequentially. To demonstrate the practical benefits of our approach, we compare Reef with Triton, a widely-adopted production-level serving system. Our evaluation shows that Reef outperforms Triton by 1.12× to 5.20× in end-to-end latency for real-time tasks, while maintaining comparable throughput.

Emerging mobile applications, such as cognitive assistance based on deep neural network (DNN), require low latency as well as high computation power. To meet these requirements, edge computing (also called fog computing) has been proposed, which offloads computations to edge servers located near mobile clients. This paradigm shift from cloud to edge requires new computing infrastructure where edge servers are pervasively distributed over a region.This paper presents PerDNN, a system that executes DNNs of mobile clients collaboratively with pervasive edge servers. PerDNN dynamically partitions DNN computation between a client and an edge server to minimize execution latency. It predicts the next edge server the client will visit, calculates a speculative partitioning plan, and transfers the server-side DNN layers to the predicted server in advance, which reduces the initialization overhead needed to start offloading, thus avoiding cold starts. We do not incur excessive network traffic between edge servers, though, by migrating only a tiny fraction of the server-side DNN layers with negligible performance loss. We also use GPU statistics of edge servers for DNN partitioning to deal with the resource contention caused by multi-client offloading. In the simulation with human trace datasets and execution profile of real hardware, PerDNN reduced the occurrence of cold starts by up to 90%, achieving 58% higher throughput when clients change their offloading servers, compared to a baseline without proactive DNN transmission.

Deep neural networks (DNNs) have become a critical component for inference in modern mobile applications, but the efficient provisioning of DNNs is non-trivial. Existing mobile- and server-based approaches compromise either the inference accuracy or latency. Instead, a hybrid approach can reap the benefits of the two by splitting the DNN at an appropriate layer and running the two parts separately on the mobile and the server respectively. Nevertheless, the DNN throughput in the hybrid approach has not been carefully examined, which is particularly important for edge servers where limited compute resources are shared among multiple DNNs. This article presents HiTDL, a runtime framework for managing multiple DNNs provisioned following the hybrid approach at the edge. HiTDL's mission is to improve edge resource efficiency by optimizing the combined throughput of all co-located DNNs, while still guaranteeing their SLAs. To this end, HiTDL first builds comprehensive performance models for DNN inference latency and throughout with respect to multiple factors including resource availability, DNN partition plan, and cross-DNN interference. HiTDL then uses these models to generate a set of candidate partition plans with SLA guarantees for each DNN. Finally, HiTDL makes global throughput-optimal resource allocation decisions by selecting partition plans from the candidate set for each DNN via solving a fairness-aware multiple-choice knapsack problem. Experimental results based on a prototype implementation show that HiTDL improves the overall throughput of the edge by <inline-formula><tex-math notation="LaTeX">$4.3\times$</tex-math></inline-formula> compared with the state-of-the-art.

This paper covers machine learning approaches to improve the accuracy of a Deep Neural Network (DNN) inference in computer vision, particularly in image segmentation. A few benefits of object recognition aided by computer vision is to help aircraft crew avoid adversaries in various weather conditions, to offload the crew's attention from finding desired objects as well as estimating the position of those objects throughout missions. Since DNN inference is trained without explicit programming, it is challenging to precisely locate its sub-optimality when it performs less accurate prediction. This paper explores three machine learning approaches that improve a DNN model prediction accuracy: 1) Prediction result analysis using the author's original work called Directional Pixel Delta as well as Jaccard's Intersection over Union (IoU) loss function 2) Bayesian optimization hyper-parameter tuning 3) Training data optimization via Bootstrap Aggregating.

Deep Neural Network (DNN) inference has become a cornerstone of latency-sensitive applications such as autonomous driving and augmented reality. While GPUs offer high throughput for DNN inference, they often suffer from underutilization due to coarse-grained scheduling and limited concurrency. Existing GPU-sharing methods either lack awareness of fine-grained kernel interference or fail to meet service-level objectives (SLOs) under multi-tenant, multi-priority workloads. In this paper, we propose IasRT, a runtime framework that enables interferenceaware and SLO-driven GPU sharing for real-time DNN inference. IasRT profiles kernel-level resource usage and interference sensitivity, and dynamically partitions GPU streaming multiprocessors (SMs) to collocate jobs with minimal performance degradation. Furthermore, it introduces a dynamic SLO controller to maintain latency targets for multiple latency-sensitive (LS) jobs simultaneously. Evaluations on real-world DNN workloads show that IasRT reduces the 99th percentile latency of LS jobs by up to 38% compared to the state-of-the-art GPU sharing methods, while maintaining similar overall throughput from multiple collocated workloads, demonstrating its effectiveness in high-concurrency environments.

Deep Neural Networks (DNNs) have been successfully applied to various real-world machine learning applications. However, performing large DNN inference tasks in real-time remains a challenge due to its substantial computational costs. Recently, Spiking Neural Networks (SNNs) have emerged as an alternative way of processing DNN'fs task. Due to its eventbased, data-driven computation, SNN reduces both inference latency and complexity. With efficient conversion methods from traditional DNN, SNN exhibits similar accuracy, while leveraging many state-of-the-art network models and training methods. In this work, an efficient neuromorphic hardware architecture for image recognition task is presented. To preserve accuracy, the analog-to-spiking conversion algorithm is adopted. The system aims to minimize hardware area cost and power consumption, enabling neuromorphic hardware processing in edge devices. Simulation results have shown that, with the MNIST digit recognition task, the system has achieved $\times 20$ reduction in terms of core area cost compared to the state-of-the-art works, with an accuracy of 94.4%, core area of 15 $\mu m^{2}$ at a maximum frequency of 250 MHz.

Edge video analytics enables agile responses of machine-centric applications by streaming videos from end devices to edge servers (ESs) for resource-intensive deep neural network (DNN) inference. Quality of Inference (QoI), reflected by end-to-end analytics delay and inference accuracy, is crucial for the high-quality real-time decision-making. Due to the limited ES resources and dynamic network conditions, video configuration adaptation and fine-grained resource allocation are necessary to enhance the QoI of video streams, which involves a delicate balance between delay and accuracy. In this article, we propose an edge-enabled multivideo analytics framework, in which performing DNN inference relies on heterogeneous resources, including CPU, GPU, and memory. Considering the different impacts of heterogeneous resources on QoI and double-queue backlogs, we formulate the joint video configuration adaptation, CPU-GPU resource allocation, and batch size selection (JCCGB) problem based on semi-Markov decision process to maximize the long-term average QoI. Then, Lyapunov optimization is applied to transform the original problem into one that minimizes the Lyapunov drift-plus-penalty upper bound, thereby ensuring the stability of both the transmission and computation queues. To tackle the potential multimodality of the optimal configuration adaptation and resource allocation policy, we propose the diffusion-deep-reinforcement-learning-based JCCGB (DD-JCCGB) scheme. This approach effectively enhances the policy performance and accelerates the training process. Experiments driven by real-world network traces demonstrate that the DD-JCCGB algorithm improves the long-term average QoI and outperforms baseline schemes.

As the deep learning models keep growing larger, it becomes more difficult to carry out highly accurate inference on mobile and Internet of things (IoT) devices with acceptable latency. By using edge computing, a popular approach to reducing inference latency is to partition the deep neural network (DNN) model and off-load part of the model inference to the edge server for processing. However, the partition approach still has some limitations, such as data transmission latency and inflexible model deployment. In this research, we propose a novel two-level inference architecture for deep learning applications in mobile edge computing. Instead of partitioning a large DNN model, we use small models to do two levels of inferences on edge devices and edge servers respectively. With our architecture, the on-device first-level inference is conducted in parallel with the feature uploading to the edge server and different types of second-level models can be flexibly deployed in edge servers according to their environments. Experiments demonstrate that our architecture can decrease computing costs and inference latency in complicated edge environments.

Hardware accelerators that exploit analog in-memory computing offer an energy-efficient edge deployment solution for machine learning algorithms. We give an overview of the device requirements and hardware-software co-design principles for these systems to achieve efficient and accurate deep neural network (DNN) inference. We designed and fabricated a 40nm test chip with a $1024 \times 1024$ SONOS (siliconoxide-nitride-oxide-silicon) charge trapping memory array for DNN inference. Operating the SONOS memory in the subthreshold regime suppresses the effects of device variability on algorithm accuracy. We experimentally demonstrate accurate DNN inference using the test chip on CIFAR-100 image classification and project a chip-level efficiency of >50 TOPS/W for the SONOS inference accelerator, a $10 \times$ advantage over state-of-the-art digital inference accelerators.

Pipelining on Edge Tensor Processing Units (TPUs) optimizes the deep neural network (DNN) inference by breaking it down into multiple stages processed concurrently on multiple accelerators. Such DNN inference tasks can be modeled as sporadic non-preemptive gangs with execution times that vary with their parallelism levels. This paper proposes a strict partitioning strategy for deploying DNN inferences in real-time systems. The strategy determines tasks' parallelism levels and assigns tasks to disjoint processor partitions. Configuring the tasks in the same partition with a uniform parallelism level avoids scheduling anomalies and enables schedulability verification using well-understood uniprocessor analyses. Evaluation using real-world Edge TPU benchmarks demonstrated that the proposed method achieves a higher schedulability ratio than state-of-the-art gang scheduling techniques.

Deep neural network (DNN) models are increasingly popular in edge video analytic applications. However, the computeintensive nature of DNN models pose challenges for energyefficient inference on resource-constrained edge devices. Most existing solutions focus on optimizing DNN inference latency and accuracy, often overlooking energy efficiency. They also fail to account for the varying complexity of video frames, leading to sub-optimal performance in edge video analytics. In this paper, we propose an EnergyEfficient Early-Exit (E4) framework that enhances DNN inference efficiency for edge video analytics by integrating a novel early-exit mechanism with dynamic voltage and frequency scaling (DVFS) governors. It employs an attentionbased cascade module to analyze video frame diversity and automatically determine optimal DNN exit points. Additionally, E4 features a just-in-time (JIT) profiler that uses coordinate descent search to co-optimize CPU and GPU clock frequencies for each layer before the DNN exit points. Extensive evaluations demonstrate that E4 outperforms current state-of-the-art methods, achieving up to 2.8× speedup and 26% average energy saving while maintaining high accuracy.

Running multiple deep neural networks (DNNs) simultaneously on mobile devices introduces challenges due to constrained computing resources. Previous research has explored the use of heterogeneous processors for accelerating DNN inference but often overlooks thermal issues, which can degrade computing power. In this article, we propose Phoenix, a system specifically designed to enhance the performance of multi-instance DNNs in video applications by maximizing accuracy and ensuring the achievement of a required frame rate. Phoenix allocates DNN tasks to the most suitable hardware processors, understanding complex thermal dynamics through reinforcement learning, and postpones the onset of thermal throttling. Despite optimized task allocation, continuous inference of multiple DNNs can still lead to thermal throttling. To manage performance degradation, Phoenix employs a multi-exit network, adaptively executing inference tasks to ensure consistent frame rates. Phoenix minimizes accuracy loss from early exits by optimally generating and operating multi-exit networks. We evaluated Phoenix using two different benchmarks and Virtual Youtuber streaming application. The results demonstrated that Phoenix effectively enhances device performance by delaying thermal throttling and achieving optimal accuracy while maintaining a consistent frame rate.

The recent surge of interest in Deep Neural Networks (DNNs) has led to increasingly complex networks that tax computational and memory resources. Many DNNs presently use 16-bit or 32-bit floating point operations. Significant performance and power gains can be obtained when DNN accelerators support low-precision numerical formats. Despite considerable research, there is still a knowledge gap on how low-precision operations can be realized for both DNN training and inference. In this work, we propose a DNN architecture, Deep Positron, with posit numerical format operating successfully at ≤8 bits for inference. We propose a precision-adaptable FPGA soft core for exact multiply-and-accumulate for uniform comparison across three numerical formats, fixed, floating-point and posit. Preliminary results demonstrate that 8-bit posit has better accuracy than 8-bit fixed or floating-point for three different low-dimensional datasets. Moreover, the accuracy is comparable to 32-bit floating-point on a Xilinx Virtex-7 FPGA device. The trade-offs between DNN performance and hardware resources, i.e. latency, power, and resource utilization, show that posit outperforms in accuracy and latency at 8-bit and below.

Deep neural networks (DNN) have become indispensable tools for intelligent applications today. The demand for deploying DNN on the edge devices increases dramatically. Unfortunately, it is challenging because the DNN inference is computation-intensive, but edge devices are always resource-constraint. Prior solutions attempted to address these challenges with collaboration between cloud and edge devices, but they do not take the inference request rate into account. However, the inference delay will increase dramatically while the request rate becomes higher.In this paper, we propose a scheme to dynamic partition DNN into two or three parts and distribute them at the edge and cloud, achieving the lowest delay with the change of request rate. The scheme selects the optimal partition points of DNN with a layer evaluation model (LEM) and a total delay prediction model (DPM) under different request rates. The experiments of distributed deploying AlexNet, VGG, NiN and ResNet DNN models on image classification dataset ImageNet show that the proposed scheme significantly reduces the total end-to-end latency by fully using both the edge and cloud resources. It reduces the inference delay by 1.3 to 1.6 times and improves the throughput 1.2 to 1.7 times compared to the state of art partition approach.