• Home
  • Search
  • IGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud
  • Open Access IconOpen Access
  • Cite Icon44
  • https://doi.org/10.1109/tpds.2022.3232715Copy DOI Icon

IGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

GPUs are essential to accelerating the latency-sensitive deep neural network (DNN) inference workloads in cloud datacenters. To fully utilize GPU resources, <i>spatial sharing</i> of GPUs among co-located DNN inference workloads becomes increasingly compelling. However, GPU sharing inevitably brings <i>severe performance interference</i> among co-located inference workloads, as motivated by an empirical measurement study of DNN inference on EC2 GPU instances. While existing works on guaranteeing inference performance service level objectives (SLOs) focus on either <i>temporal sharing</i> of GPUs or <i>reactive</i> GPU resource scaling and inference migration techniques, how to <i>proactively</i> mitigate such severe performance interference has received comparatively little attention. In this paper, we propose <i>iGniter</i> , an <i>interference-aware</i> GPU resource provisioning framework for cost-efficiently achieving predictable DNN inference in the cloud. <i>iGniter</i> is comprised of two key components: (1) a <i>lightweight</i> DNN inference performance model, which leverages the system and workload metrics that are practically accessible to capture the performance interference; (2) A <i>cost-efficient</i> GPU resource provisioning strategy that <i>jointly</i> optimizes the GPU resource allocation and adaptive batching based on our inference performance model, with the aim of achieving predictable performance of DNN inference workloads. We implement a prototype of <i>iGniter</i> based on the NVIDIA Triton inference server hosted on EC2 GPU instances. Extensive prototype experiments on four representative DNN models and datasets demonstrate that <i>iGniter</i> can guarantee the performance SLOs of DNN inference workloads with practically acceptable runtime overhead, while saving the monetary cost by up to <inline-formula><tex-math notation="LaTeX">$25\%$</tex-math></inline-formula> in comparison to the state-of-the-art GPU resource provisioning strategies.

Similar Papers
  • Research Article
  • Citations113

Throughput Maximization of Delay-Aware DNN Inference in Edge Computing by Exploring DNN Model Partitioning and Inference Parallelism

  • May 01, 2023
  • IEEE Transactions on Mobile Computing
  • Jing Li +5
  • Conference Article
  • Citations10

Delay-Aware DNN Inference Throughput Maximization in Edge Computing via Jointly Exploring Partitioning and Parallelism

  • Oct 04, 2021
  • Jing Li +4
  • Research Article
  • Citations146

Deep Reinforcement Learning Based Resource Management for DNN Inference in Industrial IoT

  • Mar 24, 2021
  • IEEE Transactions on Vehicular Technology
  • Weiting Zhang +6
  • Book Chapter
  • Citations2

MagicBatch: An Energy-Aware Scheduling Framework for DNN Inference on Heterogeneous Edge Servers in Space-Air-Ground Computation

  • Jan 01, 2023
  • Di Liu +3
  • Book Chapter
  • Citations5

A Strategy to Accelerate the Inference of a Complex Deep Neural Network

  • Jan 01, 2023
  • P Haseena Rahmath +2
  • Research Article
  • Citations8

PLAYS: Minimizing DNN Inference Latency in Serverless Edge Cloud for Artificial Intelligence of Things

  • Dec 01, 2024
  • IEEE Internet of Things Journal
  • Hongmin Geng +5
  • Conference Article
  • Citations281

PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning

  • Mar 09, 2020
  • Wei Niu +7
  • Research Article
  • Citations31

Adaptive Device-Edge Collaboration on DNN Inference in AIoT: A Digital-Twin-Assisted Approach

  • Apr 01, 2024
  • IEEE Internet of Things Journal
  • Shisheng Hu +4
  • Research Article
  • Citations21

Joint Optimization of DNN Partition and Continuous Task Scheduling for Digital Twin-Aided MEC Network With Deep Reinforcement Learning

  • Jan 01, 2023
  • IEEE Access
  • Siyu Yuan +4
  • Conference Article
  • Citations5

Performance Evaluation of State-of-the-Art Edge Computing Devices for DNN Inference

  • Oct 18, 2020
  • Xalo Rancano +4
  • Research Article
  • Citations69

Accelerating DNN Inference With Reliability Guarantee in Vehicular Edge Computing

  • Dec 01, 2023
  • IEEE/ACM Transactions on Networking
  • Kai Liu +4
  • Research Article
  • Citations11

Joint Optimization of the Partition and Scheduling of DNN Tasks in Computing and Network Convergence

  • Jun 01, 2023
  • IEEE Networking Letters
  • Zhenyu Zhang +4
  • Research Article
  • Citations2

NeFT: Negative Feedback Training to Improve Robustness of Compute-in-Memory DNN Accelerators

  • Feb 01, 2026
  • IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
  • Yifan Qin +7
  • Conference Article
  • Citations28

COINN: Crypto/ML Codesign for Oblivious Inference via Neural Networks

  • Nov 12, 2021
  • Siam Umar Hussain +3
  • Research Article
  • Citations44

Elastic-DF: Scaling Performance of DNN Inference in FPGA Clouds through Automatic Partitioning

  • Dec 06, 2021
  • ACM Transactions on Reconfigurable Technology and Systems
  • Tobias Alonso +8
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.