• Home
  • Search
  • FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices
  • Cite Icon34
  • https://doi.org/10.1145/3636534.3649391Copy DOI Icon

FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices

  • May 29, 2024
  • Xiangyu Li +4 more
Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Due to the popularity of deep neural networks (DNNs) and considerations over network overhead, data privacy, and inference latency, there is a growing interest in deploying DNNs to edge devices in recent years. However, the limited memory becomes a major bottleneck for on-device DNN deployment, making it crucial to reduce the memory footprint of DNN. The mainstream model customization solutions require intensive deployment efforts and may lead to severe accuracy degradation, and existing deep learning (DL) frameworks don't take memory as a priority. Besides, recent works to enhance the memory management scheme cannot be directly applied because of several challenges, including the unbalanced memory footprint across layers, the inevitable overhead of memory management, and the memory budget dynamicity. To tackle these challenges, we introduce FlexNN, an efficient and adaptive memory management framework for DNN inference on memory-constrained devices. FlexNN uses a slicing-loading-computing joint planning approach, to achieve optimal memory utilization and minimal memory management overhead. We implemented FlexNN atop NCNN, and conducted comprehensive evaluations with common model architectures on various devices. The results have shown that our approach is able to adapt to different memory constraints with optimal latency-memory trade-offs. For example, FlexNN can reduce the memory consumption by 93.81% with only a 3.64% increase in latency, as compared with the original NCNN on smartphones.

Loading PDF

Similar Papers
  • Research Article
  • Citations28

PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End Latency

  • Jan 10, 2024
  • ACM Transactions on Embedded Computing Systems
  • Soumendu Kumar Ghosh +3
  • Research Article
  • Citations21

Joint Optimization of DNN Partition and Continuous Task Scheduling for Digital Twin-Aided MEC Network With Deep Reinforcement Learning

  • Jan 01, 2023
  • IEEE Access
  • Siyu Yuan +4
  • Research Article
  • Citations77

An adaptive DNN inference acceleration framework with end–edge–cloud collaborative computing

  • Nov 04, 2022
  • Future Generation Computer Systems
  • Guozhi Liu +6
  • Research Article
  • Citations113

Throughput Maximization of Delay-Aware DNN Inference in Edge Computing by Exploring DNN Model Partitioning and Inference Parallelism

  • May 01, 2023
  • IEEE Transactions on Mobile Computing
  • Jing Li +5
  • Conference Article
  • Citations5

Performance Evaluation of State-of-the-Art Edge Computing Devices for DNN Inference

  • Oct 18, 2020
  • Xalo Rancano +4
  • Conference Article
  • Citations10

Delay-Aware DNN Inference Throughput Maximization in Edge Computing via Jointly Exploring Partitioning and Parallelism

  • Oct 04, 2021
  • Jing Li +4
  • Conference Article
  • Citations2

Model Parallelism Optimization for Distributed DNN Inference on Edge Devices

  • Nov 24, 2023
  • Meng Wang +4
  • Research Article
  • Citations20

Cooperative Inference of DNNs for Delay- and Memory-Constrained Wireless IoT Systems

  • Sep 01, 2022
  • IEEE Internet of Things Journal
  • Sangseok Yun +2
  • Conference Article
  • Citations3

Towards Ultra-Efficient DNN Inference Acceleration on Edge Devices for Wellbeing Applications

  • Jun 19, 2020
  • Yanzhi Wang
  • Research Article
  • Citations26

Performance analysis of local exit for distributed deep neural networks over cloud and edge computing

  • Oct 01, 2020
  • ETRI Journal
  • Changsik Lee +3
  • Research Article
  • Citations146

Deep Reinforcement Learning Based Resource Management for DNN Inference in Industrial IoT

  • Mar 24, 2021
  • IEEE Transactions on Vehicular Technology
  • Weiting Zhang +6
  • PDF
  • Research Article
  • Citations7

LHDNN: Maintaining High Precision and Low Latency Inference of Deep Neural Networks on Encrypted Data

  • Apr 11, 2023
  • Applied Sciences
  • Jiaming Qian +5
  • PDF
  • Research Article
  • Citations6

Auto-Tiler: Variable-Dimension Autoencoder with Tiling for Compressing Intermediate Feature Space of Deep Neural Networks for Internet of Things

  • Jan 29, 2021
  • Sensors (Basel, Switzerland)
  • Jeongsoo Park +2
  • Research Article
  • Citations44

IGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud

  • Mar 01, 2023
  • IEEE Transactions on Parallel and Distributed Systems
  • Fei Xu +6
  • Research Article
  • Citations8

PLAYS: Minimizing DNN Inference Latency in Serverless Edge Cloud for Artificial Intelligence of Things

  • Dec 01, 2024
  • IEEE Internet of Things Journal
  • Hongmin Geng +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.