• Home
  • Search
  • A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms
  • Cite Icon8
  • https://doi.org/10.1109/icpads47876.2019.00068Copy DOI Icon

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 1, 2019
  • Bontak Gu +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Deep neural network (DNN) training is generally performed by cloud computing platforms. However, cloud-based training has several problems such as network bottleneck, server management cost, and privacy. To overcome these problems, one of the most promising solutions is distributed DNN model training which trains the model with not only high-performance servers but also low-end power-efficient mobile edge or user devices. However, due to the lack of a framework which can provide an optimal cluster configuration (i.e., determining which computing devices participate in DNN training tasks), it is difficult to perform efficient DNN model training considering DNN service providers' preferences such as training time or energy efficiency. In this paper, we introduce a novel framework for distributed DNN training that determines the best training cluster configuration with available heterogeneous computing resources. Our proposed framework utilizes pre-training with a small number of training steps and estimates training time, power, energy, and energy-delay product (EDP) for each possible training cluster configuration. Based on the estimated metrics, our framework performs DNN training for the remaining steps with the chosen best cluster configurations depending on DNN service providers' preferences. Our framework is implemented in TensorFlow and evaluated with three heterogeneous computing platforms and five widely used DNN models. According to our experimental results, in 76.67% of the cases, our framework chooses the best cluster configuration depending on DNN service providers' preferences with only a small training time overhead.

Similar Papers
  • Research Article
  • Citations4

SoftMemoryBox II: A Scalable, Shared Memory Buffer Framework for Accelerating Distributed Training of Large-Scale Deep Neural Networks

  • Jan 01, 2020
  • IEEE Access
  • Shinyoung Ahn +1
  • Research Article
  • Citations10

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

  • Aug 01, 2023
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Qing Ye +4
  • Conference Article
  • Citations3

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

  • Oct 28, 2023
  • Vahid Janfaza +3
  • Research Article
  • Citations2

Priority-based parameter propagation for distributed deep neural network training

  • Jan 01, 2019
  • Open Collections
  • Anand Jayarajan
  • Research Article
  • Citations179

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

  • May 04, 2021
  • IEEE Transactions on Artificial Intelligence
  • Edgar Galván +1
  • Conference Article
  • Citations2

Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters

  • Dec 02, 2024
  • Chengyi Nie +2
  • Conference Article
  • Citations11

Efficient All-Reduce for Distributed DNN Training in Optical Interconnect Systems

  • Feb 21, 2023
  • Fei Dai +4
  • Research Article
  • Citations3

Scale-Train: A Scalable DNN Training Framework for a Heterogeneous GPU Cloud

  • Jan 01, 2022
  • IEEE Access
  • Kyeonglok Kim +3
  • Research Article
  • Citations6

A Guessing Entropy-Based Framework for Deep Learning-Assisted Side-Channel Analysis

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Ziyue Zhang +2
  • Conference Article
  • Citations49

Cynthia

  • Aug 05, 2019
  • Haoyue Zheng +4
  • Conference Article
  • Citations3

AccDP: Accelerated Data-Parallel Distributed DNN Training for Modern GPU-Based HPC Clusters

  • Dec 01, 2022
  • Nawras Alnaasan +4
  • Conference Article
  • Citations36

Gradient Compression Supercharged High-Performance Data Parallel DNN Training

  • Oct 26, 2021
  • Youhui Bai +7
  • Research Article
  • Citations2

A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training

  • Aug 01, 2025
  • IEEE Transactions on Parallel and Distributed Systems
  • Hao Wu +8
  • Research Article

SVRG-AALR: Stochastic Variance-Reduced Gradient Method with Adaptive Alternating Learning Rate for Training Deep Neural Networks

  • Jul 25, 2025
  • Electronics
  • Shiyun Zou +3
  • Research Article
  • Citations23

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

  • Aug 04, 2023
  • Neurocomputing
  • Jinghui Zhang +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.