• Home
  • Search
  • Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters
  • Cite Icon2
  • https://doi.org/10.1145/3652892.3700767Copy DOI Icon

Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters

  • Dec 2, 2024
  • Chengyi Nie +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Adjusting batch sizes and adaptively tuning other hyperparameters can significantly speed up deep neural network (DNN) training. Despite the ubiquity of heterogeneous clusters, existing adaptive DNN training techniques solely consider homogeneous environments. Optimizing distributed DNN training over heterogeneous clusters is technically challenging, and directly adapting existing techniques results in low utilization and poor performance. To solve this problem, we introduce Cannikin - a novel data-parallel distributed training system. Cannikin achieves efficient and near optimal performance by accurately modeling the optimal system performance and predicting adaptive batch size training metrics for DNNs in heterogeneous clusters. We implemented Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon. Empirical results show that Cannikin reduces DNN training in heterogeneous clusters by up to 52% compared to the state-of-art adaptive training system and up to 85% compared to native PyTorch DistributedDataParallel.

Similar Papers
  • Conference Article
  • Citations8

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 01, 2019
  • Bontak Gu +3
  • Research Article
  • Citations179

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

  • May 04, 2021
  • IEEE Transactions on Artificial Intelligence
  • Edgar Galván +1
  • Research Article
  • Citations10

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

  • Aug 01, 2023
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Qing Ye +4
  • Research Article
  • Citations3

Scale-Train: A Scalable DNN Training Framework for a Heterogeneous GPU Cloud

  • Jan 01, 2022
  • IEEE Access
  • Kyeonglok Kim +3
  • Conference Article
  • Citations11

Efficient All-Reduce for Distributed DNN Training in Optical Interconnect Systems

  • Feb 21, 2023
  • Fei Dai +4
  • Research Article
  • Citations23

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

  • Aug 04, 2023
  • Neurocomputing
  • Jinghui Zhang +6
  • Conference Article
  • Citations3

AccDP: Accelerated Data-Parallel Distributed DNN Training for Modern GPU-Based HPC Clusters

  • Dec 01, 2022
  • Nawras Alnaasan +4
  • Conference Article
  • Citations3

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

  • Oct 28, 2023
  • Vahid Janfaza +3
  • Research Article
  • Citations59

HNPU: An Adaptive DNN Training Processor Utilizing Stochastic Dynamic Fixed-Point and Active Bit-Precision Searching

  • Mar 24, 2021
  • IEEE Journal of Solid-State Circuits
  • Donghyeon Han +6
  • Research Article
  • Citations6

A Guessing Entropy-Based Framework for Deep Learning-Assisted Side-Channel Analysis

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Ziyue Zhang +2
  • Conference Article
  • Citations4

MalleTrain: Deep Neural Networks Training on Unfillable Supercomputer Nodes

  • May 07, 2024
  • Xiaolong Ma +6
  • Research Article
  • Citations2

Accelerating deep network training for radar identification using batch normalization

  • Nov 25, 2024
  • Defence Science Journal
  • Preeti Gupta +2
  • Research Article
  • Citations2

Priority-based parameter propagation for distributed deep neural network training

  • Jan 01, 2019
  • Open Collections
  • Anand Jayarajan
  • Research Article
  • Citations4

SoftMemoryBox II: A Scalable, Shared Memory Buffer Framework for Accelerating Distributed Training of Large-Scale Deep Neural Networks

  • Jan 01, 2020
  • IEEE Access
  • Shinyoung Ahn +1
  • Research Article
  • Citations52

TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems

  • Mar 31, 2021
  • IEEE Transactions on Very Large Scale Integration (VLSI) Systems
  • Sourjya Roy +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.