• Home
  • Search
  • MalleTrain: Deep Neural Networks Training on Unfillable Supercomputer Nodes
  • Cite Icon4
  • https://doi.org/10.1145/3629526.3645035Copy DOI Icon

MalleTrain: Deep Neural Networks Training on Unfillable Supercomputer Nodes

  • May 7, 2024
  • Xiaolong Ma +6 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

First-come first-serve scheduling can result in substantial (up to 10%) of transiently idle nodes on supercomputers. Recognizing that such unfilled nodes are well-suited for deep neural network (DNN) training, due to the flexible nature of DNN training tasks, Liu et al. proposed that the re-scaling DNN training tasks to fit gaps in schedules be formulated as a mixed-integer linear programming (MILP) problem, and demonstrated via simulation the potential benefits of the approach. Here, we introduce MalleTrain, a system that provides the first practical implementation of this approach and that furthermore generalizes it by allowing it to be used even for DNN training applications for which model information is unknown before runtime. Key to this latter innovation is the use of a lightweight online job profiling advisor (JPA) to collect critical scalability information for DNN jobs---information that it then employs to optimize resource allocations dynamically, in real time. We describe the MalleTrain architecture and present the results of a detailed experimental evaluation on a supercomputer GPU cluster and several representative DNN training workloads, including neural architecture search and hyperparameter optimization. Our results not only confirm the practical feasibility of leveraging idle supercomputer nodes for DNN training but improve significantly on prior results, improving training throughput by up to 22.3% without requiring users to provide job scalability information.

Similar Papers
  • Research Article
  • Citations179

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

  • May 04, 2021
  • IEEE Transactions on Artificial Intelligence
  • Edgar Galván +1
  • Conference Article
  • Citations3

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

  • Oct 28, 2023
  • Vahid Janfaza +3
  • Conference Article
  • Citations8

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 01, 2019
  • Bontak Gu +3
  • Research Article
  • Citations6

A Guessing Entropy-Based Framework for Deep Learning-Assisted Side-Channel Analysis

  • Jan 01, 2023
  • IEEE Transactions on Information Forensics and Security
  • Ziyue Zhang +2
  • Conference Article
  • Citations3

AccDP: Accelerated Data-Parallel Distributed DNN Training for Modern GPU-Based HPC Clusters

  • Dec 01, 2022
  • Nawras Alnaasan +4
  • Research Article
  • Citations52

TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems

  • Mar 31, 2021
  • IEEE Transactions on Very Large Scale Integration (VLSI) Systems
  • Sourjya Roy +3
  • Research Article
  • Citations59

HNPU: An Adaptive DNN Training Processor Utilizing Stochastic Dynamic Fixed-Point and Active Bit-Precision Searching

  • Mar 24, 2021
  • IEEE Journal of Solid-State Circuits
  • Donghyeon Han +6
  • Conference Article
  • Citations2

Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters

  • Dec 02, 2024
  • Chengyi Nie +2
  • Research Article
  • Citations2

Accelerating deep network training for radar identification using batch normalization

  • Nov 25, 2024
  • Defence Science Journal
  • Preeti Gupta +2
  • Conference Article
  • Citations7

Comparison of Loss Functions for Training of Deep Neural Networks in Shogi

  • Nov 01, 2018
  • Hanhua Zhu +1
  • Conference Article
  • Citations11

Efficient All-Reduce for Distributed DNN Training in Optical Interconnect Systems

  • Feb 21, 2023
  • Fei Dai +4
  • Conference Article
  • Citations36

Gradient Compression Supercharged High-Performance Data Parallel DNN Training

  • Oct 26, 2021
  • Youhui Bai +7
  • Conference Article
  • Citations64

FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding

  • Apr 01, 2022
  • Sai Qian Zhang +2
  • Research Article
  • Citations2

A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training

  • Aug 01, 2025
  • IEEE Transactions on Parallel and Distributed Systems
  • Hao Wu +8
  • Conference Article
  • Citations15

PCM: Precision-Controlled Memory System for Energy Efficient Deep Neural Network Training

  • Mar 01, 2020
  • Boyeal Kim +9
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.