• Home
  • Search
  • SoftMemoryBox II: A Scalable, Shared Memory Buffer Framework for Accelerating Distributed Training of Large-Scale Deep Neural Networks
  • Cite Icon4
  • https://doi.org/10.1109/access.2020.3038112Copy DOI Icon

SoftMemoryBox II: A Scalable, Shared Memory Buffer Framework for Accelerating Distributed Training of Large-Scale Deep Neural Networks

Show More
  • Abstract
  • Highlights & Summary
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Distributed processing using high-performance computing resources is essential for developers to train large-scale deep neural networks (DNNs). The major impediment to distributed DNN training is the communication bottleneck during the parameter exchange among the distributed DNN training workers. The communication bottleneck increases training time and decreases the utilization of the computational resources. Our previous study, SoftMemoryBox (SMB1) presented considerably superior performance compared to message passing interface (MPI) in the parameter communication of distributed DNN training. However, SMB1 had disadvantages such as the limited scalability of the distributed DNN training due to the restricted communication bandwidth from a single memory server, inability to provide a synchronization function for the shared memory buffer, and low portability/usability as a consequence of the kernel-level implementation. This paper proposes a scalable, shared memory buffer framework, called SoftMemoryBox II (SMB2), which overcomes the shortcomings of SMB1. With SMB2, distributed training processes can easily share virtually unified shared memory buffers composed of memory segments provided from remote memory servers and can exchange DNN parameters at high speed through the shared memory buffer. The scalable communication bandwidth of the SMB2 framework facilitates the reduction of DNN distributed training times compared to SMB1. According to intensive evaluation results, the communication bandwidth of the proposed SMB2 is 6.3 times greater than that of SMB1 when the SMB2 framework is scaled out to use eight memory servers. Moreover, the training time of SMB2-based asynchronous distributed training of five DNN models is up to 2.4 times faster than SMB1-based training.

Similar Papers
  • Conference Article
  • Citations8

A Framework for Distributed Deep Neural Network Training with Heterogeneous Computing Platforms

  • Dec 01, 2019
  • Bontak Gu +3
  • Research Article
  • Citations10

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

  • Aug 01, 2023
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Qing Ye +4
  • Research Article
  • Citations9

Adaptive partitioning and efficient scheduling for distributed DNN training in heterogeneous IoT environment

  • Dec 30, 2023
  • Computer Communications
  • Binbin Huang +5
  • Conference Article
  • Citations25

Huffman Coding Based Encoding Techniques for Fast Distributed Deep Learning

  • Dec 01, 2020
  • Rishikesh R Gajjala +5
  • Research Article
  • Citations3

Scale-Train: A Scalable DNN Training Framework for a Heterogeneous GPU Cloud

  • Jan 01, 2022
  • IEEE Access
  • Kyeonglok Kim +3
  • Research Article
  • Citations11

Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster

  • Feb 22, 2021
  • Journal of Grid Computing
  • Jie Xu +5
  • Conference Article
  • Citations2

Cannikin: Optimal Adaptive Distributed DNN Training over Heterogeneous Clusters

  • Dec 02, 2024
  • Chengyi Nie +2
  • Research Article
  • Citations5

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

  • Aug 01, 2024
  • IEEE Transactions on Emerging Topics in Computational Intelligence
  • Xiaofeng Zou +5
  • Conference Article

Deep Learning Framework using Scalable Shared Memory Buffer Framework

  • Jan 31, 2021
  • Eun-Ji Lim +1
  • Conference Article
  • Citations12

High performance distributed deep learning

  • Feb 16, 2019
  • Dhabaleswar K (Dk) Panda +2
  • Conference Article
  • Citations11

Efficient All-Reduce for Distributed DNN Training in Optical Interconnect Systems

  • Feb 21, 2023
  • Fei Dai +4
  • Research Article
  • Citations54

A Robustness-Assured White-Box Watermark in Neural Networks

  • Nov 01, 2023
  • IEEE Transactions on Dependable and Secure Computing
  • Peizhuo Lv +7
  • Conference Article
  • Citations14

Near-Optimal Topology-adaptive Parameter Synchronization in Distributed DNN Training

  • May 10, 2021
  • Zhe Zhang +2
  • Research Article
  • Citations2

Priority-based parameter propagation for distributed deep neural network training

  • Jan 01, 2019
  • Open Collections
  • Anand Jayarajan
  • Research Article
  • Citations23

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

  • Aug 04, 2023
  • Neurocomputing
  • Jinghui Zhang +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.