TENSILE: A Tensor Granularity Dynamic GPU Memory Scheduling Method Toward Multiple Dynamic Workloads System

To achieve high accuracy when performing deep learning, it is necessary to use a large-scale training model. However, due to the limitations of GPU memory, it is difficult to train large-scale training models within a single GPU. NVIDIA introduced a technology called CUDA Unified Memory with CUDA 6 to overcome the limitations of GPU memory by virtually combining GPU memory and CPU memory. In addition, in CUDA 8, memory advise options are introduced to efficiently utilize CUDA Unified Memory. In this work, we propose a newly optimized scheme based on CUDA Unified Memory to efficiently use GPU memory by applying different memory advise to each data type according to access patterns in deep learning training. We apply CUDA Unified Memory technology to PyTorch to see the performance of large-scale learning models through the expanded GPU memory. We conduct comprehensive experiments on how to efficiently utilize Unified Memory by applying memory advises when performing deep learning. As a result, when the data used for deep learning are divided into three types and a memory advise is applied to the data according to the access pattern, the deep learning execution time is reduced by 9.4% compared to the default Unified Memory.

In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model. In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs (40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed 1.

Deep Learning (DL) has been successfully implemented and deployed to various software service applications. During the training process of DL, a large amount of GPU computing resources is required, but it is difficult for developers to accurately calculate the GPU resources that the model may consume before running, which brings great inconvenience to the development of DL systems. Especially in today’s cloud-based model training. Therefore, it is very important to estimate the GPU memory resources that the DL model may use in a certain computing framework. Existing work has focused on static analysis methods to assess GPU memory consumption, highly coupled with the framework, and lack of research on low-coupled GPU memory consumption of the framework. In this article, we propose TBEM, which is a test-based method for estimating the memory usage of the DL model. First, TBEM generates enough DL models using an orthogonal array testing strategy and a classical neural network design pattern. Then, TBEM generates DL model tested in a real environment to obtain the real-time GPU memory usage values corresponding to the model. After obtaining the data of different models and corresponding GPU usage values, the data is analyzed by regression.

A dynamic memory management system has to take care of the allocation and deallocation of memory blocks in a software system. Real-time embedded systems add some more constraints to the design and the implementation of dynamic memory management systems if compared with the PC world. An increasing number of features are added to embedded mobile devices, however, resources like dynamic memory are limited. In addition, in real-time systems, real-time deadlines must be respected and allocations and deallocations must be done in due time. In this paper we present a case study on evaluating dynamic memory management in embedded real-time systems. We have used a scenario-based approach and used a simulation environment to evaluate the performance of different dynamic memory management systems. Our contribution is to present a practical approach, the tools and the rationales to evaluate dynamic memory management in embedded real-time systems.

Modern cluster management systems, such as Kubernetes, support heterogeneous workloads and resources. However, existing resource schedulers in these systems do not differentiate between heterogeneous G PU resources-which are becoming a norm-and do not support GPU sharing-which is necessary to support emerging collocation of jobs and multi-tenant applications. Thus the systems suffer from low GPU resource utilization, higher queuing delays, and an increase in application makespan, i.e., the duration between the arrival of the first job and the completion of the last job of a workflow. This is especially a problem in supporting crucial deep learning (DL) applications. To this end, in this paper, we profile and analyze DL jobs on heterogeneous GPUs, investigate the interference caused by collocating jobs on GPUs, and use this information to predict the GPU memory demand and job completion times. We propose SCHEDTUNE, a machine-learning-based heterogeneity-aware scheduler that ensures higher GPU memory utilization and reduced out-of-memory (OOM) failures, while supporting improved makespan. Our evaluation shows that SCHEDTUNE GPU memory predictors and scheduler outperform the state-of-the-art predictors by achieving 81% higher GPU memory utilization, 100% detection and avoidance of OOM errors, and 17.5% reduction in makespan compared to the default Kubernetes scheduler.

The increasing computational complexity of DNNs achieved unprecedented successes in various areas such as machine vision and natural language processing (NLP), e.g., the recent advanced Transformer has billions of parameters. However, as large-scale DNNs significantly exceed GPU's physical memory limit, they cannot be trained by conventional methods such as data parallelism. Pipeline parallelism that partitions a large DNN into small subnets and trains them on different GPUs is a plausible solution. Unfortunately, the layer partitioning and memory management in existing pipeline parallel systems are fixed during training, making them easily impeded by out-of-memory errors and the GPU under-utilization. These drawbacks amplify when performing neural architecture search (NAS) such as the evolved Transformer, where different network architectures of Transformer needed to be trained repeatedly. vPipe is the first system that transparently provides dynamic layer partitioning and memory management for pipeline parallelism. vPipe has two unique contributions, including (1) an online algorithm for searching a near-optimal layer partitioning and memory management plan, and (2) a live layer migration protocol for re-balancing the layer distribution across a training pipeline. vPipe improved the training throughput of two notable baselines (Pipedream and GPipe) by 61.4-463.4 percent and 24.8-291.3 percent on various large DNNs and training settings.

DMMX: Dynamic memory management extensions

Introduction:Spatio-temporal MRI methods enable whole-brain multi-parametric mapping at ultra-fast acquisition times through efficient k-space encoding, but can have very long reconstruction times, which limit their integration into clinical practice. Deep learning (DL) is a promising approach to accelerate reconstruction, but can be computationally intensive to train and deploy due to the large dimensionality of spatio-temporal MRI. DL methods also need large training data sets and can produce results that don’t match the acquired data if data consistency is not enforced. The aim of this project is to reduce reconstruction time using DL whilst simultaneously limiting the risk of deep learning induced hallucinations, all with modest hardware requirements.Methods:Deep Learning Initialized Compressed Sensing (Deli-CS) is proposed to reduce the reconstruction time of iterative reconstructions by “kick-starting” the iterative reconstruction with a DL generated starting point. The proposed framework is applied to volumetric multi-axis spiral projection MRF that achieves whole-brain T1 and T2 mapping at 1-mm isotropic resolution for a 2-minute acquisition. First, the traditional reconstruction is optimized from over two hours to less than 40 minutes while using more than 90% less RAM and only 4.7 GB GPU memory, by using a memory-efficient GPU implementation. The Deli-CS framework is then implemented and evaluated against the above reconstruction.Results:Deli-CS achieves comparable reconstruction quality with 50% fewer iterations bringing the full reconstruction time to 20 minutes.Conclusion:Deli-CS reduces the reconstruction time of subspace reconstruction of volumetric spatio-temporal acquisitions by providing a warm start to the iterative reconstruction algorithm.

Deep learning models have a wide spectrum of GPU execution time and memory size. When running distributed training jobs, however, their GPU execution time and memory size have not been taken into account, which leads to the high variance of job completion time (JCT). Moreover, the jobs often run into the GPU out-of-memory (OoM) problem so that the unlucky job has to restart all over. To address the problems, we propose Xonar to profile the deep learning jobs and order them in the queue. The experiments show that Xonar with TensorFlow v1.6 reduces the tail JCT by 44% with the OoM problem eliminated.

Dynamic memory management has been a high cost component in many software systems. A study has shown that memory intensive C programs can consume up to 30% of the program run time in memory allocation and liberation. Especially, in C++ programs, they tend to have object creation and deletion prolifically. The C++ memory allocation rate can be as much as ten times higher than the comparable applications written in C. Despite the importance of dynamic memory management in C++, there exist few software tools to study dynamic memory in C++. This paper introduces a tracing tool, called mtrace++, to study the dynamic memory allocation behavior in C++ programs. mtrace++ is a source code level instrumented tracing tool which produces records of allocation and deallocation information. mtrace++ identifies the origins of allocated memories and life-spans of objects. With limited overheads, mtrace++ can help programmers to solve dynamic memory problems with affordable cost.

An integrated dynamic memory tracing tool for C++

Due to the high level of parallelism, there are unique challenges in developing system software on massively parallel hardware such as GPUs. One such challenge is designing a dynamic memory allocator whose task is to allocate memory chunks to requesting threads at runtime. State-of-the-art GPU memory allocators maintain a global data structure holding metadata to facilitate allocation/deallocation. However, the centralized data structure can easily become a bottleneck in a massively parallel system. In this paper, we present a novel approach for designing dynamic memory allocation without a centralized data structure. The core idea is to let threads follow a random search procedure to locate free pages. Then we further extend to more advanced designs and algorithms that can achieve an order of magnitude improvement over the basic idea. We present mathematical proofs to demonstrate that (1) the basic random search design achieves asymptotically lower latency than the traditional queue-based design and (2) the advanced designs achieve significant improvement over the basic idea. Extensive experiments show consistency to our mathematical models and demonstrate that our solutions can achieve up to two orders of magnitude improvement in latency over the best-known existing solutions.

The problem of Cortical Surface Reconstruction from magnetic resonance imaging has been traditionally addressed using lengthy pipelines of image processing techniques like FreeSurfer, CAT, or CIVET. These frameworks require very long runtimes deemed unfeasible for real-time applications and unpractical for large-scale studies. Recently, supervised deep learning approaches have been introduced to speed up this task cutting down the reconstruction time from hours to seconds. Using the state-of-the-art CorticalFlow model as a blueprint, this paper proposes three modifications to improve its accuracy and interoperability with existing surface analysis tools, while not sacrificing its fast inference time and low GPU memory consumption. First, we employ a more accurate ODE solver to reduce the diffeomorphic mapping approximation error. Second, we devise a routine to produce smoother template meshes avoiding mesh artifacts caused by sharp edges in CorticalFlow’s convex-hull based template. Last, we recast pial surface prediction as the deformation of the predicted white surface leading to a one-to-one mapping between white and pial surface vertices. This mapping is essential to many existing surface analysis tools for cortical morphometry. We name the resulting method CorticalFlow\(^{++}\). Using large-scale datasets, we demonstrate the proposed changes provide more geometric accuracy and surface regularity while keeping the reconstruction time and GPU memory requirements almost unchanged. KeywordsCortical surface reconstructionCorticalFlow3D deep learning

Breast cancer detection from histopathological images has benefited significantly from advances in deep learning; however, many high-performing models rely on deep architectures and substantial computational resources, limiting their applicability in resource-constrained clinical settings. Recent studies have demonstrated that convolutional neural networks (CNNs) can effectively support invasive ductal carcinoma (IDC) classification, but often at the cost of increased training time and memory usage [1, 13].In this study, we propose BCDNet_Updated, an optimized lightweight CNN derived from the reference BCDNet architecture [1] for IDC detection. The proposed model introduces targeted architectural refinements aimed at improving classification performance while preserving computational efficiency. Experiments conducted on the Kaggle IDC dataset demonstrate that BCDNet_Updated achieves improved accuracy, recall, and F1-score compared to the reference BCDNet, while requiring significantly less GPU memory and comparable training time.While the evaluation is limited to a single dataset and does not include formal statistical significance testing or explainability analysis, the results indicate that careful optimization of lightweight CNN architectures can yield meaningful performance improvements without increasing computational cost. Future work will incorporate multi-dataset validation, statistical rigor through k-fold cross-validation, and explainability mechanisms (Grad-CAM, attention visualization) to support clinical adoption. This work contributes to ongoing efforts toward efficient deep learning models for histopathological image analysis and provides a foundation for future studies involving expanded validation and model interpretability.

In 21cm intensity mapping of the large-scale structure (LSS), regions in Fourier space could be compromised by foreground contamination. In interferometric observations, this contamination, known as the foreground wedge, is exacerbated by the chromatic response of antennas, leading to substantial data loss. Meanwhile, the baryonic acoustic oscillation (BAO) reconstruction, which operates in configuration space to “linearize” the BAO signature, offers improved constraints on the sound horizon scale. However, missing modes within these contaminated regions can negatively impact the BAO reconstruction algorithm.To address this challenge, we employ the deep learning model U-Net to recover the lost modes before applying the BAO reconstruction algorithm. Despite hardware limitations, such as GPU memory, our results demonstrate that the AI-restored 21cm temperature map achieves a high correlation with the original signal, with a correlation ratio of approximately 0.9 at k ∼ 1h/Mpc. Furthermore, subsequent BAO reconstruction indicates that the AI restoration has minimal impact on the performance of the `linearized' BAO signal, proving the effectiveness of the machine learning approach to mitigate the impact of foreground contamination. Interestingly, we demonstrate that the AI model trained on coarser fields can be effectively applied to finer fields, achieving even higher correlation ratio. This success is likely attributable to the scale-invariance properties of non-linear mode coupling in large-scale structure and the hierarchical structure of the U-Net architecture.