Memory-efficient Combination of Tensor Decomposition and Recomputation for Training CNNs

Abstract

Translate Article

Tensor decomposition and recomputation are two common strategies used to to accelerate training and save memory during the training phase of deep neural network, respectively, allowing researchers to consider larger models, batch sizes, or higher-resolution data. Tensor decomposition reduces parameters and computation by decomposing weight tensors, while recomputation trades computation time for memory. As These two strategies are independent and have their own advantages and disadvantages, it is applealing to consider the simultaneous combination of two strategies to save more memory and complement each other. We analyze existing tensor decomposition algorithms and recomputation techniques, and we propose a memory-optimized algorithm that effectively combines them, and finally evaluate the performance of the proposed algorithm on commonly used deep neural networks and computing platforms. Our experiments show that tensor decomposition can partially or even completely eliminate the extra time overhead caused by recomputation, and that together they can reduce the memory consumption of the training phase by about 41 %, while the loss of accuracy is less than 1.5%.