• Home
  • Search
  • A parallel framework for data input pipelines and online data augmentation in deep learning
  • https://doi.org/10.1007/s11227-026-08543-0Copy DOI Icon

A parallel framework for data input pipelines and online data augmentation in deep learning

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Abstract Efficient data ingestion and online data augmentation remain challenges in deep learning workflows, particularly when dealing with datasets containing non-standard formats or massive multidimensional arrays that natively optimised functions cannot fully manage. This work presents a parallel framework that integrates and global shared memory through a ring buffer architecture, enabling high-throughput data loading and flexible on-the-fly augmentation. The framework decouples data production from consumption, allowing multiple CPU workers to load and preprocess batches in parallel while completely bypassing the Python GIL and memory bottlenecks. Crucially, the framework supports both CPU-side and GPU-side augmentation strategies, adapting to whether complex conditional transformations or framework-native operations are required. The proposed approach was validated on two representative tasks: (i) sign language recognition from human pose CSV sequences, and (ii) hyperspectral image classification using massive arrays. Relative to standard sequential baselines, the proposed framework achieved up to $$27\times $$ 27 × acceleration in isolated data ingestion and up to $$28\times $$ 28 × in end-to-end training. Importantly, even against natively optimised parallel TensorFlow and PyTorch pipelines, it still delivered up to $$8\times $$ 8 × faster data loading and up to $$7\times $$ 7 × faster full training in memory-intensive scenarios. Overall, the proposed framework provides a scalable, multi-GPU compatible solution for deep learning pipelines, showing robust performance across both I/O-bound and memory-constrained scenarios in TensorFlow and PyTorch while alleviating memory fragmentation and allocation constraints.

Similar Papers
  • Research Article
  • Citations4

The Effect of Data Augmentation in Deep Learning with Drone Object Detection

  • Jul 31, 2023
  • IJCCS (Indonesian Journal of Computing and Cybernetics Systems)
  • Ariel Yonatan Alin +2
  • Research Article
  • Citations1

Physically Consistent Image Augmentation for Deep Learning in Mueller Matrix Polarimetry.

  • Jan 01, 2025
  • IEEE transactions on image processing : a publication of the IEEE Signal Processing Society
  • Christopher Hahne +8
  • Book Chapter
  • Citations1

Efficient Deep Belief Network Based Hyperspectral Image Classification

  • Jan 01, 2017
  • Atif Mughees +1
  • Research Article
  • Citations160

Deep Learning Ensemble for Hyperspectral Image Classification

  • Jun 01, 2019
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
  • Yushi Chen +5
  • Research Article
  • Citations1852

Deep Learning for Hyperspectral Image Classification: An Overview

  • Sep 01, 2019
  • IEEE Transactions on Geoscience and Remote Sensing
  • Shutao Li +5
  • Research Article
  • Citations22

Category-Level Band Learning-Based Feature Extraction for Hyperspectral Image Classification

  • Jan 01, 2024
  • IEEE Transactions on Geoscience and Remote Sensing
  • Ying Fu +5
  • Research Article
  • Citations359

Hyperspectral Image Classification With Convolutional Neural Network and Active Learning

  • Jul 01, 2020
  • IEEE Transactions on Geoscience and Remote Sensing
  • Xiangyong Cao +3
  • PDF
  • Research Article
  • Citations147

Hyperspectral Image Classification with Capsule Network Using Limited Training Samples.

  • Sep 18, 2018
  • Sensors
  • Fei Deng +5
  • Conference Article
  • Citations32

Efficient Deep Auto-Encoder Learning for the Classification of Hyperspectral Images

  • Sep 01, 2016
  • Atif Mughees +1
  • Research Article
  • Citations5

A Dual-Branch Deep Stochastic Adaptive Fourier Decomposition Network for Hyperspectral Image Classification

  • Jan 01, 2023
  • IEEE Transactions on Geoscience and Remote Sensing
  • Chunbo Cheng +3
  • Conference Article

Hyperspectral Image Classification using Archimedes Optimized 3D CNN layers

  • Nov 26, 2025
  • Athul Ghosh P +1
  • Research Article
  • Citations1

Sign language recognition method based on deep learning

  • Jan 01, 2025
  • ITM Web of Conferences
  • Mu He
  • Research Article
  • Citations39

Internal Wave Signature Extraction From SAR and Optical Satellite Imagery Based on Deep Learning

  • Jan 01, 2023
  • IEEE Transactions on Geoscience and Remote Sensing
  • Shuangshang Zhang +2
  • Research Article
  • Citations1

A STUDY OF WORLDWIDE PATTERNS IN ALPHABET SIGN LANGUAGE RECOGNITION USING CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS

  • Feb 12, 2025
  • Jurnal Teknik Informatika (Jutif)
  • Aris Rakhmadi +2
  • PDF
  • Research Article
  • Citations11

A Hybrid 3D–2D Feature Hierarchy CNN with Focal Loss for Hyperspectral Image Classification

  • Sep 09, 2023
  • Remote Sensing
  • Xiaoyan Wen +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.