• Home
  • Search
  • Severely imbalanced Big Data challenges: investigating data sampling approaches
  • Cite Icon138
  • https://doi.org/10.1186/s40537-019-0274-4Copy DOI Icon

Severely imbalanced Big Data challenges: investigating data sampling approaches

Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Severe class imbalance between majority and minority classes in Big Data can bias the predictive performance of Machine Learning algorithms toward the majority (negative) class. Where the minority (positive) class holds greater value than the majority (negative) class and the occurrence of false negatives incurs a greater penalty than false positives, the bias may lead to adverse consequences. Our paper incorporates two case studies, each utilizing three learners, six sampling approaches, two performance metrics, and five sampled distribution ratios, to uniquely investigate the effect of severe class imbalance on Big Data analytics. The learners (Gradient-Boosted Trees, Logistic Regression, Random Forest) were implemented within the Apache Spark framework. The first case study is based on a Medicare fraud detection dataset. The second case study, unlike the first, includes training data from one source (SlowlorisBig Dataset) and test data from a separate source (POST dataset). Results from the Medicare case study are not conclusive regarding the best sampling approach using Area Under the Receiver Operating Characteristic Curve and Geometric Mean performance metrics. However, it should be noted that the Random Undersampling approach performs adequately in the first case study. For the SlowlorisBig case study, Random Undersampling convincingly outperforms the other five sampling approaches (Random Oversampling, Synthetic Minority Over-sampling TEchnique, SMOTE-borderline1 , SMOTE-borderline2 , ADAptive SYNthetic) when measuring performance with Area Under the Receiver Operating Characteristic Curve and Geometric Mean metrics. Based on its classification performance in both case studies, Random Undersampling is the best choice as it results in models with a significantly smaller number of samples, thus reducing computational burden and training time.

Loading PDF

Similar Papers
  • Conference Article
  • Citations15

A Comparison of Performance Metrics with Severely Imbalanced Network Security Big Data

  • Jul 01, 2019
  • Tawfiq Hasanin +2
  • PDF
  • Research Article
  • Citations41

Examining characteristics of predictive models with imbalanced big data

  • Jul 31, 2019
  • Journal of Big Data
  • Tawfiq Hasanin +3
  • PDF
  • Research Article
  • Citations21

Investigating class rarity in big data

  • Mar 16, 2020
  • Journal of Big Data
  • Tawfiq Hasanin +3
  • Book Chapter
  • Citations9

Addressing the Big Data Multi-class Imbalance Problem with Oversampling and Deep Learning Neural Networks

  • Jan 01, 2019
  • V M González-Barcenas +4
  • Conference Article
  • Citations5

Towards Optimizing Performance of Machine Learning Algorithms on Unbalanced Dataset

  • Oct 28, 2023
  • Asitha Thumpati +1
  • Research Article

Balanced DATA by DBSCAN and Weighted Arithmetic Mean to Improve Performance of Machine Learning Algorithms

  • Dec 31, 2021
  • Bitlis Eren Üniversitesi Fen Bilimleri Dergisi
  • Serkan Güldal
  • Research Article

Application of Synthetic Minority Over-Sampling Technique to Mitigate Class Imbalance in Stroke Prediction

  • Nov 30, 2023
  • Advances in Multidisciplinary & Scientific Research Journal Publication
  • Fatimah Adamu-Fika +6
  • Research Article

Application of Synthetic Minority Over-Sampling Technique to Mitigate Class Imbalance in Stroke Prediction

  • Nov 30, 2023
  • Advances in Multidisciplinary & Scientific Research Journal Publication
  • Fatimah Adamu-Fika +6
  • PDF
  • Research Article
  • Citations1107

SMOTE for high-dimensional class-imbalanced data

  • Mar 22, 2013
  • BMC Bioinformatics
  • Rok Blagus +1
  • Dissertation
  • Citations1

Genetic Programming for Classification with Unbalanced Data

  • Jan 01, 2012
  • Urvesh Bhowan
  • Dissertation
  • Citations1

Genetic Programming for Classification with Unbalanced Data

  • Jan 01, 2012
  • Urvesh Bhowan
  • PDF
  • Research Article
  • Citations31941

SMOTE: Synthetic Minority Over-sampling Technique

  • Jun 01, 2002
  • Journal of Artificial Intelligence Research
  • N V Chawla +3
  • Research Article
  • Citations4

Feature-based ensemble modeling for addressing diabetes data imbalance using the SMOTE, RUS, and random forest methods: a prediction study.

  • Apr 30, 2025
  • Ewha medical journal
  • Younseo Jang
  • Conference Article
  • Citations31

Investigating Random Undersampling and Feature Selection on Bioinformatics Big Data

  • Apr 01, 2019
  • Tawfiq Hasanin +3
  • Research Article

M-Tune: imbalanced data handling in machine learning by tuning the decision threshold.

  • May 29, 2026
  • Molecular diversity
  • Tapos Dutta +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.