Performance Evaluation of Classification Methods Utilizing Resampling Techniques for Water Quality Prediction on Imbalanced Data

<abstract><p>In the realm of machine learning, where data-driven insights guide decision-making, addressing the challenges posed by class imbalance in datasets has emerged as a crucial concern. The effectiveness of classification algorithms hinges not only on their intrinsic capabilities but also on their adaptability to uneven class distributions, a common issue encountered across diverse domains. This study delves into the intricate interplay between varying class imbalance levels and the performance of ten distinct classification models, unravelling the critical impact of this imbalance on the landscape of predictive analytics. Results showed that random forest (RF) and decision tree (DT) models outperformed others, exhibiting robustness to class imbalance. Logistic regression (LR), stochastic gradient descent classifier (SGDC) and naïve Bayes (NB) models struggled with imbalanced datasets. Adaptive boosting (ADA), gradient boosting (GB), extreme gradient boosting (XGB), light gradient boosting machine (LGBM), and k-nearest neighbour (kNN) models improved with balanced data. Adaptive synthetic sampling (ADASYN) yielded more reliable predictions than the under-sampling (UNDER) technique. This study provides insights for practitioners and researchers dealing with imbalanced datasets, guiding model selection and data balancing techniques. RF and DT models demonstrate superior performance, while LR, SGDC and NB models have limitations. By leveraging the strengths of RF and DT models and addressing class imbalance, classification performance in imbalanced datasets can be enhanced. This study enriches credit risk modelling literature by revealing how class imbalance impacts default probability estimation. The research deepens our understanding of class imbalance's critical role in predictive analytics. Serving as a roadmap for practitioners and researchers dealing with imbalanced data, the findings guide model selection and data balancing strategies, enhancing classification performance despite class imbalance.</p></abstract>

Anemia is one of the common adverse reactions after hip fracture surgery. The traditional method to solve anemia is allogeneic transfusion. However, the transfusion may lead to some complications such as septicemia and fever. So far, few studies have reported roles of machine learning in predicting whether blood transfusion is needed or not after hip fracture surgery. Therefore, the purpose of this study is to develop machine learning models to predict the likelihood of postoperative blood transfusion in patients undergoing hip fracture surgery. This study enrolled 1355 patients who underwent hip fracture surgery at the Affiliated Hospital of Qingdao University from January 2016 to December 2021. Among all patients, 210 cases received postoperative blood transfusion. All patients were randomly divided into a training group and a testing group at a ratio of 7:3. In the training group, univariate and multivariate logistic regression analyses were used to determine independent risk factors for the postoperative transfusion. Then, based on these independent risk factors, tenfold cross-validation method was utilized to develop five machine learning models, including logistic, multilayer perceptron (MLP), extreme gradient boosting (XGBoost), random forest (RF), and support vector machine (SVM). The receiver operating characteristic (ROC) curve, area under ROC curve (AUC), and Matthews correlation coefficient (MCC) were generated to evaluate the performance of the models. Calibration plot and decision curve analysis (DCA) were used to test the performance, stability, and clinical applicability of the models. The models were validated using the testing group; and the ROC curve, MCC, calibration plot, and DCA curves were also generated to validate the performance, stability, and clinical applicability of the models. To further verify the robustness of the model, we randomly grabbed 70% of the samples in the testing set, performed 1000 iterations, and calculated the AUC and confidence interval of the five models. Finally, we used SHapley Additive exPlanations (SHAP) to explain these models. Multivariate logistic regression analysis showed that there were 8 independent risk factors, including age, blood transfusion history, albumin (ALB), globulin (GLO), total bilirubin (TBIL), indirect bilirubin (IBIL), hemoglobin (HB), and blood loss > 200ml. We finally selected five independent risk factors including HB, GLO, age, IBIL, and blood loss > 200ml. Based on these five independent risk factors, we generated six characteristic variables, namely HB, HB × HB, HB × blood loss, GLO × HB, age, age × IBIL, and established five machine learning models using a tenfold cross-validation method. In the training group, the AUC values of logistic, RF, MLP, SVM, and XGB were 0.9320, 0.8911, 0.9327, 0.9225, and 0.8825, respectively, and the average AUC was 0.9122 ± 0.0212. The MCC values were 0.65, 0.77, 0.65, 0.66, and 0.68, respectively, and the calibration plot and DCA performed well. In the testing group the AUC values of logistic, RF, MLP, SVM, and XGB were 0.8483, 0.7978, 0.8576, 0.8598, and 0.8216, respectively. The average AUC was 0.8370 ± 0.0238, and the MCC values were 0.41, 0.35, 0.40, 0.41, and 0.41, respectively. The calibration plot and DCA in the testing group also showed good performance. The AUC values and confidence intervals of the 1000-iteration model were: logistic (AUC, min confidence interval [CI]-max confidence interval [CI] 0.848, 0.804-0.903), RF (AUC, minCI-maxCI 0.797, 0.734-0.857), MLP (AUC, minCI-maxCI 0.858, 0.812-0.902), SVM (AUC, minCI-maxCI 0.859, 0.819-0.910), and XGB (AUC, minCI-maxCI 0.821, 0.764-0.894). The model performed well. Finally, according to SHAP, among all five models, HB played the most important role in model prediction and interpretation. The five models we developed all performed well in predicting the likelihood of blood transfusion after hip fracture surgery. Therefore, we believed that the prediction model based on machine learning had great application prospects in clinical practice, which could help clinicians better predict the risk of blood transfusion after hip fracture surgery.

Objective: Cancer-associated thrombosis (CAT) presents a complex challenge in oncology, exacerbated by data imbalances in related datasets that often lead to suboptimal outcomes in machine learning (ML) classification. Many ML algorithms were originally designed for balanced datasets, prompting this study to evaluate the interaction between logistic regression (LR) and eXtreme Gradient Boost (XGBoost) and data resampling techniques for improving prediction on imbalances in Malaysian data on CAT (MDCAT). Methods: Random oversampling (ROS), random undersampling (RUS), and a combined oversampling and undersampling approach (BOTH) were applied to MDCAT dataset. Classification tasks were performed using LR and XGBoost in R version 4.3.1. Classifier performance was assessed using accuracy, sensitivity, specificity, and the area under the ROC curve (AUROC) to evaluate the impact of different resampling techniques. Results: Applying LR and XGBoost to the imbalanced data revealed high specificity but low sensitivity in testing samples. A substantial decline in XGBoost performance was observed, with the AUC decreasing from 0.794 in training to 0.381. Metastasis, surgery, and Indian ethnicity showed statistically significant associated with the CAT event across all resampling techniques. Among XGBoost models, oversampling (XO) exhibited excellent training performance (Accuracy 0.99; AUC 0.98) but showed a large performance drop on the test set (Accuracy 0.82; AUC 0.72). Among LR models, logistic undersampling yielded the highest training accuracy (0.83) and AUC of 0.82. Tuning amplified the differences between resampling strategies and highlighted clear classifier–resampling interactions. XGBoost benefited most, particularly when trained on mixed and oversampled datasets, while LR remained comparatively stable. Conclusion: This study demonstrated that the effectiveness of prediction models in imbalanced MDCAT dataset is strongly influenced by the interaction between classifier characteristics and resampling strategies. A tuned XGBoost model with mixed resampling outweighed the benefits of LR’s simplicity and stability, making it our recommended approach given the primary importance of AUC.

<title>Abstract</title> <bold>Background:</bold> Functional attrition (non-response) of Patient-Reported Outcomes (PROs) and Quality of Life (QoL) deterioration are common and non-random in oncology trials. While functional attrition is informative, it is often treated as simple missing data, which biases the interpretation of QoL trajectories and prediction models. <bold>Objectives and Methods:</bold> This study aimed (1) To examine prognostic factors of transitions between active participation, functional attrition, and death using multinomial models and (2) To evaluate how different methods of dealing with missing data affect the reliability of QoL deterioration prediction models. A discrete-state modeling framework was used to analyze transitions between active participation, functional attrition, and death in advanced cancer patients. QoL decline was then predicted using LASSO, Random Forest, and XGBoost algorithms, while Inverse Probability of Completion Weighting (IPCW) under a conditional Missing At Random (MAR) assumption was systematically compared against Multiple Imputation by Chained Equations (MICE). At the end, we examined robustness to deviations from MAR through delta-adjusted Missing Not At Random (MNAR) sensitivity analyses. <bold>Results:</bold> Functional attrition accumulated more rapidly than death within the first 12 weeks, confirming its high prevalence and distinct determinants. Lower educational attainment was associated with functional attrition, whereas poor performance status and depressive symptoms primarily predicted mortality. Among predictive models, LASSO combined with MICE achieved the best performance (median AUC = 0.671 [IQR 0.136]), reflecting the modest predictability of dynamic PRO outcomes when relying on baseline covariates. Methodological comparison demonstrated superior discrimination and substantially improved calibration for MICE relative to IPCW, with IPCW models showing marked miscalibration. These findings remained stable across MNAR sensitivity scenarios. <bold>Conclusion:</bold> By modeling the functional attrition as a distinct state, we revealed its specific socio-demographic determinants and clarified how covariate profiles shape patient trajectories <bold>.</bold> For QoL prediction under conditional MAR, MICE yields more stable and better-calibrated models than IPCW. Robustness across MNAR scenarios supports the methodological validity of these conclusions and highlights the importance of principled handling of informative non-response in longitudinal PRO research.

Employee churn is a significant challenge for organizations, leading to substantial costs associated with recruiting, onboarding, and training new employees. High turnover rates can negatively impact overall productivity, employee morale, and organizational stability. Therefore, accurately predicting employee churn is crucial for companies to implement targeted retention strategies, minimize turnover, and reduce associated expenses. In this study, we leveraged machine learning techniques to predict employee churn using the "HR Analytics" dataset from Kaggle. One of the key challenges in churn prediction is class imbalance, where the number of employees who leave is significantly lower than those who stay. To address this, we applied two data-balancing techniques: Synthetic Minority Over-sampling Technique (SMOTE) and Random Over-Sampling (ROS). We then trained and evaluated four machine learning models Logistic Regression, Random Forest, Decision Tree, and Extreme Gradient Boosting (XGBoost) on the balanced datasets. The F1 scores for the SMOTE-balanced data were: Logistic Regression (0.5990), Random Forest (0.9753), Decision Tree (0.9319), and XGBoost (0.9634). Meanwhile, the ROS-balanced data produced F1 scores of: Logistic Regression (0.5978), Random Forest (0.9760), Decision Tree (0.9475), and XGBoost (0.9703). The results demonstrated that ROS yielded superior performance, particularly for the Random Forest and XGBoost models, leading us to select ROS for further hyperparameter tuning. Using RandomizedSearchCV for optimization, the Random Forest model achieved the highest F1 score of 0.9779. Finally, we deployed the optimized Random Forest model via a Flask API, enabling HR professionals to access a user-friendly web interface for realtime churn prediction. This research highlights the effectiveness of machine learning in HR analytics and underscores the practical benefits of predictive modeling in workforce management, helping organizations proactively address employee retetion challenges.

The rapid expansion of network connections has significantly increased network traffic activity, introducing new cybersecurity challenges and heightened vulnerability to cyber attacks. To address these challenges, researchers have leveraged intelligent techniques such as machine learning (ML) to enhance attack detection accuracy in network traffic. However, ML models often face a data imbalance issue in their training sets. This imbalance, typically due to the uneven distribution of attack classes, hampers the classification performance of ML models in network intrusion detection. To mitigate class imbalance, various resampling techniques can be employed. This study evaluates several resampling techniques, including Random Oversampling, SMOTE, ADASYN, Random Undersampling, Tomek Links, and SMOTE-Tomek. Using the UNSW-NB15 dataset, we trained and tested ML models, including Decision Tree, Random Forest, Gradient Boosting, XGBoost, and 1D-CNN algorithms. Our analysis demonstrates that resampling techniques significantly impact the performance of machine learning models. The Tomek Links technique applied to the 1D-CNN model achieved the highest performance, with an accuracy of 75.27%, a precision of 87.58%, and an F1-score of 76.22%. Notably, the best recall score of 67.57% was obtained from the 1D-CNN model without resampling. These findings provide valuable insights for researchers and engineers, aiding in selecting appropriate resampling techniques for developing robust detection models for network traffic attacks.

Breast cancer remains a leading cause of mortality among women worldwide, with early and accurate diagnosis being critical for effective treatment. Gene expressionprofiling has emerged as a powerful tool for understanding the molecular mechanisms of cancer and for developing predictive models. However, the high dimensionality and class imbalance inherent in gene expression data pose significant challenges for developing robust predictive models.This study aims to develop and evaluate a predictive model for classifying breast cancer subtypes using high-dimensional gene expression data, addressing the challenges of class imbalance. The objective is to improve the accuracy and reliability of breast cancer subtype prediction to facilitate better diagnostic and treatment strategies.A comprehensive dataset of breast cancer gene expression profiles was utilized, comprising numerous gene expression levels across multiple samples. To address class imbalance, resampling techniques such as Synthetic Minority Over-sampling Technique (SMOTE) and Random Under-sampling, were employed. The machine learning algorithms employed included Support Vector Machines (SVM), Random Forests, and Neural Networks. The algorithms were trained and evaluated using cross-validation to identify the most effective model. The performance of these models was assessed based on metrics such as accuracy, precision, recall, and F1-score. The results indicate that the use of SMOTE in combination with SVM provided the most balanced and accurate predictions, with an F1-score significantly higher than models without resampling. The Random Forest algorithm also showed promising results, particularly in handling the high-dimensionality aspect of the data.The study demonstrates that addressing class imbalance through advanced resampling techniques can significantly enhance the predictive accuracy of models trained on high-dimensional gene expression data. The findings underscore the potential of machine learning models, particularly SVMs and Random Forests, in improving breast cancer subtype classification.

Biomedical data are widely accepted in developing prediction models for identifying a specific tumour, drug discovery and human cancers detection. However, previous studies usually focused on different classifiers, and overlook the class imbalance problem in real-world biomedical datasets. This paper mainly focuses on reviewing and evaluating some popular and recently developed resampling and feature selection (FS) methods for class imbalance learning with data distribution being considered. Experimental results show that: 1) resampling and FS techniques exhibit better performance using support vector machine (SVM) classifier; 2) techniques such as random undersampling and FS perform better than other data pre-processing methods with T location-scale distribution when using SVM and K-nearest neighbours (KNN) classifiers. Random oversampling outperforms other methods on negative binomial distribution using Random Forest with lower level of imbalance ratio; 3) FS outperforms other data pre-processing methods in most cases, thus, FS with SVM classifier is the best choice for imbalanced biomedical data learning.

BackgroundThe growing demand for healthcare services challenges patient flow management in health systems. Alternative Level of Care (ALC) patients who no longer need acute care yet face discharge barriers contribute to prolonged stays and hospital overcrowding. Predicting these patients at admission allows for better resource planning, reducing bottlenecks, and improving flow. This study addresses three objectives: identifying likely ALC patients, key predictive features, and preparing guidelines for early ALC identification at admission.MethodsData from Nova Scotia Health (2015-2022) covering patient demographics, diagnoses, and clinical information was extracted. Data preparation involved managing outliers, feature engineering, handling missing values, transforming categorical variables, and standardizing. Data imbalance was addressed using class weights, random oversampling, and the Synthetic Minority Over-Sampling Technique (SMOTE). Three ML classifiers, Random Forest (RF), Artificial Neural Network (ANN), and eXtreme Gradient Boosting (XGB), were tested to classify patients as ALC or not. Also, to ensure accurate ALC prediction at admission, only features available at that time were used in a separate model iteration.ResultsModel performance was assessed using recall, F1-Score, and AUC metrics. The XGB model with SMOTE achieved the highest performance, with a recall of 0.95 and an AUC of 0.97, excelling in identifying ALC patients. The next best models were XGB with random oversampling and ANN with class weights. When limited to admission-only features, the XGB with SMOTE still performed well, achieving a recall of 0.91 and an AUC of 0.94, demonstrating its effectiveness in early ALC prediction. Additionally, the analysis identified diagnosis 1, patient age, and entry code as the top three predictors of ALC status.ConclusionsThe results demonstrate the potential of ML models to predict ALC status at admission. The findings support real-time decision-making to improve patient flow and reduce hospital overcrowding. The ALC guideline groups patients first by diagnosis, then by age, and finally by entry code, categorizing prediction outcomes into three probability ranges: below 30%, 30-70%, and above 70%. This framework assesses whether ALC status can be accurately predicted at admission or during the patient’s stay before discharge.

While analyzing health data is important for improving health outcomes, class imbalance in datasets poses major challenges to machine learning classification models. This work, therefore, considers the class imbalance problem in stroke prediction using models such as K‐nearest neighbors, support vector machine, logistic regression, random forest, and decision tree. This work balances the stroke dataset, thereby enhancing model performance, through various oversampling strategies: random oversampling (RO), ADASYN, SMOTE, and SMOTE–Tomek. Compared to the results of the imbalanced dataset, all applied oversampling techniques enhanced the correct classification of stroke events by the ML model. Among these, RO–SVM with RBF kernel was the best in terms of sensitivity, specificity, G‐mean, F1‐score, and accuracy values, offering the highest results with respective values of 89.87%, 94.91%, 92.36%, 89.64%, and 89.87%. After applying oversampling techniques, all the machine learning classifications were good enough to classify stroke status, especially for the minority class. This study has highlighted the importance of class imbalance issues in health datasets. Precise detection of instances of minority classes can be enhanced considerably by employing classification models with the implementation of hybrid strategies to effectively solve class imbalance issues, which, in turn, will help improve healthcare outcomes. Further research in integrating more advanced deep learning techniques into other health datasets with imbalances is encouraged to further validate or refine class imbalance approaches, as effective handling of imbalanced classes can substantially promote predictive model performance in the analysis of healthcare.

oan default poses a significant threat to the sustainability of financial institutions, necessitating the development of intelligent, data-driven systems for early risk detection. This research presents a robust and interpretable machine learning framework for predicting loan default risk using a real-world financial dataset comprising 255,347 anonymized loan records with a pronounced class imbalance (11.6% default rate). To address the skewed class distribution, the Synthetic Minority Oversampling Technique (SMOTE) was applied to the training set, enhancing the model's sensitivity to defaulters. Four supervised learning algorithms Logistic Regression, Support Vector Machine (SVM), Random Forest, and Extreme Gradient Boosting (XGBoost) were implemented and rigorously evaluated using stratified 5-fold cross-validation. Performance metrics included Accuracy, Precision, Recall, F1-Score, and Area Under the Receiver Operating Characteristic Curve (AUC-ROC), with particular emphasis on metrics sensitive to class imbalance. Among the models tested, the Random Forest classifier achieved the best overall performance, attaining a test accuracy of 96.26%, F1-Score of 0.8014, and AUC-ROC of 0.9215, thereby offering a balanced and reliable prediction of default risk. To ensure model transparency and support regulatory compliance, HasMortgage, EmploymentType, and LoanPurpose as key drivers of default risk, aligning with domain knowledge and enhancing stakeholder trust. The study concludes that combining ensemble machine learning models with class imbalance handling and explainable AI techniques offers a practical and effective solution for credit risk assessment. Recommendations were made for financial institutions, data scientists, and policymakers to adopt interpretable, fair, and performance-optimized predictive systems. This research contributes to the growing body of literature on responsible AI in finance and lays a foundation for future advancements in ethical and data-driven credit decision-making.

BackgroundThe multiple imputation by chained equations (MICE) is a widely used approach for handling missing data. However, its robustness, especially for high missing proportions in health indicators, is under-researched. The study aimed to provide a preliminary guideline for the choice of the extent of missing proportion to impute longitudinal health-related data using the MICE method.MethodsThe study obtained complete data on five mortality-related health indicators of 100 countries (2015–2019) from the Global Health Observatory. Nine incomplete datasets with missing rates from 10 to 90% were generated and imputed using MICE. The robustness of MICE was assessed through three approaches: comparison of means using the Repeated Measures- Analysis of variance, estimation of evaluation metrics (Root mean square error, mean absolute deviation, Bias, and proportionate variance), and visual inspection of box plots of imputed and non-imputed data.ResultsThe Repeated Measures- Analysis of variance revealed significant differences between complete and imputed data, primarily in imputed data with over 50% missing proportions. Evaluation metrics exhibited ‘high performance’ for the dataset with a 50% missing proportion for various health indicators However, with missing proportions exceeding 70%, the majority of indicators demonstrated a ‘low’ performance level in terms of most evaluation metrics. The visual inspection of the box plot revealed severe variance shrinkage in imputed datasets with missing proportions beyond 70%, corroborating the findings from the evaluation metrics.ConclusionIt demonstrates high robustness up to 50% missing values, with marginal deviations from complete datasets. Caution is warranted for missing proportions between 50 and 70%, as moderate alterations are observed. Proportions beyond 70% lead to significant variance shrinkage and compromised data reliability, emphasizing the importance of acknowledging imputation limitations for practical decision-making.

Imbalanced data is a challenge for classification models. It reduces the overall performance of traditional learning algorithms. Besides, the minority class of imbalanced datasets is misclassified with a high ratio even though this is a crucial object of the classification process. In this paper, a new model called the Lasso-Logistic ensemble is proposed to deal with imbalanced data by utilizing two popular techniques, random over-sampling and random under-sampling. The model was applied to two real imbalanced credit data sets. The results show that the Lasso-Logistic ensemble model offers better performance than the single traditional methods, such as random over-sampling, random under-sampling, Synthetic Minority Oversampling Technique (SMOTE), and cost-sensitive learning.This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium provided the original work is properly cited.

Identifying the actual Single Nucleotide Polymorphisms (SNPs) by sourcing Next Generation Sequencing (NGS) data emerges an imbalanced problem due to the inherent high error rate of NGS technology. The imbalance problem has been found to have a negative impact on machine learning algorithms because it produces biased models and poor performance, particularly in detecting actual SNP that belong to the underrepresented class in question. This study evaluates the effectiveness of several resampling techniques, including Borderline-SMOTE, Random Undersampling, and Tomek-Link, in enhancing the performance of machine learning algorithms, specifically Random Forest (RF) and Artificial Neural Networks (ANN). Furthermore, we compare these techniques to determine the most effective approach. Our results indicate that Borderline-SMOTE improves the F-Measure of RF from 69.72 to 91.52 (a 31.2% increase) and ANN from 79.75 to 91.32 (a 14.5% increase) and outperforms other resampling methods. These findings highlight the crucial role of resampling techniques and the careful selection of algorithms in improving classification accuracy for imbalanced datasets.

Multiple imputation (MI) has been widely used for handling missing data in biomedical research. In the presence of high-dimensional data, regularized regression has been used as a natural strategy for building imputation models, but limited research has been conducted for handling general missing data patterns where multiple variables have missing values. Using the idea of multiple imputation by chained equations (MICE), we investigate two approaches of using regularized regression to impute missing values of high-dimensional data that can handle general missing data patterns. We compare our MICE methods with several existing imputation methods in simulation studies. Our simulation results demonstrate the superiority of the proposed MICE approach based on an indirect use of regularized regression in terms of bias. We further illustrate the proposed methods using two data examples.