Enhancing classification performance in imbalanced datasets: A comparative analysis of machine learning models

The rise of IoT technologies has increased the need for robust threat detection to address growing cyber-physical risks. Traditional machine learning (ML) models often struggle with imbalanced datasets, particularly in detecting rare threats. This study tackles this challenge using the "IoT_Modbus" dataset, a benchmark for multiclass cybersecurity classification, and explores advanced resampling techniques to enhance model performance. By combining methods like Synthetic Minority Oversampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN) with ensemble classifiers, Random Forest (RF), Gradient Boosting, XGBoost, LightGBM, and CatBoost, significant improvements in accuracy were achieved. RF outperformed others, reaching 99.06% accuracy with SMOTE and 99.00% with ADASYN. Among boosting-based models, XGBoost led with 86.21% (SMOTE) and 86.63% (ADASYN), while CatBoost followed closely. These results highlight the effectiveness of resampling in addressing class imbalance and improving predictive performance. Visual tools like heatmaps and accuracy charts further clarify these trends. This research provides cybersecurity professionals with a practical framework to enhance IoT threat detection and underscores the importance of resampling strategies for better classification outcomes. By guiding model selection for imbalanced datasets, it offers valuable insights for securing IoT systems in an interconnected world.

Background Suicide is a leading cause of death worldwide, yet its prediction remains a challenge. This difficulty arises not only because suicidal behavior is a rare event in the general population, leading to significant class imbalance in datasets, but also due to its complex, multi-causal nature involving a non-linear interplay of sociodemographic and clinical factors. Furthermore, while the majority of suicides occur in middle-income countries, there is a lack of predictive models tailored to these specific social contexts. This study evaluates machine learning strategies in an enriched clinical setting: a pediatric psychiatric emergency center in Brazil. Methods We analyzed a comprehensive database of 2,365 youth seeking emergency care. We benchmarked three machine learning algorithms, namely Logistic Regression, Random Forest, and XGBoost, to predict three outcomes: self-harm, suicidal ideation, and suicide attempts. To address class imbalance, we applied oversampling techniques to the training data. We also used SHapley Additive exPlanations (SHAP) values to quantify each feature's contribution to the predictions. Findings and interpretation In this setting, suicide-related behaviors represented 28.7% of the clinical demand. The Random Forest model combined with oversampling was the most effective strategy, achieving sensitivities of 78.04% for suicidal ideation, 71.18% for suicide attempts, and 69.37% for self-harm. Specificity remained consistently above 75%. SHAP value analysis revealed that social determinants were critical predictors, highlighting that social conditions in middle-income populations introduce unique variables that significantly influence suicidal risk. While accuracy for suicide attempts remained a challenge, SHAP provided clear clinical insights into the drivers of risk. Conclusions Machine learning, specifically Random Forest models together with oversampling and SHAP, demonstrates strong potential for identifying suicidal risk in pediatric emergency settings. By integrating clinical data with social determinants, these models provide a transparent and scalable strategy for early identification in regions with limited specialized psychiatric resources.

As the Chinese population continues to age, the prevalence of gastric ulcers, a common nutrition and diet-related disorder, is rising among the elderly. Gastric ulcers pose a significant public health challenge in China, yet there is limited research to predict gastric ulcers accurately. Our study aims to employ machine learning algorithms to predict the occurrence of gastric ulcers and develop an online tool to assess the risk of gastric ulcers for elderly individuals, both currently and in the future, while identifying important predictors. We used baseline data from the Chinese Longitudinal Healthy Longevity Survey in 2011 and 2014, with a follow-up endpoint of 2018. We employed nine machine learning algorithms to construct predictive models for gastric ulcers over the next seven years (2011-2018, with 1482 samples) and the next three years (2014-2018, with 2659 samples). Additionally, we utilized cross-sectional data from 2018 (with 13,775 samples) to construct a predictive model for current gastric ulcers. Noninvasive predictors such as demographic, behavioral, nutritional, and physical examination factors were utilized to predict the current and future occurrence of gastric ulcers. In our study, Support Vector Machine (SVM), Random Forest (RF), and Light Gradient Boosting Machine (LGBM) achieved an accuracy of 0.97 for predicting gastric ulcers over seven years; Logistic Regression, Adaptive Boosting, SVM, RF, Gradient Boosting Machine, LGBM, and K-Nearest Neighbors reached 0.98 for three-year predictions; and SVM, Extreme Gradient Boosting, RF, and LGBM attained 0.95 for current gastric ulcer prediction. We developed MyGutRisk, built on optimal machine learning models, relatively accurately predicts gastric ulcer risk in elderly adults using noninvasive factors like diet and lifestyle. It supports self-assessment via a public link and clinical screening in community health settings to guide preventive measures. However, as a prototype, it requires further validation to ensure accuracy and generalizability across diverse populations and real-world applications.

Pulse wave velocity (PWV) can evaluate potential atherosclerosis (AS) and ultrafast pulse wave velocity (ufPWV) is a new technique to accurately assess PWV. However, few studies have examined the predictive value of ufPWV for AS risk. We aimed to establish a classification model for AS risk diagnosis based on ufPWV, so that AS can be diagnosed and prevented in advance. We collected imaging data, as well as clinical and laboratory data. A total of 613 patients with 20 attributes were admitted in this study. There were 392 patients with hyperlipidemia (AS risk group) and 221 healthy adults as the control group. In order to build AS risk prediction models, we considered decision tree, five different ensemble learning (EL) models [random forest (RF), adaptive boosting (AdaBoost), gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost) and light gradient boosting machine (LGBM)] and two different feature selection methods [statistical analysis and RF]. Accuracy and the area under the ROC curve (AUC) were used as the main criterion for model evaluation. In the prediction of AS risk with statistical analysis as the feature selection method, the performances of XGBoost (accuracy: 0.851; AUC: 0.884) and RF (accuracy: 0.844; AUC: 0.889) were better than other models. Besides, in the prediction of AS risk with RF as the feature selection method, the performances of LGBM (accuracy: 0.870; AUC: 0.903) and XGBoost (accuracy: 0.857; AUC: 0.903) were better than other models. In conclusions, EL models with RF as the feature selection method might provide accurate results in predicting AS risk. Besides, ufPWV, especially PWV of left common carotid artery at the end of systole, was an important feature in the AS risk prediction models.

While analyzing health data is important for improving health outcomes, class imbalance in datasets poses major challenges to machine learning classification models. This work, therefore, considers the class imbalance problem in stroke prediction using models such as K‐nearest neighbors, support vector machine, logistic regression, random forest, and decision tree. This work balances the stroke dataset, thereby enhancing model performance, through various oversampling strategies: random oversampling (RO), ADASYN, SMOTE, and SMOTE–Tomek. Compared to the results of the imbalanced dataset, all applied oversampling techniques enhanced the correct classification of stroke events by the ML model. Among these, RO–SVM with RBF kernel was the best in terms of sensitivity, specificity, G‐mean, F1‐score, and accuracy values, offering the highest results with respective values of 89.87%, 94.91%, 92.36%, 89.64%, and 89.87%. After applying oversampling techniques, all the machine learning classifications were good enough to classify stroke status, especially for the minority class. This study has highlighted the importance of class imbalance issues in health datasets. Precise detection of instances of minority classes can be enhanced considerably by employing classification models with the implementation of hybrid strategies to effectively solve class imbalance issues, which, in turn, will help improve healthcare outcomes. Further research in integrating more advanced deep learning techniques into other health datasets with imbalances is encouraged to further validate or refine class imbalance approaches, as effective handling of imbalanced classes can substantially promote predictive model performance in the analysis of healthcare.

Background/Objectives: Smartphone addiction (SA) and gaming addiction (GA) have become risk factors for individuals of all ages in recent years. Especially during adolescence, it has become very difficult for parents to control this situation. Physical activity and the effective use of free time are the most important factors in eliminating such addictions. This study aimed to test a new machine learning method by combining routine regression analysis with the gradient-boosting machine (GBM) and random forest (RF) methods to analyze the relationship between SA and GA with leisure time management (LTM) and the enjoyment of physical activity (EPA) among adolescents. Methods: This study presents the results obtained using our developed GBM + RF hybrid model, which incorporates LTM and EPA scores as inputs for predicting SA and GA, following the preprocessing of data collected from 1107 high school students aged 15-19 years. The results were compared with those obtained using routine regression results and the lasso, ElasticNet, RF, GBM, AdaBoost, bagging, support vector regression (SVR), K-nearest neighbors (KNN), multi-layer perceptron (MLP), and light gradient-boosting machine (LightGBM) models. In the GBM + RF model, probability scores obtained from GBM were used as input to RF to produce final predictions. The performance of the models was evaluated using the R2, mean absolute error (MAE), and mean squared error (MSE) metrics. Results: Classical regression analyses revealed a significant negative relationship between SA scores and both LTM and EPA scores. Specifically, as LTM and EPA scores increased, SA scores decreased significantly. In contrast, GA scores showed a significant negative relationship only with LTM scores, whereas EPA was not a significant determinant of GA. In contrast to the relatively low explanatory power of classical regression models, ML algorithms have demonstrated significantly higher prediction accuracy. The best performance for SA prediction was achieved using the Hybrid GBM + RF model (MAE = 0.095, MSE = 0.010, R2 = 0.9299), whereas the SVR model showed the weakest performance (MAE = 0.310, MSE = 0.096, R2 = 0.8615). Similarly, the Hybrid GBM + RF model also showed the highest performance for GA prediction (MAE = 0.090, MSE = 0.014, R2 = 0.9699). Conclusions: These findings demonstrate that classical regression analyses have limited explanatory power in capturing complex relationships between variables, whereas ML algorithms, particularly our GBM + RF hybrid model, offer more robust and accurate modeling capabilities for multifactorial cognitive and performance-related predictions.

Background and aimsThe birth weight of a newborn is a crucial factor that affects their overall health and future well-being. Low birth weight (LBW) is a widespread global issue, which the World Health Organization defines as weighing less than 2,500 g. LBW can have severe negative consequences on an individual’s health, including neonatal mortality and various health concerns throughout their life. To address this problem, this study has been conducted using BDHS 2017–2018 data to uncover important aspects of LBW using a variety of machine learning (ML) approaches and to determine the best feature selection technique and best predictive ML model.MethodsTo pick out the key features, the Boruta algorithm and wrapper method were used. Logistic Regression (LR) used as traditional method and several machine learning classifiers were then used, including, DT (Decision Tree), SVM (Support Vector Machine), NB (Naïve Bayes), RF (Random Forest), XGBoost (eXtreme Gradient Boosting), and AdaBoost (Adaptive Boosting), to determine the best model for predicting LBW. The model’s performance was evaluated based on the specificity, sensitivity, accuracy, F1 score and AUC value.ResultsResult shows, Boruta algorithm identifies eleven significant features including respondent’s age, highest education level, educational attainment, wealth index, age at first birth, weight, height, BMI, age at first sexual intercourse, birth order number, and whether the child is a twin. Incorporating Boruta algorithm’s significant features, the performance of traditional LR and ML methods including DT, SVM, NB, RF, XGBoost, and AB were evaluated where LR, had a specificity, sensitivity, accuracy and F1 score of 0.85, 0.5, 85.15% and 0.915. While the ML methods DT, SVM, NB, RF, XGBoost, and AB model’s respective accuracy values were 85.35%, 85.15%, 84.54%, 81.18%, and 84.41%. Based on the specificity, sensitivity, accuracy, F1 score and AUC, RF (specificity = 0.99, sensitivity = 0.58, accuracy = 85.86%, F1 score = 0.9243, AUC = 0.549) outperformed the other methods. Both the classical (LR) and machine learning (ML) models’ performance has improved dramatically when important characteristics are extracted using the wrapper method. The LR method identified five significant features with a specificity, sensitivity, accuracy and F1 score of 0.87, 0.33, 87.12% and 0.9309. The region, whether the infant is a twin, and cesarean delivery were the three key features discovered by the DT and RF models, which were implemented using the wrapper technique. All three models had the identical F1 score of 0.9318. However, “child is twin” was recognized as a significant feature by the SVM, NB, and AB models, with an F1 score of 0.9315. Ultimately, with an F1 score of 0.9315, the XGBoost model recognized “child is twin” and “age at first sex” as relevant features. Random Forest again beat the other approaches in this instance.ConclusionsThe study reveals Wrapper method as the optimal feature selection technique. The ML method outperforms traditional methods, with Random Forest (RF) being the most effective predictive model for Low-Birth-Weight prediction. The study suggests that policymakers in Bangladesh can mitigate low birth weight newborns by considering identified risk factors.

Predicting Caregiver Burden in Multiple Myeloma: Insights from the Carmma Study Using Machine Learning Models

Climate change is a pressing global challenge, notably impacting sensitive regions like the Alpine area. Its diverse terrain and ecology make it vulnerable to heightened climate risks, including intensified weather extremes due to global warming. Precise local climate predictions are vital for managing risks in vulnerable areas like the Alpine region, necessitating reliable high-resolution climate data and forecasts. Global products often fall short in providing the fine-grained information needed for accurate localized assessments. This work aims to address the critical need for refined, high-resolution seasonal climate forecasts in the Alpine region as a tool to increase the ability to manage and anticipate climate variability and hazardous conditions. The study endeavors to utilize Perfect Prognosis (PP) within Statistical Downscaling (SD), leveraging regression-based Machine Learning (ML) algorithms to enhance the spatial resolution of daily temperature and total precipitation of ECMWF (European Centre for Medium Range Weather Forecasts) SEAS5 (Seasonal Forecast System 5) seasonal forecasts. Four ensemble learning methods — random forest, light gradient-boosting machine (LGBM), Adaptive Boosting (AdaBoost) and Extreme Gradient Boosting (XGBoost) are considered, while CERRA (Copernicus European Regional Reanalysis) reanalysis (5.5 km) is used as reference target. In order to define the optimal ML model and configuration, a preparatory phase is performed in which ML methods are implemented, optimized and inter-compared by considering ERA5 reanalysis predictor fields (~ 31 km) for the training period 1985-2015 and validation period 2016-2020. Initial findings show that LGBM reports better performance in training and validation, demonstrating superior computational speed and efficiency with respect to the others. LGBM reconstructs daily variability with R2 scores of 0.95 for mean temperature and 0.67 for precipitation. Remaining bias as yearly average is -0.05°C fo daily mean temperature and 5.34% for daily precipitation. Other error metrics, e.g., mean absolute error (MAE) and root mean squared error (RMSE) suggest room for improvements, especially in extreme value predictions and annual precipitation averages. LGBM is thus applied and further optimized on SEAS5. The contribution will further elaborate the inter-comparison of ML models and their downscaling skills for seasonal forecasts will be presented and discussed. The expected outcomes of this study in particular will serve as a crucial input of a drought prediction module in the framework of the EU-funded interTwin project. This research has been funded by the European Union through the interTwin project (101058386).

<b>:</b> Cardiotocography (CTG) represents the fetus’s health inside the womb during labor. However, assessment of its readings can be a highly subjective process depending on the expertise of the obstetrician. Digital signals from fetal monitors acquire parameters (i.e., fetal heart rate, contractions, acceleration). <b>Objective::</b> This paper aims to classify the CTG readings containing imbalanced healthy, suspected, and pathological fetus readings. <b>Method::</b> We perform two sets of experiments. Firstly, we employ five classifiers: Random Forest (RF), Adaptive Boosting (AdaBoost), Categorical Boosting (CatBoost), Extreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LGBM) without over-sampling to classify CTG readings into three categories: healthy, suspected, and pathological. Secondly, we employ an ensemble of the above-described classifiers with the over-sampling method. We use a random over-sampling technique to balance CTG records to train the ensemble models. We use 3602 CTG readings to train the ensemble classifiers and 1201 records to evaluate them. The outcomes of these classifiers are then fed into the soft voting classifier to obtain the most accurate results. <b>Results::</b> Each classifier evaluates accuracy, Precision, Recall, F1-scores, and Area Under the Receiver Operating Curve (AUROC) values. Results reveal that the XGBoost, LGBM, and CatBoost classifiers yielded 99% accuracy. <b>Conclusion::</b> Using ensemble classifiers over a balanced CTG dataset improves the detection accuracy compared to the previous studies and our first experiment. A soft voting classifier then eliminates the weakness of one individual classifier to yield superior performance of the overall model.

A study on road accident prediction and contributing factors using explainable machine learning models: analysis and performance

The green concretes industry benefits from utilizing gel to replace parts of the cement in concretes. However, measuring the compressive strength of geo-polymer concretes (CSGPoC) needs a significant amount of work and expenditure. Therefore, the best idea is predicting CSGPoC with a high level of accuracy. To do this, the base learner and super learner machine learning models were proposed in this study to anticipate CSGPoC. The decision tree (DT) is applied as base learner, and the random forest and extreme gradient boosting (XGBoost) techniques are used as super learner system. In this regard, a database was provided involving 259 CSGPoC data samples, of which four-fifths of is considered for the training model and one-fifth is selected for the testing models. The values of fly ash, ground-granulated blast-furnace slag (GGBS), Na2SiO3, NaOH, fine aggregate, gravel 4/10 mm, gravel 10/20 mm, water/solids ratio, and NaOH molarity were considered as input of the models to estimate CSGPoC. To evaluate the reliability and performance of the decision tree (DT), XGBoost, and random forest (RF) models, 12 performance evaluation metrics were determined. Based on the obtained results, the highest degree of accuracy is achieved by the XGBoost model with mean absolute error (MAE) of 2.073, mean absolute percentage error (MAPE) of 5.547, Nash-Sutcliffe (NS) of 0.981, correlation coefficient (R) of 0.991, R2 of 0.982, root mean square error (RMSE) of 2.458, Willmott's index (WI) of 0.795, weighted mean absolute percentage error (WMAPE) of 0.046, Bias of 2.073, square index (SI) of 0.054, p of 0.027, mean relative error (MRE) of -0.014, and a20 of 0.983 for the training model and MAE of 2.06, MAPE of 6.553, NS of 0.985, R of 0.993, R2 of 0.986, RMSE of 2.307, WI of 0.818, WMAPE of 0.05, Bias of 2.06, SI of 0.056, p of 0.028, MRE of -0.015, and a20 of 0.949 for the testing model. By importing the testing set into trained models, values of 0.8969, 0.9857, and 0.9424 for R2 were obtained for DT, XGBoost, and RF, respectively, which show the superiority of the XGBoost model in CSGPoC estimation. In conclusion, the XGBoost model is capable of more accurately predicting CSGPoC than DT and RF models.

Handling class imbalance is a challenge in software defect prediction. Imbalanced datasets can cause bias in machine learning models, hindering their ability to detect defects. This paper proposes an integration of Adaptive Synthetic Sampling (ADASYN) and ensemble learning methods to improve prediction accuracy. ADASYN enhances the handling of imbalanced data by generating synthetic samples for hard-to-classify instances, while the ensemble stacking technique leverages the strengths of multiple models to reduce bias and variance. The machine learning model used in this study is K-Nearest Neighbors (KNN), Decision Tree (DT), and Random Forest (RF). The results demonstrate that ADASYN combined with ensemble stacking outperforms the traditional SMOTE technique in most cases. For instance, in the Ant-1.7 dataset, ADASYN achieved a stacking accuracy of 90.60% compared to 89.32% with SMOTE. Similarly, in the Camel-1.6 dataset, ADASYN achieved 91.56%, slightly exceeding SMOTE’s 91.32%. However, SMOTE performed better in simpler models like Decision Tree for certain datasets, highlighting the importance of choosing the appropriate resampling method. Across all datasets, ensemble stacking consistently provided the highest accuracy, benefiting from ADASYN's adaptive resampling strategy. These results underscore the importance of combining advanced sampling methods with ensemble learning techniques to address class imbalance effectively. This approach improves prediction accuracy and provides a practical framework for reliable software defect prediction in real-world scenarios. Future work will explore hybrid techniques and broader evaluations across diverse datasets and classifiers.

BackgroundThis study aims to develop a advanced machine learning model to predict the fall risk among community-dwelling elders. This study could present actionable advices for early prevention of fall risk.MethodsBetween October and December 2022, 977 older adults from the Hannan District of Wuhan were recruited. Data was collected using structured questionnaires. The sample was randomly split into training (732 participants) and testing (245 participants) sets at a 3:1 ratio. The primary outcome was the occurrence of fall. Five machine learning models—Random Forest (RF), Gradient Boosted Decision Tree (GBDT), Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting (XGBoost), and Categorical Features Gradient Boosting (CatBoost)—were evaluated against a Logistic Regression (LR) model. Model performance was assessed using AUC, accuracy, precision, sensitivity, specificity, and F1 score.ResultsAmong the 977 older adults, 195 experienced falls (20.0%). ROC curve analysis showed AUC values of LR, RF, LGBM, GBDT, XGBoost, and CatBoost were, respectively, 0.8390, 0.8632, 0.8614, 0.8544, 0.8705, and 0.8719. CatBoost had the highest AUC, indicating the best predictive performance. SHapley Additive exPlanations (SHAP) analysis identified key features influencing the CatBoost model: history of falls, comorbidities, polypharmacy, sleep disorders, ADL, TUG results, frailty status, and use of assistive devices.ConclusionThe fall risk prediction model for community-dwelling older adults, developed with CatBoost, showed excellent performance and can aid in early clinical assessment and fall prevention.

A lot of people have been focusing on developing AI that can spot fraudulent financial transactions recently. Fraudulent activity detection in bank payment systems is increasingly challenging due to rapidly changing transaction behaviors and class imbalance in datasets. As compared to more conventional forms of fraud detection, AI-powered systems may be able to spot suspicious behavior and stop it in its tracks at any moment. This research solves the issue by creating a strong machine learning and deep learning system using Kaggle-obtained bank payment data, which includes attributes like merchant category, age, gender, and transaction amount. The methodology involves comprehensive data preprocessing steps—dropping irrelevant features, performing category-wise behavioral analysis, and applying the SMOTE Tomek technique to balance the dataset. The cleaned and balanced data is then used to train and evaluate five models: Cat Boost, Gradient Boost, and Deep Neural Network (DNN). Results show that Cat Boost outperforms all models with an accuracy of 99.07%, followed by Gradient Boost and DNN with slightly lower metrics, while Logistic Regression and Bernoulli NB serve as baseline models with comparatively lower performance. The findings open the door to a more reliable and moral use of AI in detecting fraud. This work's main addition is that it uses SMOTE Tomek in conjunction with deep learning and powerful ensemble models to identify fraud in very unbalanced financial data. The significance of this work is shown by the outstanding performance of the Cat Boost model. It provides high accuracy, resilience, and practical application, making it an intelligent and dependable option for improving real-time fraud detection in financial systems.