MLcps: machine learning cumulative performance score for classification problems.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Understanding the Performance of Machine Learning Models to Predict Credit Default: A Novel Approach for Supervisory Evaluation

Accurate wind speed forecasting is essential for daily activities and socio‐economic production. This study employs an explainable machine learning (ML) model to correct wind speed forecasts from a numerical weather prediction (NWP) model over weather stations across lead times of 1–24 hr in Zhejiang Province, China. Eleven meteorological variables derived from the NWP model are used as predictors. The results demonstrate that ML model substantially improves the original NWP forecasts, reducing the root mean square error of approximately 20%–40% at most stations and about 10% at certain coastal stations. The corrected forecasts also consistently achieve higher Kling‐Gupta Efficiency scores and Forecast Accuracy across all lead times. To enhance model interpretability, we apply Split Count and SHapley Additive exPlanation analyses. The results indicate that the friction velocity (UST) is the most influential predictor and reveal a threshold beyond which the ML model's correction performance improves markedly. The ML model performance slightly declines in coastal regions, likely due to the inconsistent representation of UST between the land and water surface in the NWP model. Monthly analysis further shows superior model performance in November and December, when UST values are higher. This study underscores the value of explainable ML models in improving NWP wind forecasts and provides insights for correcting meteorological variables in NWP models.

To enhance the performance of machine learning (ML) models for the post-embolization recanalization of cerebral aneurysms, we evaluated the impact of hemodynamic feature derivation and selection method on six ML algorithms. We utilized computational fluid dynamics (CFD) to simulate hemodynamics in 66 cerebral aneurysms from 65 patients, including 57 stable and nine recanalized aneurysms. We derived a total of 107 features for each aneurysm, encompassing four clinical features, 12 morphological features, and 91 hemodynamic features. To investigate the influence of feature derivation and selection methods on the ML models, we employed two derivation methods, simplified and fully derived, in combination with four selection methods: all features, statistically significant analysis, stepwise multivariate logistic regression analysis (stepwise-LR), and recursive feature elimination (RFE). Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and precision-recall curve (AUPRC) on both the training and testing datasets. The AUROC values on the testing dataset exhibited a wide-ranging spectrum, spanning from 0.373 to 0.863. Fully derived features and the RFE selection method demonstrated superior performance in intra-model comparisons. The multi-layer perceptron (MLP) model, trained with RFE-selected fully derived features, achieved the best performance on the testing dataset, with an AUROC value of 0.863 (95% CI: 0.684- 1.000). Our study demonstrated the importance of feature derivation and selection in determining the performance of ML models. This enabled the development of accurate decision-making models without the need to invade the patient.

Machine Learning (ML) models have been extensively applied in various fields to enhance prediction. For instance, in cybersecurity, they examine large amounts of data, establish trends in the data, and draw insights from previous events, to enhance detection and respond to cyber threats. Random Forest, Logistic Regression, K-Nearest Neighbor and LSTM are some of the popular ML models vastly used for anomaly detection. The accuracy of these models is therefore the cornerstone of the organization’s information systems security since wrong predictions result to false positives and negatives which significantly reduce employees’ output and may result into workers’ frustrations when interacting with the information systems. Among the many factors that affect ML model performance, data pre-processing has been underscored. Using the various publicly available datasets, this paper examines the impact of data preprocessing techniques on selected ML model architectures’ performance. Training time, Accuracy, Precision, Recall and F1 scores are used for evaluating the ML models’ performance.

Introduction Ensuring equitable performance of sleep-related machine learning (ML) models is vital for public health and health equity. This research sought to determine the impact of sociodemographic and health disparities factors on the performance and characteristics of ML models designed to predict OSA treatment initiation. Methods Our data source was the All-Payer Claims Database (APCD) for the Wisconsin Health Information Organization (WHIO). Inclusion criteria included continuous insurance coverage for >12 months prior and >30 months after OSA diagnosis (defined by OCD code G47.33), and having undergone OSA diagnostic testing (defined by CPT codes). OSA treatment was defined based on durable medical equipment charges for PAP machines and supplies. Sociodemographic variables were extracted from APCD and included race, gender, age, and area socioeconomic deprivation (Area Deprivation Index; ADI). The ADI is a validated marker of health risk based on 17 health disparities factors ranking relative disadvantage across communities. Random Forest ML models were trained to predict OSA treatment initiation using sociodemographic variables and a medication history including 39,712 unique medications codified across 94 medication categories. Results Of N=6,026,463 subjects in the ACPD, n=154,821 underwent OSA diagnostic testing, and n=43,601 were diagnosed with OSA. 10-Fold Cross-Validation training-testing was applied to estimate sensitivity-specificity of ML models for predicting treatment initiation. Receiver operating characteristic curve area-under-the-curve (ROC-AUC) analyses were used to compare relative differences in predictive power of each variable. In ROC-AUC analysis of individual variables, the power for predicting OSA treatment were observed in relative rank order by age (0.568), race (0.547), ADI national-level (0.545), ADI state-level (0.544), gender (0.524), and medication history (0.516). In ROC-AUC analysis of combination variables, the highest ML model performance was observed in the combination of only three-variables (age-gender-race, 0.600), while the combination of all-variables showed an ROC-AUC value of (0.594), and the resulting performance difference was not statistically significant based on comparison of the ROC-AUC measures. Conclusion Demographic and health disparity factors may play an important role in future development of predictive AI/ML models. Population sleep health data represents an important resource to identify and bridge care gaps, reduce sleep health disparities, and achieve health equity. Support (if any)

Understanding the BRAF alterations preoperatively could remarkably assist in predicting tumor behavior, which leads to a more precise prognostication and management strategy. Recent advances in artificial intelligence (AI) have resulted in effective predictive models. Therefore, for the first time, this study aimed to review the performance of machine learning and deep learning models in predicting the BRAF alterations in low-grade gliomas (LGGs)using imaging data. PubMed/MEDLINE, Embase, and the Cochrane Library were systematically searched for studies published up to June 1, 2024, and evaluated the performance of AI models in predicting BRAF alterations in LGGs using imaging data. Pooled sensitivity, specificity, and area under the receiver operating curve (AUROC) were meta-analyzed. A total of 6 studies with 951 patients were included in this systematic review. The pooled AUROC of internal validation cohorts for their best-performing model was 84.44%, with models detecting BRAF mutation, BRAF fusion, BRAF fusion from mutation, and BRAF wild type producing similar AUROCs of 90.75%, 84.59%, 82.33%, and 82%, respectively. The best-performing models had pooled sensitivities of 80.3%, 87.51%, and 74.14% and pooled specificities of 88.57%, 70.41%, and 83.98% for detection of BRAF fusion from mutation, BRAF fusion, and BRAF mutation, respectively. AI models may perform relatively well in predicting BRAF alterations in LGG using imaging data and appear to be capable of high sensitivities and specificities. However, future studies with larger sample sizes implementing different machine learning or deep learning algorithms are required to reduce imprecision.

MP07-06 EVALUATING THE EFFICACY OF ARTIFICIAL INTELLIGENCE IN PREDICTING OUTCOMES FOR PEDIATRIC HYDRONEPHROSIS: A SYSTEMATIC REVIEW

Student’s mental health problems have been explored previously in higher education literature in various contexts including empirical work involving quantitative and qualitative methods. Nevertheless, comparatively few research could be found, aiming for computational methods that learn information directly from data without relying on set parameters for a predetermined equation as an analytical method. This study aims to investigate the performance of Machine learning (ML) models used in higher education. ML models considered are Naïve Bayes, Support Vector Machine, K-Nearest Neighbor, Logistic Regression, Stochastic Gradient Descent, Decision Tree, Random Forest, XGBoost (Extreme Gradient Boosting Decision Tree), and NGBoost (Natural) algorithm. Considering the factors of mental health illness among students, we follow three phases of data processing: segmentation, feature extraction, and classification. We evaluate these ML models against classification performance metrics such as accuracy, precision, recall, F1 score, and predicted run time. The empirical analysis includes two contributions: 1. It examines the performance of various ML models on a survey-based educational dataset, inferring a significant classification performance by a tree-based XGBoost algorithm; 2. It explores the feature importance [variables] from the datasets to infer the significant importance of social support, learning environment, and childhood adversities on a student’s mental health illness.

Machine Learning Models in the Prediction of One-Year Mortality in Patients With Advanced Hepatocellular Cancer on Immunotherapy

PD27-01 DEVELOPMENT OF A MACHINE LEARNING (ML) MODEL TO AUTOMATICALLY AND PRECISELY IDENTIFY KIDNEY STONES FROM URETEROSCOPY VIDEO RECORDINGS

Optimizing asphalt mix design through predicting effective asphalt content and absorbed asphalt content using machine learning

Transfer learning enables prediction of steel corrosion in concrete under natural environments

Integrating machine learning (ML) models into healthcare systems is a rapidly evolving field with the potential to revolutionize care delivery. This study aimed to classify fertility rates and identify significant predictors using ML models among reproductive women in Ethiopia. This study utilized eight ML models in 5864 reproductive-age women using Ethiopian Demographic Health Survey (EDHS), 2019 data. Phyton programming language was used to develop these models. Predictors of fertility rate were determined using the feature important techniques. The performance of models was evaluated using accuracy, area under the curve (AUC), precision, recall, F1-score, specificity, and sensitivity. The mean age of participants was 32.7 (± 5.6) years. The random forest classifier (accuracy = 0.901 and AUC = 0.961) followed by a one-dimensional convolutional neural network (accuracy = 0.899 and AUC = 0.958), logistic regression (accuracy = 0.874 and AUC = 0.937), and gradient boost classifier (accuracy = 0.851 and AUC 0.927) were the top performing ML models. Family size, age, occupation, and education with an average importance score of 0.198, 0.151, 0.118, and 0.081, respectively were the top significant predictors of the fertility rate. The best ML models to classify and predict fertility rates were random forest, one-dimensional convolutional neural network, logistic regression, and gradient boost classifier. The findings on important factors of fertility rate can inform targeted public health, programs that address disparities related to family size, occupation, education, and other socioeconomic factors.

Comparative Performance of Machine Learning Models in Reducing Unnecessary Targeted Prostate Biopsies.