Comparative Performance of Machine Learning Models in Reducing Unnecessary Targeted Prostate Biopsies.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Understanding the Performance of Machine Learning Models to Predict Credit Default: A Novel Approach for Supervisory Evaluation

Accurate wind speed forecasting is essential for daily activities and socio‐economic production. This study employs an explainable machine learning (ML) model to correct wind speed forecasts from a numerical weather prediction (NWP) model over weather stations across lead times of 1–24 hr in Zhejiang Province, China. Eleven meteorological variables derived from the NWP model are used as predictors. The results demonstrate that ML model substantially improves the original NWP forecasts, reducing the root mean square error of approximately 20%–40% at most stations and about 10% at certain coastal stations. The corrected forecasts also consistently achieve higher Kling‐Gupta Efficiency scores and Forecast Accuracy across all lead times. To enhance model interpretability, we apply Split Count and SHapley Additive exPlanation analyses. The results indicate that the friction velocity (UST) is the most influential predictor and reveal a threshold beyond which the ML model's correction performance improves markedly. The ML model performance slightly declines in coastal regions, likely due to the inconsistent representation of UST between the land and water surface in the NWP model. Monthly analysis further shows superior model performance in November and December, when UST values are higher. This study underscores the value of explainable ML models in improving NWP wind forecasts and provides insights for correcting meteorological variables in NWP models.

To enhance the performance of machine learning (ML) models for the post-embolization recanalization of cerebral aneurysms, we evaluated the impact of hemodynamic feature derivation and selection method on six ML algorithms. We utilized computational fluid dynamics (CFD) to simulate hemodynamics in 66 cerebral aneurysms from 65 patients, including 57 stable and nine recanalized aneurysms. We derived a total of 107 features for each aneurysm, encompassing four clinical features, 12 morphological features, and 91 hemodynamic features. To investigate the influence of feature derivation and selection methods on the ML models, we employed two derivation methods, simplified and fully derived, in combination with four selection methods: all features, statistically significant analysis, stepwise multivariate logistic regression analysis (stepwise-LR), and recursive feature elimination (RFE). Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and precision-recall curve (AUPRC) on both the training and testing datasets. The AUROC values on the testing dataset exhibited a wide-ranging spectrum, spanning from 0.373 to 0.863. Fully derived features and the RFE selection method demonstrated superior performance in intra-model comparisons. The multi-layer perceptron (MLP) model, trained with RFE-selected fully derived features, achieved the best performance on the testing dataset, with an AUROC value of 0.863 (95% CI: 0.684- 1.000). Our study demonstrated the importance of feature derivation and selection in determining the performance of ML models. This enabled the development of accurate decision-making models without the need to invade the patient.

Machine Learning (ML) models have been extensively applied in various fields to enhance prediction. For instance, in cybersecurity, they examine large amounts of data, establish trends in the data, and draw insights from previous events, to enhance detection and respond to cyber threats. Random Forest, Logistic Regression, K-Nearest Neighbor and LSTM are some of the popular ML models vastly used for anomaly detection. The accuracy of these models is therefore the cornerstone of the organization’s information systems security since wrong predictions result to false positives and negatives which significantly reduce employees’ output and may result into workers’ frustrations when interacting with the information systems. Among the many factors that affect ML model performance, data pre-processing has been underscored. Using the various publicly available datasets, this paper examines the impact of data preprocessing techniques on selected ML model architectures’ performance. Training time, Accuracy, Precision, Recall and F1 scores are used for evaluating the ML models’ performance.

Objective: To investigate the feasibility of a machine learning (ML) model based on radiomic features to identify active giant cell arteritis (GCA) in the aorta and differentiate it from atherosclerosis in follow-up [18F]FDG-PET/CT images for therapy monitoring. Methods: To train the ML model, 64 [18F]FDG-PET scans of 34 patients with proven GCA and 34 control subjects with type 2 diabetes mellitus were retrospectively included. The aorta was delineated into the ascending, arch, descending, and abdominal aorta. From each segment, 95 features were extracted. All segments were randomly split into a training/validation (n = 192; 80%) and test set (n = 46; 20%). In total, 441 ML models were trained, using combinations of seven feature selection methods, seven classifiers, and nine different numbers of features. The performance was assessed by area under the curve (AUC). The best performing ML model was compared to the clinical report of nuclear medicine physicians in 19 follow-up scans (7 active GCA, 12 inactive GCA). For explainability, an occlusion map was created to illustrate the important regions of the aorta for the decision of the ML model. Results: The ten-feature model with ANOVA as the feature selector and random forest classifier demonstrated the highest performance (AUC = 0.92 ± 0.01). Compared with the clinical report, this model showed a higher PPV (0.83 vs. 0.80), NPV (0.85 vs. 0.79), and accuracy (0.84 vs. 0.79) in the detection of active GCA in follow-up scans. Conclusions: The current radiomics ML model was able to identify active GCA and differentiate GCA from atherosclerosis in follow-up [18F]FDG-PET/CT scans. This demonstrates the potential of the ML model as a monitoring tool in challenging [18F]FDG-PET scans of GCA patients.

Machine Learning Models in the Prediction of One-Year Mortality in Patients With Advanced Hepatocellular Cancer on Immunotherapy

Optimizing asphalt mix design through predicting effective asphalt content and absorbed asphalt content using machine learning

Transfer learning enables prediction of steel corrosion in concrete under natural environments

Introduction Ensuring equitable performance of sleep-related machine learning (ML) models is vital for public health and health equity. This research sought to determine the impact of sociodemographic and health disparities factors on the performance and characteristics of ML models designed to predict OSA treatment initiation. Methods Our data source was the All-Payer Claims Database (APCD) for the Wisconsin Health Information Organization (WHIO). Inclusion criteria included continuous insurance coverage for >12 months prior and >30 months after OSA diagnosis (defined by OCD code G47.33), and having undergone OSA diagnostic testing (defined by CPT codes). OSA treatment was defined based on durable medical equipment charges for PAP machines and supplies. Sociodemographic variables were extracted from APCD and included race, gender, age, and area socioeconomic deprivation (Area Deprivation Index; ADI). The ADI is a validated marker of health risk based on 17 health disparities factors ranking relative disadvantage across communities. Random Forest ML models were trained to predict OSA treatment initiation using sociodemographic variables and a medication history including 39,712 unique medications codified across 94 medication categories. Results Of N=6,026,463 subjects in the ACPD, n=154,821 underwent OSA diagnostic testing, and n=43,601 were diagnosed with OSA. 10-Fold Cross-Validation training-testing was applied to estimate sensitivity-specificity of ML models for predicting treatment initiation. Receiver operating characteristic curve area-under-the-curve (ROC-AUC) analyses were used to compare relative differences in predictive power of each variable. In ROC-AUC analysis of individual variables, the power for predicting OSA treatment were observed in relative rank order by age (0.568), race (0.547), ADI national-level (0.545), ADI state-level (0.544), gender (0.524), and medication history (0.516). In ROC-AUC analysis of combination variables, the highest ML model performance was observed in the combination of only three-variables (age-gender-race, 0.600), while the combination of all-variables showed an ROC-AUC value of (0.594), and the resulting performance difference was not statistically significant based on comparison of the ROC-AUC measures. Conclusion Demographic and health disparity factors may play an important role in future development of predictive AI/ML models. Population sleep health data represents an important resource to identify and bridge care gaps, reduce sleep health disparities, and achieve health equity. Support (if any)

MP07-06 EVALUATING THE EFFICACY OF ARTIFICIAL INTELLIGENCE IN PREDICTING OUTCOMES FOR PEDIATRIC HYDRONEPHROSIS: A SYSTEMATIC REVIEW

Introduction: Early and accurate diagnosis of transthyretin amyloid cardiomyopathy (ATTR-CM) is known to improve prognosis. However, it is sometimes challenging to distinguish ATTR-CM from other cardiac conditions that cause left ventricular hypertrophy (LVH), such as hypertrophic cardiomyopathy (HCM), hypertensive LVH (h-LVH), and aortic stenosis (AS). The Mayo ATTR-CM score is commonly used to distinguish ATTR-CM from these mimickers. Herein, we constructed a machine learning (ML) model and compared its performance to the Mayo score. Hypothesis: A ML model based on readily available clinical variables will outperform the Mayo score in distinguishing ATTR-CM from other cardiac conditions that cause LVH. Methods: We performed a multicenter case-control study of 1,605 patients, including 303 ATTR-CM cases and 1,302 controls – i.e., 935 with HCM, 331 with h-LVH, and 36 with AS. We allocated the former two-thirds of each disease group to the training set and the latter one-third to the prospective test set for validation. We developed a main ML model in the training set using XGBoost with 18 clinical variables including demographics, comorbidities, laboratory values, and echocardiographic data. We examined its performance in the test set using the area under the receiver-operating-characteristic curve (AUC) and decision curve analysis (DCA). In addition, we developed a simplified ML model using only the top 5 variables identified by feature importance and tested its performance. Results: The top 5 most important variables were age, LV diameter in systole, LV posterior wall thickness in diastole, LV ejection fraction, and left atrial diameter ( Figure 1 ). The main ML model (AUC: 0.96, 95% CI: 0.93–0.98) outperformed the Mayo score (AUC: 0.84, 95% CI: 0.79–0.89, DeLong’s P <0.0001; Figure 2 ) in the test set. DCA demonstrated that the ML model provided greater net benefit across all threshold probabilities ( Figure 3 ). The simplified 5-variable model also outperformed the Mayo score (AUC: 0.94, 95% CI: 0.91–0.97, DeLong’s P <0.0001). Conclusions: In this multicenter study, we developed ML models that effectively distinguish ATTR-CM from its mimickers that cause LVH using readily available clinical variables. Given the simplicity and accuracy of our models, the present study serves as an important initial step toward the development of a robust, data-driven tool that could be readily automated and integrated into electronic medical record system to screen for ATTR-CM.

Background Peripheral Arterial Disease(PAD) is a prevalent cardiovascular condition affecting millions worldwide, and the advent of machine learning(ML) techniques offers promising opportunities to enhance risk prediction and personalise patient management in this complex disease. This systematic review and meta-analysis aimed to evaluate the performance of ML models in predicting adverse outcomes in PAD patients compared to traditional statistical approaches. Methods A comprehensive search of major databases(MEDLINE,EMBASE,CINAHL,Cochrane CENTRAL) was conducted for studies published between 2000 and 2024. Two reviewers independently screened studies, extracted data, and assessed risk of bias using the PROBAST tool. The quality of included studies was also assessed using criteria outlined by Qiao et al. A modified Hierarchical Summary Receiver Operating Characteristic(HSROC) analysis was performed to compare the predictive performance of various ML models and traditional regression methods. Results Thirteen studies met the inclusion criteria. Gradient boosted models demonstrated the highest predictive performance with a diagnostic odds ratio(DOR) of 36.593(95% CI:24.44-54.79), sensitivity of 0.853, and specificity of 0.863. The best performing ML models for outcomes within 2 years, showed a DOR of 34.169(95% CI:22.840-51.117). Traditional regression models consistently underperformed compared to ML approaches, with the lowest DOR of 3.326(95% CI:1.814-6.097). The quality assessment revealed a mix of methodological rigor, with 54% of studies rated as low risk of bias and 46% as unclear. Conclusion Advanced ML techniques demonstrate superior predictive power for adverse outcomes in PAD patients compared to traditional regression methods.

Integrating machine learning (ML) models into healthcare systems is a rapidly evolving field with the potential to revolutionize care delivery. This study aimed to classify fertility rates and identify significant predictors using ML models among reproductive women in Ethiopia. This study utilized eight ML models in 5864 reproductive-age women using Ethiopian Demographic Health Survey (EDHS), 2019 data. Phyton programming language was used to develop these models. Predictors of fertility rate were determined using the feature important techniques. The performance of models was evaluated using accuracy, area under the curve (AUC), precision, recall, F1-score, specificity, and sensitivity. The mean age of participants was 32.7 (± 5.6) years. The random forest classifier (accuracy = 0.901 and AUC = 0.961) followed by a one-dimensional convolutional neural network (accuracy = 0.899 and AUC = 0.958), logistic regression (accuracy = 0.874 and AUC = 0.937), and gradient boost classifier (accuracy = 0.851 and AUC 0.927) were the top performing ML models. Family size, age, occupation, and education with an average importance score of 0.198, 0.151, 0.118, and 0.081, respectively were the top significant predictors of the fertility rate. The best ML models to classify and predict fertility rates were random forest, one-dimensional convolutional neural network, logistic regression, and gradient boost classifier. The findings on important factors of fertility rate can inform targeted public health, programs that address disparities related to family size, occupation, education, and other socioeconomic factors.

In recent years, infections and damage caused by malware have increased at exponential rates. At the same time, machine learning (ML) techniques have shown tremendous promise in many domains, often out performing human efforts by learning from large amounts of data. Results in the open literature suggest that ML is able to provide similar results for malware detection, achieving greater than 99% classification accuracy [49]. However, the same detection rates when applied in deployed settings have not been achieved. Malware is distinct from many other domains in which ML has shown success in that (1) it purposefully tries to hide, leading to noisy labels and (2) often its behavior is similar to benign software only differing in intent, among other complicating factors. This report details the reasons for the difficultly of detecting novel malware by ML methods and offers solutions to improve the detection of novel malware. We propose to detect malware by detecting behaviors commonly exhibited by malware such as DLL injection, and process hollowing. This is based on the assumption that there is a set of behaviors that are common to most malware samples and detecting them will generalize to novel malware. Additionally, detected behaviors point analysts toward appropriate handling and mitigation strategies, which is not the case with a binary benign/malicious classification. A behavior labeling method was developed and was used to label an existing malware dataset. Results show that detecting malicious behaviors is much more difficult than simply classifying malware and goodware?achieving 80% accuracy compared to reported 99% accuracy from classifying malware and goodware. This drop is due to several reasons which are detailed in the report. We also propose to evaluate the performance of detecting novel malware by holding out a malware family for testing and training on the other families. Traditional ML evaluation will shuffle the data and then split the data into training and testing. Our method addresses the use-case when novel malware families are encountered and they require more than just a malicious or benign designation. Our results suggest that this type of evaluation is much more difficult than traditional methods and provides more realistic results, albeit, significantly worse. For our behavior detection, accuracy decreases from 80% to 68% across all behaviors when holding out a malware family from training. We show that the degradation in performance is because each malware family has distinct characteristics resulting in high extrapolations by an ML model. Here, an ML model should return an "I do not know" response and request further analysis from an analyst. We run a number of experiments that compare novel malware families to the training data using different feature representations including a genomics-inspired distance measure and features extracted by deep learning. Generally, held-out families are significantly different from the training data, resulting in unpredictable results. This has been observed generally in the ML community [22, 9]. We empirically demonstrate this in the domain of malware detection. In an attempt to improve the detection of malware behaviors, we examine the impact that additional synthetic data has on the performance of an ML model in detecting behaviors in novel malware families. We find that while synthetic data does improve the performance of ML models, often simpler methods perform better than more complicated ones. Two generative modeling techniques were examined to produce synthetic malware samples such that the behaviors present are able to be specified externally. The difficulty is due to finer grained analysis of the executable and modifying the problem from a binary classification problem to a multi-label problem. The addition of synthetic data increases the overall accuracy from 68% to 70%. While far less accurate than measures presented in academic analyses, we believe that this is more representative of real-world performance and allows models to be properly placed within a malware detection system. We suggest that in highly dynamic environments ML pipelines should determine whether an ML model is competent in the area of new data and should involve mechanisms to improve over time with a human in the loop.