Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction

BackgroundBluetongue (BT) is a disease of concern to animal breeders, so the question on their minds is whether they can predict the risk of the disease before it occurs. The main objective of this study is to enhance the accuracy of BT risk prediction by relying on machine learning (ML) approaches to help in fulfilling this inquiry. Several risk factors of BT that affect the occurrence and magnitude of animal infection with the virus have been reported globally. Additionally, risk factors, such as sex, age, species, and season, unevenly affect animal health and welfare. Therefore, the seroprevalence study data of 233 apparently healthy animals (125 sheep and 108 goats) from five different provinces in Egypt were used to analyze and compare the performance of the algorithms in predicting BT risk.ResultsLogistic regression (LR), decision tree (DT), random forest (RF), and a feedforward artificial neural network (ANN) were used to develop predictive BT risk models and compare their performance to the base model (LR). Model performance was assessed by the area under the receiver operating characteristics curve (AUC), accuracy, true positive rate (TPR), false positive rate (FPR), false negative rate (FNR), precision, and F1 score. The results indicated that RF performed better than other models, with an AUC score of 81%, ANN of 79.6%, and DT of 72.85%. In terms of performance and prediction, LR showed a much lower value (AUC = 69%). Upon further observation of the results, it was discovered that age and season were the most important predictor variables reported in classification and prediction.ConclusionThe findings of this study can be utilized to predict and control BT risk factors in sheep and goats, with better diagnostic discrimination in terms of accuracy, TPR, FNR, FPR, and precision of ML models over traditional and commonly used LR models. Our findings advocate that the implementation of ML algorithms, mainly RF, in farm decision making and prediction is a promising technique for analyzing cross-section studies, providing adequate predictive power and significant competence in identifying and ranking predictors representing potential risk factors for BT.

The application of machine learning models for prediction and prognosis of disease development has become an irrevocable part of cancer studies aimed at improving the subsequent therapy and management of patients. The application of machine learning models for accurate prediction of survival time in breast cancer on the basis of clinical data is the main objective of the presented study. The paper discusses an approach to the problem in which the main factor used to predict survival time is the originally developed tumor-integrated clinical feature, which combines tumor stage, tumor size, and age at diagnosis. Two datasets from corresponding breast cancer studies are united by applying a data integration approach based on horizontal and vertical integration by using proper document-oriented and graph databases which show good performance and no data losses. Aside from data normalization and classification, the applied machine learning methods provide promising results in terms of accuracy of survival time prediction. The analysis of our experiments shows an advantage of the linear Support Vector Regression, Lasso regression, Kernel Ridge regression, K-neighborhood regression, and Decision Tree regression—these models achieve most accurate survival prognosis results. The cross-validation for accuracy demonstrates best performance of the same models on the studied breast cancer data. As a support for the proposed approach, a Python-based workflow has been developed and the plans for its further improvement are finally discussed in the paper.

BackgroundIn an intensive care units, experts in mechanical ventilation are not continuously at patient’s bedside to adjust ventilation settings and to analyze the impact of these adjustments on gas exchange. The development of clinical decision support systems analyzing patients’ data in real time offers an opportunity to fill this gap.ObjectiveThe objective of this study was to determine whether a machine learning predictive model could be trained on a set of clinical data and used to predict transcutaneous hemoglobin oxygen saturation 5 min (5min SpO2) after a ventilator setting change.Data sourcesData of mechanically ventilated children admitted between May 2015 and April 2017 were included and extracted from a high-resolution research database. More than 776,727 data rows were obtained from 610 patients, discretized into 3 class labels (< 84%, 85% to 91% and c92% to 100%).Performance metrics of predictive modelsDue to data imbalance, four different data balancing processes were applied. Then, two machine learning models (artificial neural network and Bootstrap aggregation of complex decision trees) were trained and tested on these four different balanced datasets. The best model predicted SpO2 with area under the curves < 0.75.ConclusionThis single center pilot study using machine learning predictive model resulted in an algorithm with poor accuracy. The comparison of machine learning models showed that bagged complex trees was a promising approach. However, there is a need to improve these models before incorporating them into a clinical decision support systems. One potentially solution for improving predictive model, would be to increase the amount of data available to limit over-fitting that is potentially one of the cause for poor classification performances for 2 of the three class labels.

Machine learning (ML) methods for the analysis of electrocardiography (ECG) data are gaining importance, substantially supported by the release of large public datasets. However, these current datasets miss important derived descriptors such as ECG features that have been devised in the past hundred years and still form the basis of most automatic ECG analysis algorithms and are critical for cardiologists’ decision processes. ECG features are available from sophisticated commercial software but are not accessible to the general public. To alleviate this issue, we add ECG features from two leading commercial algorithms and an open-source implementation supplemented by a set of automatic diagnostic statements from a commercial ECG analysis software in preprocessed format. This allows the comparison of ML models trained on clinically versus automatically generated label sets. We provide an extensive technical validation of features and diagnostic statements for ML applications. We believe this release crucially enhances the usability of the PTB-XL dataset as a reference dataset for ML methods in the context of ECG data.

Accurate risk assessment of high-risk patients is essential in clinical practice. However, there is no practical method to predict or monitor the prognosis of patients with ST-segment elevation myocardial infarction (STEMI) complicated by hyperuricemia. We aimed to evaluate the performance of different machine learning models for the prediction of 1-year mortality in STEMI patients with hyperuricemia. We compared five machine learning models (logistic regression, k-nearest neighbor, CatBoost, random forest, and XGBoost) with the traditional global (GRACE) risk score for acute coronary event registrations. We registered patients aged >18 years diagnosed with STEMI and hyperuricemia at the Affiliated Hospital of Zunyi Medical University between January 2016 and January 2020. Overall, 656 patients were enrolled (average age, 62.5 ± 13.6 years; 83.6%, male). All patients underwent emergency percutaneous coronary intervention. We evaluated the performance of five machine learning classifiers and the GRACE risk model in predicting 1-year mortality. The area under the curve (AUC) of the six models, including the GRACE risk model, ranged from 0.75 to 0.88. Among all the models, CatBoost had the highest predictive accuracy (0.89), AUC (0.87), precision (0.84), and F1 value (0.44). After hybrid sampling technique optimization, CatBoost had the highest accuracy (0.96), AUC (0.99), precision (0.95), and F1 value (0.97). Machine learning algorithms, especially the CatBoost model, can accurately predict the mortality associated with STEMI complicated by hyperuricemia after a 1-year follow-up.

Rainfall plays a crucial role in agricultural practices, influencing the growth of essential crops and impacting economies reliant on agriculture. In Rajasthan, India, where agriculture is predominantly rainfed and dependent on unpredictable monsoon rains, accurate rainfall forecasting is vital for effective crop management, irrigation, optimizing agricultural productivity, managing water resources, and addressing drought conditions and. This study evaluates various machine learning models for predicting rainfall in East and West Rajasthan. The models assessed include Support Vector Regression (SVR), Random Forest (RF), Artificial Neural Networks (ANN), and K-Nearest Neighbours (KNN). These models were compared based on their accuracy measures, including Root Mean Squared Error (RMSE), Relative Root Squared Error (RRSE), and Relative Absolute Error (RAE). The results indicate that ANN and KNN perform poorly compared to SVR and RF suggesting their inadequacy for rainfall forecasting in water-scarce regions. The performance of SVR and RF are comparable as the difference is less. However SVR is identified as the most effective model for rainfall prediction in both subdivisions, as highlighted in the study. This study suggest that SVR and RF model should be includes in prediction of amount of rainfall.

e13596 Background: Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer (BC) and neoadjuvant therapy (NAT) has become an essential treatment strategy. Recent studies have showed tertiary lymphoid structures (TLSs) is associated with better treatment response to NAT in BC. However, there is still a lack of non-invasive biomarkers to predict the presence of TLSs in TNBC. This study aimed to develop a machine learning model for early identification and prediction of the presence of TLSs and treatment response to NAT in TNBC, which were essential for timely adjustments in treatment strategies for TNBC. Methods: In this study, 698 patients from multicenter were divided into a training cohort (n = 137), the TLSs validation cohort (n = 63) and the NAT response validation cohorts (n = 561). A total of 4788 radiomic features per patient were extracted from the intratumoral and peritumoral regions of DCE-MR. After extracting the optimal features, multiple machine learning models were developed to predict the presence of TLSs and were subsequently applied to predict the treatment response to NAT in NAT response validation cohorts. The performance of the models was assessed by area under curve values (AUC), and prognostic analysis were performed to evaluate its predictive value. Finally, the correlation between key radiomic features and key pathomic features was further analyzed to reveal the tumor heterogeneity of TNBC from a pathological perspective. Results: The XGBoost model, which outperformed all other predictive models, was selected as the radiomics-based TLS (rTLS) predictive model. It achieved AUCs of 0.922 in the training cohort, 0.852 in the TLS validation cohort, 0.724 in the Chinese-TNBC cohort, 0.919 in the DUKE cohort, and 0.883 in the I-SPY2 cohort. The rTLS predictive model demonstrated robust predictive performance, including across various patient subgroups defined by age, menopausal status, Ki67 level, cT stage, cN stage, and MammaPrint gene status. In YNCH cohort and DUKE cohort, prognostic analysis showed that low rTLS predictive score was significantly correlated with better disease-free survival in TNBC receiving NAT, and Cox regression analysis also confirmed the rTLS predictive score was a strong independent prognostic factor. Pathomic features further explained the pathological heterogeneity of TNBC with different responses to NAT. Conclusions: The rTLS predictive model, which accurately predicted the presence of TLSs and treatment response to NAT in TNBC, held promise for future clinical application in formulating personalized and effective treatment strategies for TNBC, ultimately improving prognosis.

This study presents a comparative analysis of machine learning models for predicting carbon monoxide (CO) emissions in automotive engines. Four models—Linear Regression, Decision Tree, Random Forest, and Support Vector Regression—were evaluated using a dataset of engine performance parameters and emission measurements. Among these, the Random Forest model demonstrated the highest predictive accuracy, achieving an R² score of 0.8965. Feature importance analysis identified nitrogen oxides (NOX), engine speed (RPM), and hydrocarbons (HC) as the most significant predictors of carbon monoxide emissions. Learning curve analysis provided insights into model generalization and highlighted potential limitations. The study underscores the value of data-driven approaches in optimizing engine design and controlling emissions. The findings contribute to the development of cleaner, more efficient vehicles, supporting sustainability efforts in the automotive industry. This research bridges data science and automotive engineering, offering a framework for advanced emission prediction and control that can be applied to other pollutants and engine types.

Knowledge of phenological events and their variability can help to determine final yield, plan management approach, tackle climate change, and model crop development. THe timing of phenological stages and phases is known to be highly correlated with temperature which is therefore an essential component for building phenological models. Satellite data and, particularly, Copernicus’ ERA5 climate reanalysis data are easily available. Weather stations, on the other hand, provide scattered temperature data, with fragmentary spatial coverage and accessibility, as such being scarcely efficacious as unique source of information for the implementation of predictive models. However, as ERA5 reanalysis data are not real temperature measurements but reanalysis products, it is necessary to verify whether these data can be used as a replacement for weather station temperature measurements. The aims of this study were: (i) to assess the validity of ERA5 data as a substitute for weather station temperature measurements, (ii) to test different machine learning models for the prediction of phenological phases while using different sets of features, and (iii) to optimize the base temperature of olive tree phenological model. The predictive capability of machine learning models and the performance of different feature subsets were assessed when comparing the recorded temperature data, ERA5 data, and a simple growing degree day phenological model as benchmark. Data on olive tree phenology observation, which were collected in Tuscany for three years, provided the phenological phases to be used as target variables. The results show that ERA5 climate reanalysis data can be used for modelling phenological phases and that these models provide better predictions in comparison with the models trained with weather station temperature measurements.

College graduates earn substantially more and are more likely to be employed. Consequently, it is critically important to understand the predictors of college dropout so that students and administrators can make a difference in college graduation outcomes. Previous studies remain limited in the scope of evaluating machine learning models for dropout prediction. Leveraging a dataset of 4,424 students that includes graduation outcome, demographic, socioeconomic and course data, and macroeconomic data, the objective of this paper is to identify the optimum machine learning model for predicting college dropout as a classification problem. We (a) perform extensive exploratory data analysis, (b) perform feature optimization (c) identify the best performing machine learning model across seven models evaluated, (d) study different testing-to-training ratios, (e) perform a comprehensive model evaluation, and (f) compare a multi-class classification approach to a binary classification one. The models were fine-tuned leveraging a grid search optimization algorithm and validated with k-fold cross-validation. Optimizing the hyperparameters, the grid search optimized random forest model performed the best in predicting college dropout with 0.85 accuracy, 0.72 sensitivity, 0.92 specificity, 0.82 precision, and 0.89 AUC-ROC. Furthermore, the optimized random forest model suggested the key predictors of dropout, in order of importance to be: number of curricular units in the second semester, number of curricular units in the first semester and whether the tuition and fees are up-to-date. The findings underscore the value of using machine learning for timely dropout risk prediction, enabling targeted resource allocation to mitigate risk and support successful graduation outcomes.

Heart failure (HF) is one of the major causes of morbidity and mortality globally, necessitating accurate tools for health outcome prediction and risk stratification. In this study, we propose an interpretable multimodal machine learning framework integrating four clinical data modalities (i.e., demographics, medications, laboratory tests, and electrocardiograms [ECGs]) to predict 30-day all-cause mortality and hospital readmission in HF patients. Using clinical data from 2868 HF patients across 43 local hospitals in Hong Kong, we trained and evaluated ten machine learning models for HF risk prediction, with the best performing model achieving an area under the receiver operating characteristic curve (AUC) of 0.881 for mortality and 0.709 for readmission. Notably, laboratory tests and ECG features dominate predictive power, and their combination alone yielded near-optimal results (AUC: 0.872), suggesting that these two modalities may be adequate for effective risk prediction in resource-constrained settings. The SHapley Additive exPlanations (SHAP) analysis identified serum albumin, high-sensitivity troponin I, lactate dehydrogenase, and QT interval dispersion as key predictors. Feature redundancy analysis further revealed strong correlations within laboratory tests and ECG features, suggesting opportunities for model simplification. To the best of our knowledge, this is the first study that comprehensively evaluates diverse configurations of four data modalities for HF risk prediction through ablation analysis, quantifying the marginal gains of each data modality and their combinations. Our findings demonstrate that an interpretable multimodal machine learning model can enhance risk prediction in HF patients, supporting personalized management and scalable deployment across diverse healthcare settings.

Arsenic, aluminum, iron, lead, chromium, copper, zinc, manganese, and cadmium are some of the heavy metal pollutants in the air that cause severe impacts on the biotic and abiotic environment. This study intended to find the accumulation capacity of the heavy metals on the leaves of tree species such as Terminalia catappa, Syzygium cumini, Saraca asoca, Pongamia glabra, and Ficus religiosa and predict their accuracy by comparing different machine learning (ML) models. The samples were collected at six different locations (likely Vellagate, Cancer Institute, CSI hospital area, Moongilmandapam, Collectorate, and Pallavarmedu) and distributed in a manner within Kanchipuram town, Tamil Nadu, in February and March of 2018 and 2019, respectively. Six ML methods were selected, such as KStar (K*), Lazy IKB, Logistic Regression Algorithm (LR), LogitBoost Classifier (LB), Meta Randomizable Filtered Classifier (MRFC), and Random Tree (RT), for prediction and to compare the efficiency of their predictions. Out of six models, Logistic functions perform well in terms of TP rate when compared to other classifiers (93.21%-99.81% TPR– 0.93–0.99) and Logitboost attained a low TP rate that ranged from 0.76 to 0.82. This study indicates the feasibility of different ML methods in the prediction of species capabilities toward the accumulation of heavy metals.

Recently, great progress has been achieved in diabetes management through minimally invasive continuous glucose monitoring, which involves piercing the skin to identify changes in glucose levels in the interstitial fluid. Nevertheless, the development of accurate fully noninvasive (NI) glucose monitors remains a topic of great interest. This work examines the processing of data collected from a multisensor system, which combines millimeter-wave (mm-wave), near-infrared (NIR), and other auxiliary sensors’ technology, coupled with a random forest machine learning (ML) model for prediction. We investigated possible correlations between mm-wave, NIR, skin temperature, pressure data, and blood glucose values and deployed an ML model for prediction. The model is based on the random forest algorithm that was trained and tested on data collected from a study involving five healthy human subjects undergoing an intravenous glucose tolerance test. Our model achieves a root mean square error of 21.06 mg/dl and a mean absolute relative difference of 7.31% for glucose prediction in the collected clinical subjects dataset. In addition, 96.1% of the model’s predictions fall within the clinically acceptable zones according to the Clarke error grid analysis. The preliminary results suggest that combining multiple sensor data with ML techniques is promising for the development of NI glucose monitoring.

Background and purpose — External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines. Material and methods — We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting. Results — We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43–89), with 6 items being reported in less than 4/18 of the studies. Interpretation — Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

Insider threat detection is challenging due to the wide variety of possible attacks and the limited availability of real threat data for testing. Most previous anomaly detection studies have relied on synthetic threat data, such as the CERT insider threat dataset. However, several previous studies have used models that arguably introduce bias, such as the selective use of metrics, and reusing the same dataset with the prior knowledge of the answer labels. In this paper, we create and test a host of models following some guidelines of good conduct to produce what we believe to be a more objective comparison of these models. Our results indicate that majority voting ensembles are a simple and cost-effective way of boosting the quality of results from individual machine learning models, both on the CERT data and on a version augmented with additional attacks. We include a comparison of models with their hyperparameters optimized for different target metrics.KeywordsAnomaly detectionInsider threatMachine learningEnsembles