The human–nature relationship across two centuries: macro factor insights from a machine learning model

Background and purpose — External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines. Material and methods — We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting. Results — We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43–89), with 6 items being reported in less than 4/18 of the studies. Interpretation — Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

Background and aim Direct oral anticoagulants (DOACs) carry a risk of gastrointestinal bleeding (GIB). We aimed to develop and validate machine learning (ML) models to predict GIB in DOAC users and compare them with conventional risk scores. Methods We retrospectively analyzed 4,494 patients aged ≥18 years prescribed DOACs from December 2014 to October 2020. Patients were allocated to the training (n = 3,147), internal (n = 677), and external (n = 670) validation cohorts. Three ML algorithms, Gradient Boosting Machine (GBM), XGBoost, and Generalized Linear Model (GLM), predicted GIB at 12 and 24 months. Performance was assessed using the area under the receiver operating characteristic curve (AUC) and specificity at 100% sensitivity, compared with the HAS-BLED, ATRIA, VTE-BLEED, and ORBIT scores. Results At 24 months, XGBoost achieved the AUCs in the training (0.862), internal validation (0.819), and external validation (0.905) sets. At 12 months, XGBoost performed with AUCs of 0.917, 0.839, and 0.948, respectively. XGBoost exceeded the conventional scores, although ORBIT was the best among the latter (AUC 0.780 at 24 months, 0.728 at 12 months). The ML models also achieved higher specificity at 100% sensitivity. At 12 months, XGBoost and GB model demonstrated 79.8% specificity at 100% sensitivity, whereas GLM showed 67.8%. The conventional models were lower, with an ORBIT of 39.8%. By 24 months, GLM and ORBIT specificities were 43.8% and 40.0%, respectively. Conclusions ML models, particularly XGBoost, outperformed traditional bleeding risk scores in predicting GIB in DOAC users. However, the performance of the ML models was unsatisfactory. Further research is warranted to achieve a better performance.

In the era of precision therapy, early classification of breast cancer (BRCA) molecular subtypes has clinical significance for disease management and prognosis. We explored the accuracy of machine learning (ML) models for early classification of BRCA molecular subtypes through a systematic review of the literature currently available. We retrieved relevant studies published in PubMed, EMBASE, Cochrane, and Web of Science until 15 April 2022. A prediction model risk of bias assessment tool (PROBAST) was applied for the assessment of risk of bias of a genomics-based ML model, and the Radiomics Quality Score (RQS) was simultaneously used to evaluate the quality of this radiomics-based ML model. A random effects model was adopted to analyze the predictive accuracy of genomics-based ML and radiomics-based ML for Luminal A, Luminal B, Basal-like or triple-negative breast cancer (TNBC), and human epidermal growth factor receptor 2 (HER2). The PROSPERO of our study was prospectively registered (CRD42022333611). Of the 38 studies were selected for analysis, 14 ML models were based on gene-transcriptomic, with only 4 external validations; and 43 ML models were based on radiomics, with only 14 external validations. Meta-analysis results showed that c-statistic values of the ML based on radiomics for the identification of BRCA molecular subtypes Luminal A, Luminal B, Basal-like or TNBC, and HER2 were 0.76 [95% confidence interval (CI): 0.60-0.96], 0.78 (95% CI: 0.69-0.87), 0.89 (95% CI: 0.83-0.91), and 0.83 (95% CI: 0.81-0.86), respectively. The c-statistic values of ML based on the gene-transcriptomic analysis cohort for the identification of the previously described BRCA molecular subtypes were 0.96 (95% CI: 0.93-0.99), 0.96 (95% CI: 0.93-0.99), 0.98 (95% CI: 0.95-1.00), and 0.97 (95% CI: 0.96-0.98) respectively. Additionally, the sensitivity of the ML model based on radiomics for each molecular subtype ranged from 0.79 to 0.85, while the sensitivity of the ML model based on gene-transcriptomic was between 0.92 and 0.99. Both radiomics and gene transcriptomics produced ideal effects on BRCA molecular subtype prediction. Compared with radiomics, gene transcriptomics yielded better prediction results, but radiomics was simpler and more convenient from a clinical point of view.

A machine learning (ML) model improved the prediction of pulmonary hypertension due to left heart disease (PH-LHD) in suspected pulmonary arterial hypertension (PAH) subjects using non-invasive predictors. (<i>K. Swinnen et al.</i>, ERS 2021) We aimed to externally validate this model to assess its generalizability to different populations. The ML model was developed using non-invasive predictors from 344 subjects from the University Hospitals of Leuven; demographics, medical history, echocardiographic, ECG and lung function parameters, and lab results were extracted from patient files. During internal validation (n=104), the ML model outperformed the Jacobs score (<i>W. Jacobs et al.</i>, ERJ 2015) with sensitivity 70%, specificity 100%, positive predictive value (PPV) 100% and negative predictive value (NPV) 78%. The external validation was performed in a cohort of 165 subjects (91 PAH and 74 PH-LHD) from Erasme Hospital Brussels. The ML model achieved similar performances (sensitivity 64%, specificity 100%, PPV 100% and NPV 78%) during external validation. Of the 74 PH-LHD subjects, 47 were correctly identified non-invasively without false positives <i>versus</i> 2 with the Jacobs score. The cohorts had significant differences in baseline characteristics, including important parameters contributing to the model predictions. Despite differences between populations, the ML model showed good generalizability when tested on subjects from a different centre with only a slight decrease in sensitivity. This model is ready to be tested in clinical practice and could substantially reduce the number of right heart catheterizations in patients with high probability for PH-LHD.

BackgroundSarcopenia is a common muscle disorder in older adults, and its early identification and management in middle-aged populations are essential for ensuring a healthier later life. Detecting sarcopenia at an earlier stage may reduce the future burden on health care systems and enhance the quality of life in older adults. Machine learning (ML) models can evaluate large datasets, identify essential variables, and find complicated correlations between input variables. However, using ML models to detect sarcopenia remains an unsatisfied need.ObjectiveThis study aimed to develop and externally validate an ML model to predict sarcopenia risk among middle-aged adults using a nationally representative dataset.MethodsWe analyzed data from 1926 participants aged 40 to 64 years and enrolled in the 2022 Korea National Health and Nutrition Examination Survey (KNHANES). Sarcopenia was diagnosed and defined based on the 2019 Asian Working Group for Sarcopenia criteria, which incorporate both low muscle mass and reduced muscle strength. Muscle mass was assessed using bioelectrical impedance analysis with cutoffs of <7.0 kg/m² for men and <5.7 kg/m² for women. Muscle strength was measured via handgrip strength using a digital dynamometer with thresholds of <28 kg for men and <18 kg for women. Participants meeting both criteria were classified as those with sarcopenia. Four ML algorithms, random forest, support vector machine, extreme gradient boosting, and logistic regression, were used to identify risk factors of sarcopenia and predict its likelihood. The top-performing model was subsequently validated in an external cohort of 2247 middle-aged adults from the 2023 KNHANES. Model performance was assessed using the F2-score, area under the curve of a receiver operating characteristic curve, and sensitivity. All analyses were conducted using Python 3.13.2 (Python Software Foundation).ResultsAmong the 4 models, the logistic regression model demonstrated the strongest performance, yielding an area under the curve of 0.85, a sensitivity of 0.92, and an F2-score of 0.66. External validation using the 2023 KNHANES dataset confirmed the model’s robust performance, indicating its potential for widespread applications.ConclusionsThis study developed and externally validated an ML model that accurately identified sarcopenia in middle-aged adults. Leveraging data from a comprehensive national survey, our findings underscore the significance of early detection and customized interventions in midlife to mitigate sarcopenia risk and optimize long-term health outcomes.

Estimating whether to treat the rupture risk of small intracranial aneurysms (IAs) with size ≤ 7mm in diameter is difficult but crucial. We aimed to construct and externally validate a convenient machine learning (ML) model for assessing the rupture risk of small IAs. One thousand four patients with small IAs recruited from two hospitals were included in our retrospective research. The patients at hospital 1 were stratified into training (70%) and internal validation set (30%) randomly, and the patients at hospital 2 were used for external validation. We selected predictive features using the least absolute shrinkage and selection operator (LASSO) method and constructed five ML models applying diverse algorithms including random forest classifier (RFC), categorical boosting (CatBoost), support vector machine (SVM) with linear kernel, light gradient boosting machine (LightGBM), and extreme gradient boosting (XGBoost). The Shapley Additive Explanations (SHAP) analysis provided interpretation for the best ML model. The training, internal, and external validation cohorts included 658, 282, and 64 IAs, respectively. The best performance was presented by SVM as AUC of 0.817 in the internal [95% confidence interval (CI), 0.769-0.866] and 0.893 in the external (95% CI, 0.808-0.979) validation cohorts, which overperformed compared with the PHASES score significantly (all P < 0.001). SHAP analysis showed maximum size, location, and irregular shape were the top three important features to predict rupture. Our SVM model based on readily accessible features presented satisfying ability of discrimination in predicting the rupture IAs with small size. Morphological parameters made important contributions to prediction result.

To evaluate the diagnostic performance of Machine Learning (ML) and Deep Learning (DL) models for predicting preoperative Lymph Node Metastasis (LNM) in Colorectal Cancer (CRC) patients. A systematic review and meta-analysis were conducted following PRISMA-DTA and AMSTAR-2 guidelines. We searched PubMed, Web of Science, Embase, and Cochrane Library databases until February 16, 2024. Study quality and risk of bias were assessed using the QUADAS-2 tool. Data were analyzed using STATA v18, applying random-effects models to all analyses. Twelve studies involving 8321 patients were included, with most published in 2021-2024 (9/12). The pooled AUC of ML models for predicting LNM in CRC patients was 0.87 (95% CI: 0.82-0.91, I2:86.17) with a sensitivity of 78% (95% CI: 69-87%) and a specificity of 77% (95% CI: 64%-90%). In addition, when assessing the AUC reported by radiologists, both junior and senior radiologists had similar performance, significantly lower than the ML models. (P < 0.001). Subgroup analysis revealed higher AUCs in prospective studies (0.95, 95% CI: 0.87-1) compared to retrospective studies (0.85, 95% CI: 0.81-0.89) (P = 0.03). Studies without external validation exhibited significantly higher AUCs than those with external validation (P < 0.01). While there was no significant difference in AUC and sensitivity between the T1-T2 and T2-T4 stages, specificity was significantly higher in the T2-T4 stages than the low stages of T1 and T2 (95%, 95% CI: 92-98% vs. 61%, 95% CI: 44-78%; P < 0.01). ML models demonstrate strong potential for preoperative LNM staging and treatment planning in CRC, potentially reducing the need for additional surgeries and related health and financial burdens. Further prospective multicenter studies, with standardized reporting of algorithms, modality parameters, and LNM staging, are needed to validate these findings.

BackgroundEarly identification of patients with acute hypoxemic respiratory failure (AHRF) who are at risk of failing high-flow nasal cannula (HFNC) therapy could facilitate closer monitoring, and timely adjustment/escalation of treatment. We aimed to establish whether machine learning (ML) models could predict HFNC outcome, early in the course of treatment, with greater accuracy than currently used clinical indices.MethodsWe developed ML models trained using measurements made within the first 2 h of treatment from 184 AHRF patients (37% HFNC failures) treated at the respiratory ICU of the University Hospital of Modena between 2018 and 2023. For external validation, we used a dataset on 567 AHRF patients (22% failures) comprising 510 patients from the recent RENOVATE trial in Brazil and 57 from the MIMIC-IV and eICU databases in the US. Predictive performance of the ML models was benchmarked against optimized thresholds of the following clinical indices: respiratory rate oxygenation index (ROX) and variants, heart rate to saturation of pulse oxygen (SpO2) ratio, SpO2/FiO2 ratio, PaO2/FiO2 ratio, sequential organ failure assessment and heart rate, acidosis, consciousness, oxygenation and respiratory rate scores.ResultsInternal and external predictive performance of a Support Vector Machine (SVM) ML model was superior to all clinical indices across all scenarios tested. In external validation on the 567-patient dataset, a SVM model trained on non-invasive measurements had an accuracy of 73%, sensitivity of 73%, specificity of 73%, and AUC of 0.79. The ROX index had an accuracy of 64%, sensitivity of 79%, specificity of 60%, and AUC of 0.74. When arterial blood gasses (ABG’s) were also used for model training, the SVM model had an accuracy of 83%, sensitivity of 84%, specificity of 82%, and AUC of 0.82 in external validation on the MIMIC-IV/eICU dataset. The modified ROX index, which requires PaO2, achieved 70% accuracy, 63% sensitivity, 74% specificity, and AUC of 0.65.ConclusionsDecision support tools based on SVM models could provide clinicians with more accurate early predictions of HFNC outcome than currently available clinical indices. If available, ABG measurements could improve the capability to accurately identify patients at risk of failing HFNC therapy.

Development and validation of an explainable machine learning model for mortality prediction among patients with infected pancreatic necrosis.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Propofol is a widely used sedative-hypnotic agent for critically-ill patients requiring invasive mechanical ventilation (IMV). Despite its clinical benefits, propofol is associated with increased risks of hypertriglyceridemia. Early identification of patients at risk for propofol-associated hypertriglyceridemia is crucial for optimizing sedation strategies and preventing adverse outcomes. Machine learning (ML) models offer a promising approach for predicting individualized patient risks of propofol-associated hypertriglyceridemia. We propose the development of a ML model aimed at predicting the risk of propofol-associated hypertriglyceridemia in ICU patients receiving IMV. The study will utilize retrospective data from four Mayo Clinic sites. Nested cross-validation (CV) will be employed, with a 10-fold inner CV loop for model tuning and selection as well as an outer loop using leave-one-site-out CV for external validation. Feature selection will be conducted using Boruta and LASSO-penalized logistic regression. Data preprocessing steps include missing data imputation, feature scaling, and dimensionality reduction techniques. Six ML algorithms will be tuned and evaluated. Bayesian optimization will be used for hyperparameter selection. Global model explainability will be assessed using permutation importance, and local model explainability will be assessed using SHapley Additive exPlanations (SHAP). The proposed ML model aims to provide a reliable and interpretable tool for clinicians to predict the risk of propofol-associated hypertriglyceridemia in ICU patients. The final model will be deployed in a web-based clinical risk calculator. The model development process and performance measures obtained during nested cross-validation will be described in a study publication to be disseminated in a peer-reviewed journal. The proposed study has received ethics approval from the Mayo Clinic Institutional Review Board (IRB #23-007416). Robust external validation using a nested cross-validation (CV) framework will help assess the generalizability of models produced from the modeling pipeline across different hospital settings.A diverse set of machine learning (ML) algorithms and advanced hyperparameter tuning techniques will be employed to identify the most optimal model configuration.Integration of feature explainability will enhance the clinical applicability of the ML models by providing transparency in predictions, which can improve clinician trust and encourage adoption.Reliance on retrospective data may introduce biases due to inconsistent or erroneous data collection, and the computational intensity of the validation approach may limit replication and future model expansion in resource-constrained settings.

This study focuses on using various machine learning (ML) models to evaluate the shear behaviors of ultra-high-performance concrete (UHPC) beams reinforced with glass fiber-reinforced polymer (GFRP) bars. The main objective of the study is to predict the shear strength of UHPC beams reinforced with GFRP bars using ML models. We use four different ML models: support vector machine (SVM), artificial neural network (ANN), random forest (R.F.), and extreme gradient boosting (XGBoost). The experimental database used in the study is acquired from various literature sources and comprises 54 test observations with 11 input features. These input features are likely parameters related to the composition, geometry, and properties of the UHPC beams and GFRP bars. To ensure the ML models' generalizability and scalability, random search methods are utilized to tune the hyperparameters of the algorithms. This tuning process helps improve the performance of the models when predicting the shear strength. The study uses the ACI318M-14 and Eurocode 2 standard building codes to predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams. The ML models' predictions are compared to the results obtained from these building code standards. According to the findings, the XGBoost model demonstrates the highest predictive test performance among the investigated ML models. The study employs the SHAP (SHapley Additive exPlanations) analysis to assess the significance of each input parameter in the ML models' predictive capabilities. A Taylor diagram is used to statistically compare the accuracy of the ML models. This study concludes that ML models, particularly XGBoost, can effectively predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams.

Systematic literature review of machine learning based software development effort estimation models

Accurate estimation of reference evapotranspiration (ETo) plays a vital role in irrigation and water resource planning. The Penman–Monteith method recommended by the Food and Agriculture Organization (FAO PM56) is widely used and considered a standard to calculate ETo. However, FAO PM56 cannot be used with limited meteorological variables, so it is compulsory to choose an alternative model for ETo estimation, which requires fewer variables. This study built ten machine learning (ML) models based on multi-function, neural network, and tree-based structure against the FAO PM56 method. For this purpose, time series temperature data on a monthly scale are only used to train ML models. The developed ML models were applied to estimate ETo at different test stations and the obtained results were compared with the FAO PM56 method to verify and validate their performance in ETo estimation for the selected stations. In addition, multiple statistical indicators, including root-mean-square error (RMSE), coefficient of determination (R2), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), and correlation coefficient (r) were calculated to compare the performance of each ML model on ETo estimation. Among the applied ML models, the ETo tree boost (TB) ML model outperformed the other ML models in estimating ETo in diverse climatic conditions based on statistical indicators (R2, NSE, r, RMSE, and MAE). Moreover, the observed R2, NSE, and r were the highest for the TB ML model, while RMSE and MAE were found to be the lowest at the study sites compared to other applied ML models. Lastly, ETo point data yielded from the TB ML model was used in an interpolation process to create monthly and annual ETo maps. Based on the ETo maps, this study suggests mainly a focus on areas with high ETo values and proper irrigation scheduling of crops to ensure water sustainability.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.