Data, Machine Learning, and Human Domain Experts: None Is Better than Their Collaboration

Data reformation – A novel data processing technique enhancing machine learning applicability for predicting streamflow extremes

Transfer learning enables prediction of steel corrosion in concrete under natural environments

The rise in stable coronary artery disease (CAD) due to improved survival rates and population growth has increased patient numbers, straining healthcare systems. Machine learning (ML) models are being developed to predict and identify individual risk factors for early treatment, reducing harm to individuals and families. These models can predict hospitalizations, enable close monitoring of high-risk patients, and optimize medical care. Researchers are developing robust models based on ML algorithms and real-world clinical data to aid in early detection, contributing to AI research in healthcare. Advanced ML models analyze medical imaging, genetic markers, lifestyle, and environmental factors to accurately predict coronary heart disease (CHD) start and progression. Our research introduces four novel models based on two-class Logistic Regression (two-class LR), two-class Neural Network (two-class NN), two-class Decision Jungle (two-class DJ), and two-class Boosted DT (two-class BDT). Our comparative analysis reveals that the two-class Boosted DT model is the most effective, achieving an AUC score of 0.991. This model excels in real-time monitoring by predicting minor changes in patient’s health markers, allowing for timely adjustments in treatment plans. It optimizes medication selection, dosing, and intervention timing based on patient characteristics, improving therapeutic efficacy and reducing side effects. The study reveals the transformative potential of these advanced ML models in CAD prediction and management. By focusing on feature selection, algorithm improvement, and integration, our models analyze medical imaging, genetic markers, lifestyle, and environmental factors to accurately predict the onset and progression of CHD. This research proposes valuable insights into the capabilities of these models to revolutionize disease detection and management, ensuring reliable and timely healthcare interventions across various datasets.

BackgroundWhile machine learning (ML) models are increasingly used to predict outcomes in health care, their practical effect on health care operations, such as bed capacity management, remains underexplored. There is a variety of traditionally used evaluation metrics to analyze ML models; however, decision makers in health care settings require a deeper understanding of their implications for resource management. Traditional performance measures often fail to provide this practical insight.MethodsIn this work, we conduct a simulation study to evaluate the impact of ML-driven length-of-stay (LOS) predictions on intensive care unit (ICU) bed capacity management. Two classification models differing in terms of explainability and interpretability, logistic regression (LR) and extreme gradient boosting (XGB), are applied to predict ICU-LOS. We use the HiRID dataset containing high-frequency data of more than 33,000 patients. The predictions of the ML models are integrated into a simulation framework that replicates real-world ICU bed management, allowing for the assessment of the practical implications of using these algorithms in a clinical setting.ResultsThe application of both classification models results in improved capacity control regarding the key performance indicators in the simulation study, with XGB outperforming LR. While LR leads to slight overoccupancy in the ICU, slight underoccupancy can be observed when XGB is applied.ConclusionOur study bridges the gap between predictive accuracy and practical application by emphasizing the importance of evaluating ML models within the context of ICU capacity management. The simulation-based approach offers a more relevant assessment for health care practitioners, providing actionable insights that go beyond classical performance measures and directly address the needs of decision makers in clinical practice.HighlightsWe apply multiple classification models for ICU-LOS prediction using time-series data. This approach enables an update of the initial prediction resulting in the possibility of efficiently managing intensive care capacities.We present a simulation-based approach to evaluate ML algorithms and their impact on bed capacity management in real-world clinical settings.Our work provides in-depth insights into the impact of using ML techniques as decision support systems in the ICU and can lead to increased acceptance in practice.

Machine Learning (ML) models have been applied to solve problems in various fields, which necessarily involves proper evaluation of models to ensure performance. Once deployed, ML models are subject to performance issues, such as those related to changes in data (drift). This type of issue has prompted efforts in model analysis and maintenance, as well as in continual learning, which seeks the ability to continuously learn from a (continuous) stream of data. Therefore, it's important to understand and develop methodologies that can be used to evaluate ML models, making their use in real-world environments feasible. Amongst current areas of application for ML, one that stands out, in particular, is Machine Learning for Healthcare, especially in conjunction with Software for Decision Support of Medical Applications, which presents specific challenges for the evaluation and monitoring of models, particularly given that incorrect prediction or classification can lead to life-threatening situations. This paper presents a systematic literature review that aims at identifying state-of-the-art techniques for evaluating and maintaining ML models for healthcare in effective use in the real world.

Background: Despite the recent success of genome-wide association studies (GWAS) in identifying 90 independent risk loci for Parkinson’s disease (PD), the genomic underpinning of PD is still largely unknown. At the same time, accurate and reliable predictive models utilizing genomic or demographic features are desired in the clinic for predicting the risk of Parkinson’s disease.Methods: To identify influential demographic and genomic factors associated with PD and to further develop predictive models, we utilized demographic data, incorporating 200 variables across 33,473 participants, along with genomic data involving 447,089 SNPs across 8,840 samples, both derived from the Fox Insight online study. We first applied correlation and GWAS analyses to find the top demographic and genomic factors associated with PD, respectively. We further developed and compared a variety of machine learning (ML) models for predicting PD. From the developed ML models, we performed feature importance analysis to reveal the predictability of each demographic or the genomic input feature for PD. Finally, we performed gene set enrichment analysis on our GWAS results to identify PD-associated pathways.Results: In our study, we identified both novel and well-known demographic and genetic factors (along with the enriched pathways) related to PD. In addition, we developed predictive models that performed robustly, with AUC = 0.89 for demographic data and AUC = 0.74 for genomic data. Our GWAS analysis identified several novel and significant variants and gene loci, including three intron variants in LMNA (p-values smaller than 4.0e-21) and one missense variant in SEMA4A (p-value = 1.11e-26). Our feature importance analysis from the PD-predictive ML models highlighted some significant and novel variants from our GWAS analysis (e.g., the intron variant rs1749409 in the RIT1 gene) and helped identify potentially causative variants that were missed by GWAS, such as rs11264300, a missense variant in the gene DCST1, and rs11584630, an intron variant in the gene KCNN3.Conclusion: In summary, by combining a GWAS with advanced machine learning models, we identified both known and novel demographic and genomic factors as well as built well-performing ML models for predicting Parkinson’s disease.

Multiple sclerosis (MS) is a heterogeneous immune-mediated disorder of the central nervous system, and primary progressive MS (PPMS) is characterized by insidious injury independent of clinical relapses from disease onset. Predicting disease progression remains an unmet need across the MS disease spectrum. We evaluated statistical and machine learning (ML) models for predicting progression in 596 patients with PPMS from ORATORIO, a randomized, placebo (PBO)-controlled, phase 3 clinical study of ocrelizumab (OCR). Using baseline clinical, magnetic resonance imaging, and biomarker (neurofilament light chain [NfL]) data, we trained 6 ML models to predict a progression event (logistic regression, support vector machines, random forest, neural network, gradient boosting, and extreme gradient boosting) and the time to progression (Cox proportional hazards [PH]) on upper limb function, as measured by 20% confirmed worsening on the 9-hole peg test (9HPT). Predictive performance was similar across the six ML models (area under the curve = 0.63 to 0.67) and was modest using the Cox PH model (C-index = 0.76, an improvement over previous studies). In the Cox PH model, treatment arm, baseline 9HPT, neurologic functional system scores, and serum NfL levels were identified as factors associated with progression; model performance was similar across the OCR, PBO, and combined OCR + PBO treatment arms. Accounting for potential interactions with treatment arm resulted in similar predictive performance. To our knowledge, this is the first study to systematically evaluate ML models and statistical methods for predicting progression in PPMS using a well-characterized clinical trial cohort.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

The ability to accurately predict an individual's risk for cancer is critical to the implementation of precision prevention measures. Current cancer risk predictions are frequently made with simple models that use a few proven risk factors, such as the Gail model for breast cancer, which are easy to interpret, but may theoretically be less accurate than advanced machine learning (ML) models. With the UK Biobank, a large prospective study, we developed models that predicted 13 cancer diagnoses within a 10-year time span. ML and linear models fit with all features, linear models fit with 10 features, and externally developed QCancer models, which are available to more than 4,000 general practices, were assessed. The average area under the receiver operator curve (AUC) of the linear models (0.722, SE = 0.015) was greater than the average AUC of the ML models (0.720, SE = 0.016) when all 931 features were used. Linear models with only 10 features generated an average AUC of 0.706 (SE 0.015), which was comparable to the complex models using all features and greater than the average AUC of the QCancer models (0.684, SE 0.021). The high performance of the 10-feature linear model may be caused by the consideration of often omitted feature types, including census records and genetic information. The high performance of the 10-feature linear models indicate that unbiased selection of diverse features, not ML models, may lead to impressively accurate predictions, possibly enabling personalized screening schedules that increase cancer survival.

This study focuses on using various machine learning (ML) models to evaluate the shear behaviors of ultra-high-performance concrete (UHPC) beams reinforced with glass fiber-reinforced polymer (GFRP) bars. The main objective of the study is to predict the shear strength of UHPC beams reinforced with GFRP bars using ML models. We use four different ML models: support vector machine (SVM), artificial neural network (ANN), random forest (R.F.), and extreme gradient boosting (XGBoost). The experimental database used in the study is acquired from various literature sources and comprises 54 test observations with 11 input features. These input features are likely parameters related to the composition, geometry, and properties of the UHPC beams and GFRP bars. To ensure the ML models' generalizability and scalability, random search methods are utilized to tune the hyperparameters of the algorithms. This tuning process helps improve the performance of the models when predicting the shear strength. The study uses the ACI318M-14 and Eurocode 2 standard building codes to predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams. The ML models' predictions are compared to the results obtained from these building code standards. According to the findings, the XGBoost model demonstrates the highest predictive test performance among the investigated ML models. The study employs the SHAP (SHapley Additive exPlanations) analysis to assess the significance of each input parameter in the ML models' predictive capabilities. A Taylor diagram is used to statistically compare the accuracy of the ML models. This study concludes that ML models, particularly XGBoost, can effectively predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams.

Machine learning (ML) algorithms are increasingly used to predict outcomes in orthopaedic surgery, but their utility in hip arthroscopy remains unclear. This study aimed to (1) evaluate ML models for predicting outcomes, complications, and resource use following hip arthroscopy, (2) compare ML performance to traditional statistical methods, and (3) critically appraise the quality of existing literature. A systematic search of PUBMED, MEDLINE, EMBASE and the Cochrane Central Register of Controlled Trials was conducted on 4 April 2023. Included studies used ML models for clinical prediction in hip arthroscopy. Data on study design, outcomes, model development, validation and adherence to TRIPOD guidelines were extracted. Seventeen studies involving 37,549 patients (52.8% female; mean age 34.4 ± 6.1 years) were included. ML was used to predict clinical outcomes (47.1%), adverse events (47.1%) and resource utilisation (5.9%). The median area under the curve was 0.66 (range 0.5-0.94) for clinical outcomes and 0.7 (range 0.6-0.9) for adverse events. Resource utilisation was reported with a root-mean-square error of $3800, a logarithmic error of 0.2, and R² of 0.8. Only four studies compared ML to traditional regression; two found no difference, while two favoured ML. On average, studies met 12.3 of the 22 TRIPOD criteria, with only 29.4% meeting more than two-thirds. ML models in hip arthroscopy show variable performance in predicting outcomes and complications. Furthermore, most studies comparing ML models to traditional regression did not report any significant difference. Lastly, variability in adherence to TRIPOD guidelines highlights the need for future studies to improve transparency and reporting in the development of ML algorithms for hip arthroscopy. Given the potential of ML models in enhancing clinical decision-making for hip arthroscopy, along with the findings of the current study, their application should be used with caution. Level IV, systematic review of level II-IV studies.

SummarySustained engagement in HIV care and adherence to antiretroviral therapy (ART) are essential for achieving the UNAIDS “95-95-95” targets. Despite increased ART access, disengagement from care remains a significant issue, particularly in sub-Saharan Africa. Traditional machine learning (ML) models have shown moderate success in predicting care disengagement, which would enable early intervention. We develop an enhanced large language model (LLM) fine-tuned with electronic medical records (EMRs) to predict people at risk of disengaging from HIV care in Tanzania and to provide interpretative insights into modifiable risk factors.MethodsWe developed a novel AI model by enhancing a pre-trained LLM (LLaMA 3.1, an open-source pre-trained LLM released by Meta) using routinely collected EMRs from Tanzania’s National HIV Care and Treatment Program from January 1, 2018, to June 30, 2023 (4,809,765 records for 261,192 people) to identify people at risk of disengaging from HIV care or developing adverse outcomes. Outcomes included risk of ART non-adherence, non-suppressed viral load, and loss to follow-up. Models were evaluated internally (Kagera region) and externally (Geita region), with performance compared against state-of-art ML models and zero-shot LLMs. Additionally, a team of HIV physicians in Tanzania assessed the LLM’s predictions along with LLM provided justifications for a subset of patient records to evaluate their clinical relevance and reasoning.FindingsThe enhanced LLMs consistently outperformed the supervised ML model and zero-shot LLMs across all outcomes in both internal and external validation datasets. When focusing on the 25% of PLHIV predicted as most likely to lost-to-follow-up (LTFU), the model correctly identified 78% (2,515 of 3,224) of people living with HIV (PLHIV) genuinely at risk in internal validation and 73% (7,105 of 9,733) in external validation. Attention score analysis indicated that the enhanced LLM focused on keywords such as gaps in follow-up care and ART adherence. The human expert evaluation showed alignment between clinician assessments and the LLM’s predictions in 65% of cases, with experts finding the model’s justifications reasonable and clinically relevant in 92.3% of aligned cases.InterpretationIf implemented in HIV clinics, this LLM-based AI model could help allocate resources efficiently and deliver targeted interventions, improving retention in care and advancing the UNAIDS “95-95-95” targets. By functioning like a clinician–analyzing patient summaries, predicting risks, and offering reasoning–the enhanced LLM could be integrated into clinical workflows to complement human expertise, facilitating timely interventions and informed decision-making. If implemented widely, this human-AI collaboration has the potential to improve health outcomes for people living with HIV and reduce onward transmission.FundingThe study was supported by a grant from the US National Institutes of Health (NIH): NIH NIMH 1R01MH125746.

Systematic literature review of machine learning based software development effort estimation models

Accurate estimation of reference evapotranspiration (ETo) plays a vital role in irrigation and water resource planning. The Penman–Monteith method recommended by the Food and Agriculture Organization (FAO PM56) is widely used and considered a standard to calculate ETo. However, FAO PM56 cannot be used with limited meteorological variables, so it is compulsory to choose an alternative model for ETo estimation, which requires fewer variables. This study built ten machine learning (ML) models based on multi-function, neural network, and tree-based structure against the FAO PM56 method. For this purpose, time series temperature data on a monthly scale are only used to train ML models. The developed ML models were applied to estimate ETo at different test stations and the obtained results were compared with the FAO PM56 method to verify and validate their performance in ETo estimation for the selected stations. In addition, multiple statistical indicators, including root-mean-square error (RMSE), coefficient of determination (R2), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), and correlation coefficient (r) were calculated to compare the performance of each ML model on ETo estimation. Among the applied ML models, the ETo tree boost (TB) ML model outperformed the other ML models in estimating ETo in diverse climatic conditions based on statistical indicators (R2, NSE, r, RMSE, and MAE). Moreover, the observed R2, NSE, and r were the highest for the TB ML model, while RMSE and MAE were found to be the lowest at the study sites compared to other applied ML models. Lastly, ETo point data yielded from the TB ML model was used in an interpolation process to create monthly and annual ETo maps. Based on the ETo maps, this study suggests mainly a focus on areas with high ETo values and proper irrigation scheduling of crops to ensure water sustainability.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.