Fair and optimal prediction via post‐processing

Interpretable vs. noninterpretable machine learning models for data-driven hydro-climatological process modeling

Federated learning (FL) has arisen as an alternative to collecting large amounts of data in a central place to train a machine learning (ML) model. FL is privacy-friendly, allowing multiple parties to collaboratively train an ML model without exchanging or transmitting their training data. For this purpose, an aggregator iteratively coordinates the training process among parties, and parties simply share with the aggregator model updates, which contain information pertinent to the model such as neural network weights. Besides privacy, generalization has been another key driver for FL: parties who do not have enough data to train a good performing model by themselves can now engage in FL to obtain an ML model suitable for their tasks. Products and real applications in the industry and consumer space have demonstrated the power of this learning paradigm. Recently, foundation models have taken the AI community by storm, promising to solve the shortage of labeled data. A foundation model is a powerful model that can be recycled for a variety of use cases by applying techniques such as zero-shot learning and full or parameter-efficient fine tuning. The premise is that the amount of data required to fine tune a foundation model for a new task is much smaller than fully training a traditional model from scratch. The reason why this is the case is that a good foundation model has already learned relevant general representations, and thus, adapting it to a new task only requires a minimal number of additional samples. This raises the question: Is FL still alive in the era of foundation models? In this talk, I will address this question. I will present some use cases where FL is very much alive. In these use cases, finding a foundation model with a desired representation is difficult if not impossible. With this pragmatic point of view, I hope to shed some light into a real use case where disparate private data is available in isolation at different parties and where labels may be located at a single party that doesn’t have any other information, making it impossible for a single party to train a model on its own. Furthermore, in some vertically-partitioned scenarios, cleaning data is not an option due to privacy-related reasons and it is not clear how to apply foundation models. Finally, I will also go over a few other requirements that are often overlooked, such as unlearning of data and its implications for the lifecycle management of FL and systems based on foundation models.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

PD27-01 DEVELOPMENT OF A MACHINE LEARNING (ML) MODEL TO AUTOMATICALLY AND PRECISELY IDENTIFY KIDNEY STONES FROM URETEROSCOPY VIDEO RECORDINGS

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.

Accurate estimation of reference evapotranspiration (ETo) plays a vital role in irrigation and water resource planning. The Penman–Monteith method recommended by the Food and Agriculture Organization (FAO PM56) is widely used and considered a standard to calculate ETo. However, FAO PM56 cannot be used with limited meteorological variables, so it is compulsory to choose an alternative model for ETo estimation, which requires fewer variables. This study built ten machine learning (ML) models based on multi-function, neural network, and tree-based structure against the FAO PM56 method. For this purpose, time series temperature data on a monthly scale are only used to train ML models. The developed ML models were applied to estimate ETo at different test stations and the obtained results were compared with the FAO PM56 method to verify and validate their performance in ETo estimation for the selected stations. In addition, multiple statistical indicators, including root-mean-square error (RMSE), coefficient of determination (R2), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), and correlation coefficient (r) were calculated to compare the performance of each ML model on ETo estimation. Among the applied ML models, the ETo tree boost (TB) ML model outperformed the other ML models in estimating ETo in diverse climatic conditions based on statistical indicators (R2, NSE, r, RMSE, and MAE). Moreover, the observed R2, NSE, and r were the highest for the TB ML model, while RMSE and MAE were found to be the lowest at the study sites compared to other applied ML models. Lastly, ETo point data yielded from the TB ML model was used in an interpolation process to create monthly and annual ETo maps. Based on the ETo maps, this study suggests mainly a focus on areas with high ETo values and proper irrigation scheduling of crops to ensure water sustainability.

12074 Background: Skeletal-related events (SREs) are detrimental clinical events in bone metastasis from advanced cancer, defined by pathologic fracture, spinal cord compression, and inevitable surgical or radiational intervention to the bone. Given their negative impact on quality of life and prognosis and interpatient heterogeneity in the SRE risk, accurate identification of patients with high SRE risk is critical. Methods: The patient-level data from three randomized clinical trials that administered zoledronic acid (ZA) to patients with bone-metastatic breast cancer, castration-resistant prostate cancer (CRPC), and other types of cancer were analyzed. (N = 460, 452, and 315, respectively) Machine learning (ML) models to predict SREs within 18 months (breast cancer), 12 months (CRPC), and 9 months (other cancers) were developed based on more than 40 baseline clinical and laboratory data. Seven ML algorithms and five feature selection methods were utilized to develop multiple models. The ML model with the best performance was identified based on the F1 score and the area under the receiver operating characteristic curve (AUC-ROC) for each cancer type and interrogated for important features with Shapley additive explanations. Lastly, the ML models’ ability to stratify patients by the cumulative SRE risk was evaluated by calculating hazard ratio (HR) with Cox-proportional hazards models. Results: Among the multiple ML models developed with different algorithms and feature selection methods, the model developed utilizing the random forests algorithm and the Boruta method for selecting features demonstrated the best performance in all types of cancer (F1 0.70, 0.67, and 0.67, and AUC-ROC 0.72, 0.68, and 0.73 for breast cancer, CRPC, and other cancers, respectively). In the ML model for breast cancer, performance status (PS), history of SRE, serum alkaline phosphatase (ALP), history of anti-neoplastic surgery, radiation therapy, and pathologic fracture were included as important features. Serum ALP, albumin, sodium, Gleason scores, and geographic regions were shown to be relevant in the CRPC model. For other cancers, serum ALP, albumin, total protein, phosphorus, red blood cell count, white blood cell count, visceral metastases, and history of arthritis were incorporated in the ML model. The ML model prediction successfully stratified the patients for cumulative SRE risk in all three cohorts. (HR with 95% confidence interval: 2.43 [1.86 – 3.18], 1.92 [1.51 – 2.45], and 3.06 [2.29 – 4.09] for breast cancer, CRPC, and other types of cancer, respectively). Conclusions: ML models incorporating baseline clinical and laboratory data can identify patients with bone metastasis on ZA harboring a high SRE risk.

This paper analyzes the optimization features of machine learning (ML) model training procedures using multi-GPU systems to enhance cyber security in telecommunication networks. A key aspect of the study is the use of data parallelism, which allows the distribution of the training load across multiple GPUs, significantly reducing training time and improving model accuracy-critical factors for rapid threat detection in cyberspace. A novel approach for optimizing data batch size using Mutual Information (MI) is proposed, which harmonizes the utilization of computational resources with the information content of the training data. MI helps to determine the optimal data batch size that minimizes training errors and improves model accuracy without a significant increase in training time. Experimental results demonstrate the substantial advantages of multi-GPU configurations compared to single-GPU setups, providing faster training and improved model accuracy. It was particularly emphasized that MI-guided batch size tuning significantly outperforms traditional manual tuning methods, ensuring higher validation accuracy and reducing training time. The study showed that the MI-based approach is an effective tool for addressing the problem of optimizing ML model training processes in real-world scenarios where cyber security is critical. The proposed methods allow ML models to train faster and more accurately identify potential threats, making them particularly relevant for telecommunication networks where a rapid response to new threats in real time is required. The implementation of modern computational technologies such as multi-GPU systems and MI-optimized training enhances the efficiency and accuracy of machine learning models. This, in turn, improves cyber security measures and ensures a more reliable defence of telecommunication networks against malicious attacks. It is noted that the proposed approaches can be adapted not only for cyber security but also for other areas where high model accuracy and fast training are important. Future research prospects include the development of new machine learning methods, particularly deep neural networks, the exploration of alternative computational architectures such as quantum computing or distributed systems, and their integration into real-time systems. Special attention should be paid to the ethical aspects of implementing automated cyber security systems, particularly in preventing bias in algorithms and ensuring fairness in their application.

Prediction of infarct extent among patients with acute ischemic stroke using computed tomography perfusion is defined by predefined discrete computed tomography perfusion thresholds. Our objective is to develop a threshold-free computed tomography perfusion-based machine learning (ML) model to predict follow-up infarct in patients with acute ischemic stroke. Sixty-eight patients from the PRoveIT study (Measuring Collaterals With Multi-Phase CT Angiography in Patients With Ischemic Stroke) were used to derive a ML model using random forest to predict follow-up infarction voxel by voxel, and 137 patients from the HERMES study (Highly Effective Reperfusion Evaluated in Multiple Endovascular Stroke Trials) were used to test the derived ML model. Average map, Tmax, cerebral blood flow, cerebral blood volume, and time variables including stroke onset-to-imaging and imaging-to-reperfusion time, were used as features to train the ML model. Spatial and volumetric agreement between the ML model predicted follow-up infarct and actual follow-up infarct were assessed. Relative cerebral blood flow <0.3 threshold using RAPID software and time-dependent Tmax thresholds were compared with the ML model. In the test cohort (137 patients), median follow-up infarct volume predicted by the ML model was 30.9 mL (interquartile range, 16.4-54.3 mL), compared with a median 29.6 mL (interquartile range, 11.1-70.9 mL) of actual follow-up infarct volume. The Pearson correlation coefficient between 2 measurements was 0.80 (95% CI, 0.74-0.86, P<0.001) while the volumetric difference was -3.2 mL (interquartile range, -16.7 to 6.1 mL). Volumetric difference with the ML model was smaller versus the relative cerebral blood flow <0.3 threshold and the time-dependent Tmax threshold (P<0.001). A ML using computed tomography perfusion data and time estimates follow-up infarction in patients with acute ischemic stroke better than current methods.

Abstract. Despite showing great success of applications in many commercial fields, machine learning and data science models generally show limited success in many scientific fields, including hydrology (Karpatne et al., 2017). The approach is often criticized for its lack of interpretability and physical consistency. This has led to the emergence of new modelling paradigms, such as theory-guided data science (TGDS) and physics-informed machine learning. The motivation behind such approaches is to improve the physical meaningfulness of machine learning models by blending existing scientific knowledge with learning algorithms. Following the same principles in our prior work (Chadalawada et al., 2020), a new model induction framework was founded on genetic programming (GP), namely the Machine Learning Rainfall–Runoff Model Induction (ML-RR-MI) toolkit. ML-RR-MI is capable of developing fully fledged lumped conceptual rainfall–runoff models for a watershed of interest using the building blocks of two flexible rainfall–runoff modelling frameworks. In this study, we extend ML-RR-MI towards inducing semi-distributed rainfall–runoff models. The meaningfulness and reliability of hydrological inferences gained from lumped models may tend to deteriorate within large catchments where the spatial heterogeneity of forcing variables and watershed properties is significant. This was the motivation behind developing our machine learning approach for distributed rainfall–runoff modelling titled Machine Induction Knowledge Augmented – System Hydrologique Asiatique (MIKA-SHA). MIKA-SHA captures spatial variabilities and automatically induces rainfall–runoff models for the catchment of interest without any explicit user selections. Currently, MIKA-SHA learns models utilizing the model building components of two flexible modelling frameworks. However, the proposed framework can be coupled with any internally coherent collection of building blocks. MIKA-SHA's model induction capabilities have been tested on the Rappahannock River basin near Fredericksburg, Virginia, USA. MIKA-SHA builds and tests many model configurations using the model building components of the two flexible modelling frameworks and quantitatively identifies the optimal model for the watershed of concern. In this study, MIKA-SHA is utilized to identify two optimal models (one from each flexible modelling framework) to capture the runoff dynamics of the Rappahannock River basin. Both optimal models achieve high-efficiency values in hydrograph predictions (both at catchment and subcatchment outlets) and good visual matches with the observed runoff response of the catchment. Furthermore, the resulting model architectures are compatible with previously reported research findings and fieldwork insights of the watershed and are readily interpretable by hydrologists. MIKA-SHA-induced semi-distributed model performances were compared against existing lumped model performances for the same basin. MIKA-SHA-induced optimal models outperform the lumped models used in this study in terms of efficiency values while benefitting hydrologists with more meaningful hydrological inferences about the runoff dynamics of the Rappahannock River basin.

Machine learning-based water quality prediction using octennial in-situ Daphnia magna biological early warning system data

In the era of precision therapy, early classification of breast cancer (BRCA) molecular subtypes has clinical significance for disease management and prognosis. We explored the accuracy of machine learning (ML) models for early classification of BRCA molecular subtypes through a systematic review of the literature currently available. We retrieved relevant studies published in PubMed, EMBASE, Cochrane, and Web of Science until 15 April 2022. A prediction model risk of bias assessment tool (PROBAST) was applied for the assessment of risk of bias of a genomics-based ML model, and the Radiomics Quality Score (RQS) was simultaneously used to evaluate the quality of this radiomics-based ML model. A random effects model was adopted to analyze the predictive accuracy of genomics-based ML and radiomics-based ML for Luminal A, Luminal B, Basal-like or triple-negative breast cancer (TNBC), and human epidermal growth factor receptor 2 (HER2). The PROSPERO of our study was prospectively registered (CRD42022333611). Of the 38 studies were selected for analysis, 14 ML models were based on gene-transcriptomic, with only 4 external validations; and 43 ML models were based on radiomics, with only 14 external validations. Meta-analysis results showed that c-statistic values of the ML based on radiomics for the identification of BRCA molecular subtypes Luminal A, Luminal B, Basal-like or TNBC, and HER2 were 0.76 [95% confidence interval (CI): 0.60-0.96], 0.78 (95% CI: 0.69-0.87), 0.89 (95% CI: 0.83-0.91), and 0.83 (95% CI: 0.81-0.86), respectively. The c-statistic values of ML based on the gene-transcriptomic analysis cohort for the identification of the previously described BRCA molecular subtypes were 0.96 (95% CI: 0.93-0.99), 0.96 (95% CI: 0.93-0.99), 0.98 (95% CI: 0.95-1.00), and 0.97 (95% CI: 0.96-0.98) respectively. Additionally, the sensitivity of the ML model based on radiomics for each molecular subtype ranged from 0.79 to 0.85, while the sensitivity of the ML model based on gene-transcriptomic was between 0.92 and 0.99. Both radiomics and gene transcriptomics produced ideal effects on BRCA molecular subtype prediction. Compared with radiomics, gene transcriptomics yielded better prediction results, but radiomics was simpler and more convenient from a clinical point of view.

Artificial intelligence (AI) is reshaping neurosurgery, offering unprecedented opportunities to enhance diagnostics, personalize treatment, and predict outcomes. At the heart of this transformation is the ability to effectively harness big data (BD) within the electronic medical record. Understanding these data structures is essential for making sense of the vast volumes of information generated in modern neurosurgical practice. Equally important are the machine learning (ML) models driving these advancements. From supervised learning and convolutional neural networks to generative AI, these tools are already making a mark in areas such as brain tumor segmentation and spine surgery outcome predictions. Their versatility highlights the potential of ML to complement clinical expertise and streamline decision-making in neurosurgery. However, adopting BD and ML also brings ethical challenges that cannot be ignored. Bias in algorithms threatens to reinforce health disparities, whereas concerns about data privacy demand vigilance in handling sensitive patient information. In addition, the question of liability looms large as ML increasingly influences clinical decisions. The aim of the study was to provide a roadmap for neurosurgeons navigating the evolving intersection of BD, ML, and ethical responsibility in the AI era.

This study focuses on using various machine learning (ML) models to evaluate the shear behaviors of ultra-high-performance concrete (UHPC) beams reinforced with glass fiber-reinforced polymer (GFRP) bars. The main objective of the study is to predict the shear strength of UHPC beams reinforced with GFRP bars using ML models. We use four different ML models: support vector machine (SVM), artificial neural network (ANN), random forest (R.F.), and extreme gradient boosting (XGBoost). The experimental database used in the study is acquired from various literature sources and comprises 54 test observations with 11 input features. These input features are likely parameters related to the composition, geometry, and properties of the UHPC beams and GFRP bars. To ensure the ML models' generalizability and scalability, random search methods are utilized to tune the hyperparameters of the algorithms. This tuning process helps improve the performance of the models when predicting the shear strength. The study uses the ACI318M-14 and Eurocode 2 standard building codes to predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams. The ML models' predictions are compared to the results obtained from these building code standards. According to the findings, the XGBoost model demonstrates the highest predictive test performance among the investigated ML models. The study employs the SHAP (SHapley Additive exPlanations) analysis to assess the significance of each input parameter in the ML models' predictive capabilities. A Taylor diagram is used to statistically compare the accuracy of the ML models. This study concludes that ML models, particularly XGBoost, can effectively predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams.

As artificial intelligence (AI) becomes widespread, there is increasing attention on investigating bias in machine learning (ML) models. Previous research concentrated on classification problems, with little emphasis on regression models. This paper presents an easy-to-apply and effective methodology for mitigating bias in bagging and boosting regression models, that is also applicable to any model trained through minimizing a differentiable loss function. Our methodology measures bias rigorously and extends the ML model's loss function with a regularization term to penalize high correlations between model errors and protected attributes. We applied our approach to three popular tree-based ensemble models: a random forest model (RF), a gradient-boosted model (GBT), and an extreme gradient boosting model (XGBoost). We implemented our methodology on a case study for predicting road-level traffic volume, where RF, GBT, and XGBoost models were shown to have high accuracy. Despite high accuracy, the ML models were shown to perform poorly on roads in minority-populated areas. Our bias mitigation approach reduced minority-related bias by over 50%.