Operational decision-making with machine learning and causal inference

P125. Development of a novel ensemble machine learning algorithm for prediction of complications and readmission after anterior cervical spinal fusion

The best practice of watershed management is through the understanding of the hydrological processes. As a matter of fact, hydrological processes are highly associated with stochastic, non-linear, and non-stationary phenomena. Hydrological processes simulation and modeling are challenging issues in the domains of hydrology, climate and environment. Hence, the development of machine learning (ML) models for solving those complex hydrological problems took essential place over the past couple decades. It can be observed, hydrological data availability has increased remarkably, and thus computational resources has led to a resurgence in ML models’ development. It has been witnessed huge efforts on the hydrological processes modeling using the facility of ML models and several review researches have been conducted. Literature studies approved the capacity of ML models in the field of hydrology over the classical “traditional models” based on their forecastability, flexibility, precision, generalization, and modeling execution convergence speed. However, although several potential merits were observed in ML model’s development, several limitations are allied such as the interpretability of those black-box models, the practicality of the ML models in watershed management, and difficulty to explain the physical hydrological processes. In this survey, an exhibition for all the published review articles on the development of ML models for hydrological processes and recognize all the research gaps and potential research direction. The ultimate aim of the current survey is to establish a new milestone for the interested hydrology, environment and climate researchers on the applications of ML models.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.

Summary A meticulous interpretation of steady-state or unsteady-state relative permeability (Kr) experimental data is required to determine a complete set of Kr curves. In this work, different machine learning (ML) models were developed to assist in a faster estimation of these curves from steady-state drainage coreflooding experimental runs. These ML algorithms include gradient boosting (GB), random forest (RF), extreme gradient boosting (XGB), and deep neural network (DNN) with a main focus on and comparison of the two latter algorithms (XGB and DNN). Based on existing mathematical models, a leading-edge framework was developed where a large database of Kr and capillary pressure (Pc) curves were generated. This database was used to perform thousands of coreflood simulation runs representing oil-water drainage steady-state experiments. The results obtained from these simulation runs, mainly pressure drop along with other conventional core analysis data, were used to estimate analytical Kr curves based on Darcy’s law. These analytically estimated Kr curves along with the previously generated Pc curves were fed as features into the ML model. The entire data set was split into 80% for training and 20% for testing. The k-fold cross-validation technique was applied to increase the model’s accuracy by splitting 80% of the training data into 10 folds. In this manner, for each of the 10 experiments, nine folds were used for training and the remaining fold was used for model validation. Once the model was trained and validated, it was subjected to blind testing on the remaining 20% of the data set. The ML model learns to capture fluid flow behavior inside the core from the training data set. In terms of applicability of these ML models, two sets of experimental data were needed as input; the first was the analytically estimated Kr curves from the steady-state drainage coreflooding experiments, while the other was the Pc curves estimated from centrifuge or mercury injection capillary pressure (MICP) measurements. The trained/tested model was then able to estimate Kr curves based on the experimental results fed as input. Furthermore, to test the performance of the ML model when only one set of experimental data is available to an end user, a recurrent neural network (RNN) algorithm was trained/tested to predict Kr curves in the absence of Pc curves as an input. The performance of the three developed models (XGB, DNN, and RNN) was assessed using the values of the coefficient of determination (R2) along with the loss calculated during training/validation of the model. The respective crossplots along with comparisons of ground truth vs. artificial intelligence (AI)-predicted curves indicated that the model is capable of making accurate predictions with an error percentage between 0.2% and 0.6% on history-matching experimental data for all three tested ML techniques. This implies that the AI-based model exhibits better efficiency and reliability in determining Kr curves when compared to conventional methods. The developed ML models by no means replace the need to conduct drainage coreflooding or centrifuge experiments but act as an alternative to existing commercial platforms that are used to interpret experimental data to predict Kr curves. The two main advantages of the developed ML models are their capability of predicting Kr curves within a matter of a few minutes as well as with limited intervention from the end user. The results also include a comparison between classical ML approaches, shallow neural networks, and DNNs in terms of accuracy in predicting the final Kr curves. The research presented here is an extension of the state-of-the-art framework proposed by Mathew et al. (2021). However, the two main aspects of the current study are the application of deep learning for the prediction of Kr curves and the application of feature engineering. The latter not only reduces the training/testing time for the ML models but also enables the end user to obtain the final predictions with the least set of experimental data. The various models discussed in this research work currently focus on the prediction of Kr curves for drainage steady-state experiments; however, the work can be extended to capture the imbibition cycle as well.

Accurate estimation of reference evapotranspiration (ETo) plays a vital role in irrigation and water resource planning. The Penman–Monteith method recommended by the Food and Agriculture Organization (FAO PM56) is widely used and considered a standard to calculate ETo. However, FAO PM56 cannot be used with limited meteorological variables, so it is compulsory to choose an alternative model for ETo estimation, which requires fewer variables. This study built ten machine learning (ML) models based on multi-function, neural network, and tree-based structure against the FAO PM56 method. For this purpose, time series temperature data on a monthly scale are only used to train ML models. The developed ML models were applied to estimate ETo at different test stations and the obtained results were compared with the FAO PM56 method to verify and validate their performance in ETo estimation for the selected stations. In addition, multiple statistical indicators, including root-mean-square error (RMSE), coefficient of determination (R2), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), and correlation coefficient (r) were calculated to compare the performance of each ML model on ETo estimation. Among the applied ML models, the ETo tree boost (TB) ML model outperformed the other ML models in estimating ETo in diverse climatic conditions based on statistical indicators (R2, NSE, r, RMSE, and MAE). Moreover, the observed R2, NSE, and r were the highest for the TB ML model, while RMSE and MAE were found to be the lowest at the study sites compared to other applied ML models. Lastly, ETo point data yielded from the TB ML model was used in an interpolation process to create monthly and annual ETo maps. Based on the ETo maps, this study suggests mainly a focus on areas with high ETo values and proper irrigation scheduling of crops to ensure water sustainability.

Machine learning-based water quality prediction using octennial in-situ Daphnia magna biological early warning system data

BackgroundThe large amount of heterogeneous data collected in surgical/endoscopic practice calls for data-driven approaches as machine learning (ML) models. The aim of this study was to develop ML models to predict endoscopic sleeve gastroplasty (ESG) efficacy at 12 months defined by total weight loss (TWL) % and excess weight loss (EWL) % achievement. Multicentre data were used to enhance generalizability: evaluate consistency among different center of ESG practice and assess reproducibility of the models and possible clinical application. Models were designed to be dynamic and integrate follow-up clinical data into more accurate predictions, possibly assisting management and decision-making.MethodsML models were developed using data of 404 ESG procedures performed at 12 centers across Europe. Collected data included clinical and demographic variables at the time of ESG and at follow-up. Multicentre/external and single center/internal and temporal validation were performed. Training and evaluation of the models were performed on Python’s scikit-learn library. Performance of models was quantified as receiver operator curve (ROC-AUC), sensitivity, specificity, and calibration plots.ResultsMulticenter external validation: ML models using preoperative data show poor performance. Best performances were reached by linear regression (LR) and support vector machine models for TWL% and EWL%, respectively, (ROC-AUC: TWL% 0.87, EWL% 0.86) with the addition of 6-month follow-up data.Single-center internal validation: Preoperative data only ML models show suboptimal performance. Early, i.e., 3-month follow-up data addition lead to ROC-AUC of 0.79 (random forest classifiers model) and 0.81 (LR models) for TWL% and EWL% achievement prediction, respectively. Single-center temporal validation shows similar results.ConclusionsAlthough preoperative data only may not be sufficient for accurate postoperative predictions, the ability of ML models to adapt and evolve with the patients changes could assist in providing an effective and personalized postoperative care. ML models predictive capacity improvement with follow-up data is encouraging and may become a valuable support in patient management and decision-making.

Introduction: Early and accurate diagnosis of transthyretin amyloid cardiomyopathy (ATTR-CM) is known to improve prognosis. However, it is sometimes challenging to distinguish ATTR-CM from other cardiac conditions that cause left ventricular hypertrophy (LVH), such as hypertrophic cardiomyopathy (HCM), hypertensive LVH (h-LVH), and aortic stenosis (AS). The Mayo ATTR-CM score is commonly used to distinguish ATTR-CM from these mimickers. Herein, we constructed a machine learning (ML) model and compared its performance to the Mayo score. Hypothesis: A ML model based on readily available clinical variables will outperform the Mayo score in distinguishing ATTR-CM from other cardiac conditions that cause LVH. Methods: We performed a multicenter case-control study of 1,605 patients, including 303 ATTR-CM cases and 1,302 controls – i.e., 935 with HCM, 331 with h-LVH, and 36 with AS. We allocated the former two-thirds of each disease group to the training set and the latter one-third to the prospective test set for validation. We developed a main ML model in the training set using XGBoost with 18 clinical variables including demographics, comorbidities, laboratory values, and echocardiographic data. We examined its performance in the test set using the area under the receiver-operating-characteristic curve (AUC) and decision curve analysis (DCA). In addition, we developed a simplified ML model using only the top 5 variables identified by feature importance and tested its performance. Results: The top 5 most important variables were age, LV diameter in systole, LV posterior wall thickness in diastole, LV ejection fraction, and left atrial diameter ( Figure 1 ). The main ML model (AUC: 0.96, 95% CI: 0.93–0.98) outperformed the Mayo score (AUC: 0.84, 95% CI: 0.79–0.89, DeLong’s P <0.0001; Figure 2 ) in the test set. DCA demonstrated that the ML model provided greater net benefit across all threshold probabilities ( Figure 3 ). The simplified 5-variable model also outperformed the Mayo score (AUC: 0.94, 95% CI: 0.91–0.97, DeLong’s P <0.0001). Conclusions: In this multicenter study, we developed ML models that effectively distinguish ATTR-CM from its mimickers that cause LVH using readily available clinical variables. Given the simplicity and accuracy of our models, the present study serves as an important initial step toward the development of a robust, data-driven tool that could be readily automated and integrated into electronic medical record system to screen for ATTR-CM.

Interpretable vs. noninterpretable machine learning models for data-driven hydro-climatological process modeling

In recent years, infections and damage caused by malware have increased at exponential rates. At the same time, machine learning (ML) techniques have shown tremendous promise in many domains, often out performing human efforts by learning from large amounts of data. Results in the open literature suggest that ML is able to provide similar results for malware detection, achieving greater than 99% classification accuracy [49]. However, the same detection rates when applied in deployed settings have not been achieved. Malware is distinct from many other domains in which ML has shown success in that (1) it purposefully tries to hide, leading to noisy labels and (2) often its behavior is similar to benign software only differing in intent, among other complicating factors. This report details the reasons for the difficultly of detecting novel malware by ML methods and offers solutions to improve the detection of novel malware. We propose to detect malware by detecting behaviors commonly exhibited by malware such as DLL injection, and process hollowing. This is based on the assumption that there is a set of behaviors that are common to most malware samples and detecting them will generalize to novel malware. Additionally, detected behaviors point analysts toward appropriate handling and mitigation strategies, which is not the case with a binary benign/malicious classification. A behavior labeling method was developed and was used to label an existing malware dataset. Results show that detecting malicious behaviors is much more difficult than simply classifying malware and goodware?achieving 80% accuracy compared to reported 99% accuracy from classifying malware and goodware. This drop is due to several reasons which are detailed in the report. We also propose to evaluate the performance of detecting novel malware by holding out a malware family for testing and training on the other families. Traditional ML evaluation will shuffle the data and then split the data into training and testing. Our method addresses the use-case when novel malware families are encountered and they require more than just a malicious or benign designation. Our results suggest that this type of evaluation is much more difficult than traditional methods and provides more realistic results, albeit, significantly worse. For our behavior detection, accuracy decreases from 80% to 68% across all behaviors when holding out a malware family from training. We show that the degradation in performance is because each malware family has distinct characteristics resulting in high extrapolations by an ML model. Here, an ML model should return an "I do not know" response and request further analysis from an analyst. We run a number of experiments that compare novel malware families to the training data using different feature representations including a genomics-inspired distance measure and features extracted by deep learning. Generally, held-out families are significantly different from the training data, resulting in unpredictable results. This has been observed generally in the ML community [22, 9]. We empirically demonstrate this in the domain of malware detection. In an attempt to improve the detection of malware behaviors, we examine the impact that additional synthetic data has on the performance of an ML model in detecting behaviors in novel malware families. We find that while synthetic data does improve the performance of ML models, often simpler methods perform better than more complicated ones. Two generative modeling techniques were examined to produce synthetic malware samples such that the behaviors present are able to be specified externally. The difficulty is due to finer grained analysis of the executable and modifying the problem from a binary classification problem to a multi-label problem. The addition of synthetic data increases the overall accuracy from 68% to 70%. While far less accurate than measures presented in academic analyses, we believe that this is more representative of real-world performance and allows models to be properly placed within a malware detection system. We suggest that in highly dynamic environments ML pipelines should determine whether an ML model is competent in the area of new data and should involve mechanisms to improve over time with a human in the loop.

Small data, big challenges: Machine- and deep-learning strategies for data-limited drug discovery.

Cardiac rehabilitation (CR) programs are vital for people recovering from cardiac surgeries or events. However, the effectiveness of CR programs varies and some patients may not adhere to them, which might result in less favourable outcomes. Machine Learning (ML) models could help predict the effectiveness and adherence of CR programs. This study proposes two such models: a) the CR program effectiveness prediction model and b) the CR program adherence prediction model. The models were trained on data from retrospective cohort study with 1448 participants collected at the Cardiac Rehabilitation Unit of the Hospital Clinico de Santiago de Compostela in Galicia, Spain (SERGAS). Data cleaning, normalization, imputation, statistical analysis, feature selection and repeated stratified k-fold cross-validation (CV) were applied on the ML pipeline, which tested and evaluated on baseline demographic, clinical, exercise tests and behavioral features. The performance of ML models was assessed by mean Area Under operating characteristic Curve (AUC), specificity, sensitivity, and balanced accuracy with 95% confidence interval (CI). The results show that Random Forest (RF) outperformed other evaluated classifiers for the CR program effectiveness model, with the highest AUC value of 0.789 (0.775, 0.802), while the best classifier for the CR adherence model was the Logistic Regression (LR) classifier, with an AUC value of 0.757 (0.749, 0.764). SHAP plots were also used to investigate the relationships among the variables used in the analysis. Finally, a two-dimensional scoring system was developed to jointly assess predicted adherence and effectiveness, enabling personalized visualization of patient response to CR.

Rapid epicentral distance estimation is of great significance for earthquake early warning (EEW). To rapidly and reliably predict epicentral distance, we developed machine learning models with multiple feature inputs for epicentral distance estimation using a single station and explored the feasibility of three machine learning methods, namely, Random Forest, eXtreme Gradient Boosting, and Support Vector Machine, for epicentral distance estimation. We used strong-motion data recorded by the Japanese Kyoshin network within a range of 1° (∼112 km) from the epicenter to train machine learning models. We used 30 features extracted from the P-wave signal as inputs to the machine learning models and the epicentral distance as the prediction target of the models. For the same test data set, within 0.1–5 s after the P-wave arrival, the epicentral distance estimation results of these three machine learning models were similar. Furthermore, these three machine learning methods can obtain smaller mean absolute errors and root mean square errors, as well as larger coefficients of determination (R2), for epicentral distance estimation than traditional EEW epicentral distance estimation methods, indicating that these three machine learning models can effectively improve the accuracy of epicentral distance estimation to a certain extent. In addition, we analyzed the importance of different features as inputs to machine learning models using SHapley additive exPlanations. We found that using the top 15 important features as inputs, these three machine learning models can also achieve good results for epicentral distance estimation. Based on our results, we inferred that the machine learning models for estimating epicentral distance proposed in this study are meaningful in EEW.

PD27-01 DEVELOPMENT OF A MACHINE LEARNING (ML) MODEL TO AUTOMATICALLY AND PRECISELY IDENTIFY KIDNEY STONES FROM URETEROSCOPY VIDEO RECORDINGS

Study question Can a machine learning (ML) model, developed using modern neural network architecture produce comparable annotation data; utilisable for algorithmic outcome prediction, to manual time-lapse annotations? Summary answer The model automatically annotated unseen embryos with comparable results to manual methods, generating morphokinetic data to enable comparably predictive outputs from an embryo selection algorithm. What is known already The application of artificial intelligence across healthcare industries, including fertility, is increasing. Several ML models are available that seek to generate or analyse embryo images and morphokinetic data, and to determine embryo viability potential. Along with photographic images, the use of time-lapse in IVF laboratories has amassed numeric data, resulting predominantly from annotated manual assessment of images over time. Embryo annotation practice is variable in quality, can be subjective and is time-consuming; commonly taking several minutes per embryo. The development of rapid, accurate automatic annotation would represent a significant time-saving as well as an increase in reproducibility and accuracy. Study design, size, duration Multicentre quality assured annotation data from 63,383 time-lapse monitored embryos (EmbryoScope®), comprising over 400 million individual images, were used to train a ML model to automatically generate morphokinetic annotations. Data was derived from 8 UK clinics within a cohesive group between 2012-2021. Accuracy was assessed using 900 unseen embryos (with live birth outcome) by comparing the output of an established in-house, prospectively validated embryo selection model when the input was either ML-automated, or manual annotations. Participants/materials, setting, methods Multi-focal plane images were processed on the Azure cloud (Microsoft) and resampled to 300x300 pixels. A Laplacian-based focal stacking algorithm merged frames into a single image. The model consisted of an EfficientNetB4 Convolutional Neural Network classifier to extract features and classify the stage of embryo images. A Temporal Convolutional Network interpreted a time-series of image features; producing annotations from pronuclear fading through to blastocyst. Soft localisation loss function used QA data to integrate annotation subjectivities. Main results and the role of chance The ML model rapidly and automatically generated annotations. Efficacy and comparability of the ML model to automate reliable, utilisable annotations was demonstrated by comparison with manual annotation data and the ML model’s ability to auto-generate annotations which could be used to predict live birth by providing annotation data to an established, validated in house embryo selection model. Live birth-predictive capability was measured, and benchmarked against manual annotation, using the area under the receiver operating characteristic curve (AUC). When tested on time-lapse images, collected from pronuclear fading to full blastulation, representing 900 previously unseen, transferred blastocysts where live birth outcomes were blinded, the in-house developed auto-annotation ML model resulted in an AUC of 0.686 compared with 0.661 for manual annotations, for live birth prediction. Auto annotation using the developed model took only milliseconds to complete per embryo. The developed auto-annotation model, built and tested on large data, is considered suitable for productionisation with the aim of being validated and integrated into an application to support IVF laboratory practice. Limitations, reasons for caution Whilst this model was trained to recognise key morphokinetic events, there are other morphokinetic variables that may be useful in the prediction of live birth and further improve embryo selection, or deselection, ability. Akin to manual interpretation, some embryos may fail to be annotated or need second opinion. Wider implications of the findings There is increasing evidence supporting the application of ML to utilise big data from time-lapse imaging and fertility care generally. Whilst promising benefits to IVF clinics and patients, responsible use of data is required alongside large high-quality datasets, and rigorous validation, to ensure safe and robust applications. Trial registration number N/A