Reliable Machine Learning for Wearable Activity Monitoring

P125. Development of a novel ensemble machine learning algorithm for prediction of complications and readmission after anterior cervical spinal fusion

Summary A meticulous interpretation of steady-state or unsteady-state relative permeability (Kr) experimental data is required to determine a complete set of Kr curves. In this work, different machine learning (ML) models were developed to assist in a faster estimation of these curves from steady-state drainage coreflooding experimental runs. These ML algorithms include gradient boosting (GB), random forest (RF), extreme gradient boosting (XGB), and deep neural network (DNN) with a main focus on and comparison of the two latter algorithms (XGB and DNN). Based on existing mathematical models, a leading-edge framework was developed where a large database of Kr and capillary pressure (Pc) curves were generated. This database was used to perform thousands of coreflood simulation runs representing oil-water drainage steady-state experiments. The results obtained from these simulation runs, mainly pressure drop along with other conventional core analysis data, were used to estimate analytical Kr curves based on Darcy’s law. These analytically estimated Kr curves along with the previously generated Pc curves were fed as features into the ML model. The entire data set was split into 80% for training and 20% for testing. The k-fold cross-validation technique was applied to increase the model’s accuracy by splitting 80% of the training data into 10 folds. In this manner, for each of the 10 experiments, nine folds were used for training and the remaining fold was used for model validation. Once the model was trained and validated, it was subjected to blind testing on the remaining 20% of the data set. The ML model learns to capture fluid flow behavior inside the core from the training data set. In terms of applicability of these ML models, two sets of experimental data were needed as input; the first was the analytically estimated Kr curves from the steady-state drainage coreflooding experiments, while the other was the Pc curves estimated from centrifuge or mercury injection capillary pressure (MICP) measurements. The trained/tested model was then able to estimate Kr curves based on the experimental results fed as input. Furthermore, to test the performance of the ML model when only one set of experimental data is available to an end user, a recurrent neural network (RNN) algorithm was trained/tested to predict Kr curves in the absence of Pc curves as an input. The performance of the three developed models (XGB, DNN, and RNN) was assessed using the values of the coefficient of determination (R2) along with the loss calculated during training/validation of the model. The respective crossplots along with comparisons of ground truth vs. artificial intelligence (AI)-predicted curves indicated that the model is capable of making accurate predictions with an error percentage between 0.2% and 0.6% on history-matching experimental data for all three tested ML techniques. This implies that the AI-based model exhibits better efficiency and reliability in determining Kr curves when compared to conventional methods. The developed ML models by no means replace the need to conduct drainage coreflooding or centrifuge experiments but act as an alternative to existing commercial platforms that are used to interpret experimental data to predict Kr curves. The two main advantages of the developed ML models are their capability of predicting Kr curves within a matter of a few minutes as well as with limited intervention from the end user. The results also include a comparison between classical ML approaches, shallow neural networks, and DNNs in terms of accuracy in predicting the final Kr curves. The research presented here is an extension of the state-of-the-art framework proposed by Mathew et al. (2021). However, the two main aspects of the current study are the application of deep learning for the prediction of Kr curves and the application of feature engineering. The latter not only reduces the training/testing time for the ML models but also enables the end user to obtain the final predictions with the least set of experimental data. The various models discussed in this research work currently focus on the prediction of Kr curves for drainage steady-state experiments; however, the work can be extended to capture the imbibition cycle as well.

Metastatic castration-resistant prostate cancer (mCRPC) is an aggressive, lethal state of prostate cancer, for which early progression is an indicator of poor prognosis. The ability to predict this progression is of paramount clinical importance for guiding personalized treatment strategies. We aimed to develop and validate a novel machine learning (ML) model to predict early progression (≤ 12 months) to mCRPC and compare its performance against standard ML algorithms. This was a retrospective analysis of 172 patients with mHSPC from the publicly available MSK-IMPACT cohort. Inclusion criteria specified patients with mHSPC who had undergone genomic profiling and progressed to mCRPC during follow-up. Patients with incomplete data were excluded. We collected 11 clinical, pathological, and genomic variables. The primary outcome was early progression (≤ 12 months) to mCRPC. Model performance was evaluated using a stratified fivefold cross-validation, with AUC as the primary metric. A novel Rivality Index (RINH)-based model, adapted from chemoinformatics, demonstrated significantly superior predictive performance (AUC: 0.86) compared to a panel of standard ML algorithms, none of which exceeded an AUC of 0.67. The model achieved an accuracy of 0.74, a sensitivity of 0.70, and a specificity of 0.77. Key limitations include the retrospective design and use of a single-institution data set. This novel RINH model offers a robust tool for risk stratification in mHSPC patients, capable of personalizing therapeutic strategies. However, external validation in multi-center, prospective cohorts is an essential next step before its consideration as a clinical decision support tool.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.

12074 Background: Skeletal-related events (SREs) are detrimental clinical events in bone metastasis from advanced cancer, defined by pathologic fracture, spinal cord compression, and inevitable surgical or radiational intervention to the bone. Given their negative impact on quality of life and prognosis and interpatient heterogeneity in the SRE risk, accurate identification of patients with high SRE risk is critical. Methods: The patient-level data from three randomized clinical trials that administered zoledronic acid (ZA) to patients with bone-metastatic breast cancer, castration-resistant prostate cancer (CRPC), and other types of cancer were analyzed. (N = 460, 452, and 315, respectively) Machine learning (ML) models to predict SREs within 18 months (breast cancer), 12 months (CRPC), and 9 months (other cancers) were developed based on more than 40 baseline clinical and laboratory data. Seven ML algorithms and five feature selection methods were utilized to develop multiple models. The ML model with the best performance was identified based on the F1 score and the area under the receiver operating characteristic curve (AUC-ROC) for each cancer type and interrogated for important features with Shapley additive explanations. Lastly, the ML models’ ability to stratify patients by the cumulative SRE risk was evaluated by calculating hazard ratio (HR) with Cox-proportional hazards models. Results: Among the multiple ML models developed with different algorithms and feature selection methods, the model developed utilizing the random forests algorithm and the Boruta method for selecting features demonstrated the best performance in all types of cancer (F1 0.70, 0.67, and 0.67, and AUC-ROC 0.72, 0.68, and 0.73 for breast cancer, CRPC, and other cancers, respectively). In the ML model for breast cancer, performance status (PS), history of SRE, serum alkaline phosphatase (ALP), history of anti-neoplastic surgery, radiation therapy, and pathologic fracture were included as important features. Serum ALP, albumin, sodium, Gleason scores, and geographic regions were shown to be relevant in the CRPC model. For other cancers, serum ALP, albumin, total protein, phosphorus, red blood cell count, white blood cell count, visceral metastases, and history of arthritis were incorporated in the ML model. The ML model prediction successfully stratified the patients for cumulative SRE risk in all three cohorts. (HR with 95% confidence interval: 2.43 [1.86 – 3.18], 1.92 [1.51 – 2.45], and 3.06 [2.29 – 4.09] for breast cancer, CRPC, and other types of cancer, respectively). Conclusions: ML models incorporating baseline clinical and laboratory data can identify patients with bone metastasis on ZA harboring a high SRE risk.

Small data, big challenges: Machine- and deep-learning strategies for data-limited drug discovery.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

Improving autocoding performance of rare categories in injury classification: Is more training data or filtering the solution?

Preventing the expansion of perihematomal edema (PHE) represents a novel strategy for the improvement of neurological outcomes in intracerebral hemorrhage (ICH) patients. Our goal was to predict early and delayed PHE expansion using a machine learning approach. We enrolled 550 patients with spontaneous ICH to study early PHE expansion, and 389 patients to study delayed expansion. Two imaging researchers rated the shape and density of hematoma in non-contrast computed tomography (NCCT). We trained a radiological machine learning (ML) model, a radiomics ML model, and a combined ML model, using data from radiomics, traditional imaging, and clinical indicators. We then validated these models on an independent dataset by using a nested 4-fold cross-validation approach. We compared models with respect to their predictive performance, which was assessed using the receiver operating characteristic curve. For both early and delayed PHE expansion, the combined ML model was most predictive (early/delayed AUC values were 0.840/0.705), followed by the radiomics ML model (0.799/0.663), the radiological ML model (0.779/0.631), and the imaging readers (reader 1: 0.668/0.565, reader 2: 0.700/0.617). We validated a machine learning approach with high interpretability for the prediction of early and delayed PHE expansion. This new technique may assist clinical practice for the management of neurocritical patients with ICH. • This is the first study to use artificial intelligence technology for the prediction of perihematomal edema expansion. • A combined machine learning model, trained on data from radiomics, clinical indicators, and imaging features associated with hematoma expansion, outperformed all other methods.

Shale caprocks, often organic-rich, play a significant role in hydrogen sequestration, but their wettability is typically altered from water-wet to H2-wet due to the organic matter, compromising their sealing performance. Silica nanofluids can restore water-wet conditions, which is vital for effective containment of H2 in geological formations. However, laboratory experiments to test the effects of nanomaterials on wettability and interfacial tension (IFT) are resource-intensive and time-consuming. To address this challenge, data-driven machine learning (ML) models offer a promising alternative, allowing for the prediction of key UHS parameters—such as advancing and receding contact angles without the need for extensive laboratory testing. We have applied multiple ML models, including gradient boosting, extreme gradient boosting (XGboost), Catboost, random forest, and extra trees to predict the wettability of organic rich shale samples treated with silica nanofluids. The ML models were trained on a comprehensive dataset generated through laboratory experiments conducted under a wide range of pressure (0.5 – 1600psi) and temperature (298-323 K) conditions, using brine (1 wt.% KCl + 2 wt.% NaCl) and varying silica nanofluid concentrations (0.05–1.0 wt.%). Advanced visualization techniques, including swarm maps, box plots, and heatmap plots, were utilized to analyze the experimental dataset thoroughly. To enhance the generalization capabilities of the ML models, we employed Bayesian optimization for hyperparameter tuning, along with k-fold cross-validation and grid-search optimization techniques. Both statistical and graphical analyses were performed to show the reliability and performance of the models. The results demonstrated that the implemented ML models, particularly the XGboost and Catboost, accurately predicted the wettability behavior under various operating conditions. The training and testing average R² scores were above 0.998 and 0.949, respectively, confirming the high accuracy of the predictions. These findings highlight the reliability of the ML algorithms in predicting the complex interaction between shale, brine, and H2. Further investigation into the feature importance revealed that nanomaterials had the greatest influence on the wettability of the ternary system. The silica nanofluid concentrations played a key role in modifying wettability, with varying impacts on the receding contact angles, emphasizing the effect of nanomaterial treatment. The model's predictions were consistent with experimental results, suggesting that geological conditions, including pressure and temperature, have a profound influence on rock mineral wettability, directly impacting H2 geo-storage capacity. This study is the first to employ data-driven ML models to predict critical UHS parameters in organic-rich shales treated with silica nanofluids. The ability of the ML models to predict UHS parameters across a wide range of conditions provides a valuable tool for optimizing UHS system designs, offering a more efficient and scalable approach compared to traditional experimental methods.

The ongoing Russia–Ukraine conflict has exacerbated the global crisis of natural gas supply, particularly in Europe. During the winter season, major importers of liquefied natural gas (LNG), such as South Korea and Japan, were directly affected by fluctuating spot LNG prices. This study aimed to use machine learning (ML) to predict the Japan Korea Marker (JKM), a spot LNG price index, to reduce price fluctuation risks for LNG importers such as the Korean Gas Corporation (KOGAS). Hence, price prediction models were developed based on long short-term memory (LSTM), artificial neural network (ANN), and support vector machine (SVM) algorithms, which were used for time series data prediction. Eighty-seven variables were collected for JKM prediction, of which eight were selected for modeling. Four scenarios (scenarios A, B, C, and D) were devised and tested to analyze the effect of each variable on the performance of the models. Among the eight variables, JKM, national balancing point (NBP), and Brent price indexes demonstrated the largest effects on the performance of the ML models. In contrast, the variable of LNG import volume in China had the least effect. The LSTM model showed a mean absolute error (MAE) of 0.195, making it the best-performing algorithm. However, the LSTM model demonstrated a decreased in performance of at least 57% during the COVID-19 period, which raises concerns regarding the reliability of the test results obtained during that time. The study compared the ML models’ prediction performances with those of the traditional statistical model, autoregressive integrated moving averages (ARIMA), to verify their effectiveness. The comparison results showed that the LSTM model’s performance deviated by an MAE of 15–22%, which can be attributed to the constraints of the small dataset size and conceptual structural differences between the ML and ARIMA models. However, if a sufficiently large dataset can be secured for training, the ML model is expected to perform better than the ARIMA. Additionally, separate tests were conducted to predict the trends of JKM fluctuations and comprehensively validate the practicality of the ML models. Based on the test results, LSTM model, identified as the optimal ML algorithm, achieved a performance of 53% during the regular period and 57% d during the abnormal period (i.e., COVID-19). Subject matter experts agreed that the performance of the ML models could be improved through additional studies, ultimately reducing the risk of price fluctuations when purchasing spot LNG.

Optimizing operational decisions, routine actions within some business or operational process, is a key challenge across a variety of domains and application areas. The increasing availability of data, computational power, and advanced machine learning (ML) algorithms offers exciting opportunities for data-driven decision support. To advance the potential of ML for optimizing operational decision-making, we explore two research directions, aiming to develop ML models that are decision-focused and causal. This dissertation presents several developments in machine learning in these two areas. ML is effective at making predictions from historical data: for example, estimating a transaction's fraud probability by comparing it to past cases. However, decision-makers not only need to consider these predictions, but also the operational context. For example, the decision-maker uses predicted fraud probabilities to determine which transactions to investigate, while aiming to minimize monetary losses due to fraud and considering the available capacity of the fraud investigations team. Predictions can help reduce uncertainty (e.g., by predicting the fraud probability), but standard ML models are prediction-focused, instead of decision-focused. This distinction involves two challenges for data-driven decision-making. First, prediction-focused models prioritize predictive accuracy instead of the resulting decision quality (e.g., fraud losses recovered by the bank). Second, these models fail to account for operational constraints, such as the available investigation capacity. Decision-focused learning aims to improve data-driven decision-making by addressing these issues and incorporating the operational context into the optimization of ML models. In this dissertation, we analyze cost-sensitive learning within this prediction-optimization framework and evaluate general strategies for making cost-optimal decisions with ML. Additionally, we propose a novel ML method for optimal decision-making under capacity constraints based on learning to rank. To make effective decisions, a decision-maker has to estimate the causal effect of possible interventions in order to choose actions that achieve the desired outcome. Unfortunately, standard ML models identify correlations in the data instead of causal relationships. Because of this, these models cannot guarantee the effectiveness of decisions made based on their predictions. Causal inference provides a formal framework for reasoning about causality and identifying causal effects from data. This dissertation explores the intersection of causality and ML. First, we illustrate the potential of causal ML for optimizing preventive maintenance. Next, we propose novel causal ML methods for predicting causal effect distributions and for addressing informative sampling when predicting treatment outcomes over time. We also argue for a practical, end-to-end perspective for building ML pipelines for causal inference and propose an automated framework doing so. Finally, we combine decision-focused learning with causal inference by introducing ranking metalearners to optimize treatment decisions under capacity constraints.

Propofol is a widely used sedative-hypnotic agent for critically-ill patients requiring invasive mechanical ventilation (IMV). Despite its clinical benefits, propofol is associated with increased risks of hypertriglyceridemia. Early identification of patients at risk for propofol-associated hypertriglyceridemia is crucial for optimizing sedation strategies and preventing adverse outcomes. Machine learning (ML) models offer a promising approach for predicting individualized patient risks of propofol-associated hypertriglyceridemia. We propose the development of a ML model aimed at predicting the risk of propofol-associated hypertriglyceridemia in ICU patients receiving IMV. The study will utilize retrospective data from four Mayo Clinic sites. Nested cross-validation (CV) will be employed, with a 10-fold inner CV loop for model tuning and selection as well as an outer loop using leave-one-site-out CV for external validation. Feature selection will be conducted using Boruta and LASSO-penalized logistic regression. Data preprocessing steps include missing data imputation, feature scaling, and dimensionality reduction techniques. Six ML algorithms will be tuned and evaluated. Bayesian optimization will be used for hyperparameter selection. Global model explainability will be assessed using permutation importance, and local model explainability will be assessed using SHapley Additive exPlanations (SHAP). The proposed ML model aims to provide a reliable and interpretable tool for clinicians to predict the risk of propofol-associated hypertriglyceridemia in ICU patients. The final model will be deployed in a web-based clinical risk calculator. The model development process and performance measures obtained during nested cross-validation will be described in a study publication to be disseminated in a peer-reviewed journal. The proposed study has received ethics approval from the Mayo Clinic Institutional Review Board (IRB #23-007416). Robust external validation using a nested cross-validation (CV) framework will help assess the generalizability of models produced from the modeling pipeline across different hospital settings.A diverse set of machine learning (ML) algorithms and advanced hyperparameter tuning techniques will be employed to identify the most optimal model configuration.Integration of feature explainability will enhance the clinical applicability of the ML models by providing transparency in predictions, which can improve clinician trust and encourage adoption.Reliance on retrospective data may introduce biases due to inconsistent or erroneous data collection, and the computational intensity of the validation approach may limit replication and future model expansion in resource-constrained settings.

Stream temperature (Ts) is an important water quality parameter that affects ecosystem health and human water use for beneficial purposes. Accurate Ts predictions at different spatial and temporal scales can inform water management decisions that account for the effects of changing climate and extreme events. In particular, widespread predictions of Ts in unmonitored stream reaches can enable decision makers to be responsive to changes caused by unforeseen disturbances. In this study, we demonstrate the use of classical machine learning (ML) models, support vector regression and gradient boosted trees (XGBoost), for monthly Ts predictions in 78 pristine and human-impacted catchments of the Mid-Atlantic and Pacific Northwest hydrologic regions spanning different geologies, climate, and land use. The ML models were trained using long-term monitoring data from 1980–2020 for three scenarios: (1) temporal predictions at a single site, (2) temporal predictions for multiple sites within a region, and (3) spatiotemporal predictions in unmonitored basins (PUB). In the first two scenarios, the ML models predicted Ts with median root mean squared errors (RMSE) of 0.69–0.84 °C and 0.92–1.02 °C across different model types for the temporal predictions at single and multiple sites respectively. For the PUB scenario, we used a bootstrap aggregation approach using models trained with different subsets of data, for which an ensemble XGBoost implementation outperformed all other modeling configurations (median RMSE 0.62 °C).The ML models improved median monthly Ts estimates compared to baseline statistical multi-linear regression models by 15–48% depending on the site and scenario. Air temperature was found to be the primary driver of monthly Ts for all sites, with secondary influence of month of the year (seasonality) and solar radiation, while discharge was a significant predictor at only 10 sites. The predictive performance of the ML models was robust to configuration changes in model setup and inputs, but was influenced by the distance to the nearest dam with RMSE <1 °C at sites situated greater than 16 and 44 km from a dam for the temporal single site and regional scenarios, and over 1.4 km from a dam for the PUB scenario. Our results show that classical ML models with solely meteorological inputs can be used for spatial and temporal predictions of monthly Ts in pristine and managed basins with reasonable (<1 °C) accuracy for most locations.