Performance Maintenance of Machine Learning-based Emergency Patient Mortality Predictive Models

Performance evaluation of machine learning techniques in predicting cumulative absolute velocity

Sepsis is a leading cause of death in patients with trauma, and the risk of mortality increases significantly for each hour of delay in treatment. A hypermetabolic baseline and explosive inflammatory immune response mask clinical signs and symptoms of sepsis in trauma patients, making early diagnosis of sepsis more challenging. Machine learning-based predictive modeling has shown great promise in evaluating and predicting sepsis risk in the general intensive care unit (ICU) setting, but there has been no sepsis prediction model specifically developed for trauma patients so far. To develop a machine learning model to predict the risk of sepsis at an hourly scale among ICU-admitted trauma patients. We extracted data from adult trauma patients admitted to the ICU at Beth Israel Deaconess Medical Center between 2008 and 2019. A total of 42 raw variables were collected, including demographics, vital signs, arterial blood gas, and laboratory tests. We further derived a total of 485 features, including measurement pattern features, scoring features, and time-series variables, from the raw variables by feature engineering. The data set was randomly split into 70% for model development with stratified 5-fold cross-validation, 15% for calibration, and 15% for testing. An Extreme Gradient Boosting (XGBoost) model was developed to predict the hourly risk of sepsis at prediction windows of 4, 6, 8, 12, and 24 hours. We evaluated model performance for discrimination and calibration both at time-step and outcome levels. Clinical applicability of the model was evaluated with varying levels of precision, and the potential clinical net benefit was assessed with decision curve analysis (DCA). A Shapley additive explanation algorithm was applied to show the effect of features on the prediction model. In addition, we trained an L2-regularized logistic regression model to compare its performance with XGBoost. We included 4603 trauma patients in the study, 1196 (26%) of whom developed sepsis. The XGBoost model achieved an area under the receiver operating characteristics curve (AUROC) ranging from 0.83 to 0.88 at the 4-to-24-hour prediction window in the test set. With a ratio of 9 false alerts for every true alert, it predicted 73% (386/529) of sepsis-positive timesteps and 91% (163/179) of sepsis events in the subsequent 6 hours. The DCA showed our model had a positive net benefit in the threshold probability range of 0 to 0.6. In comparison, the logistic regression model achieved lower performance, with AUROC ranging from 0.76 to 0.84 at the 4-to-24-hour prediction window. The machine learning-based model had good discrimination and calibration performance for sepsis prediction in critical trauma patients. Using the model in clinical practice might help to identify patients at risk of sepsis in a time window that enables personalized intervention and early treatment.

BackgroundA brief gonadotropin-releasing hormone analogues (GnRHa) stimulation test which solely focused on LH 30-minute post-stimulation was considered to identify girls with central precocious puberty (CPP). However, it was tested using traditional statistical methods. With advanced computer science, we aimed to develop a machine learning-based diagnostic model that processed baseline CPP-related variables and a brief GnRHa stimulation test for CPP diagnosis.MethodsWe recruited girls suspected of precocious puberty and underwent a GnRHa stimulation test at Children Hospital 2, Vietnam, and Cathay General Hospital, Taiwan. Clinical data, bone age measurement, and 30-min post-stimulation blood test were used to build up the predictive model. The candidate model was developed by different machine learning algorithms that were mainly evaluated by sensitivity, specificity, the area under the receiver operator characteristic curve (AUC), and F1-score in internal and external validation data to classify girls as CPP and non-CPP at different time-points (0-min, 30-min, 60-min, and 120-min post-stimulation).ResultsAmong the 614 girls diagnosed with PP, 524 (85.3%) had CPP. The random forest algorithm yielded the highest value of F1-score (0.976), specificity (0.893), positive predicted value (0.987), and relatively high value of AUC (0.972) that contributed to high probability to identify CPP. The performance metrics of the 30-min post-stimulation diagnostic model including sensitivity and specificity surpassed those of the 0-minute model (0-min) and were equivalent to those of the model obtained 60-min and 120-min post-stimulation. Hence, our machine learning-based model helps shorten the stimulation test to 30 minutes after GnRHa injection, in general, it requires 120 minutes for a completed GnRHa stimulation test.ConclusionsWe developed a diagnostic model based on clinical features and a single sample 30-minute post-stimulation to identify CPP in girls that can reduce distress for children caused by multiple blood samplings.

Background: A mechanism to predict graft failure before the actual kidney transplantation occurs is crucial to clinical management of chronic kidney disease patients. Several kidney graft outcome prediction models, developed using machine learning methods, are available in the literature. However, most of those models used small datasets and none of the machine learning-based prediction models available in the medical literature modelled time-to-event (survival) information, but instead used the binary outcome of failure or not. The objective of this study is to develop two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using time-to-event data in a large national dataset from Australia. Methods: The dataset provided by the Australia and New Zealand Dialysis and Transplant Registry will be used for the analysis. This retrospective dataset contains the cohort of patients who underwent a kidney transplant in Australia from January 1 st, 2007, to December 31 st, 2017. This included 3,758 live donor transplants and 7,365 deceased donor transplants. Three machine learning methods (survival tree, random survival forest and survival support vector machine) and one traditional regression method, Cox proportional regression, will be used to develop the two predictive models. The best predictive model will be selected based on the model's performance. Discussion: This protocol describes the development of two separate machine learning-based predictive models to predict graft failure following live and deceased donor kidney transplant, using a large national dataset from Australia. Furthermore, these two models will be the most comprehensive kidney graft failure predictive models that have used survival data to model using machine learning techniques. Thus, these models are expected to provide valuable insight into the complex interactions between graft failure and donor and recipient characteristics.

Background: Although transplantation of patch-shaped autologous skeletal muscle-derived cells has been introduced to ischemic cardiomyopathy (ICM) or non-ICM patients, we found responders or non-responders to this treatment exist and how to predict the responsiveness to this treatment is crucial for the improvement of the effectiveness. In this study, we revealed the clinical features associated with the responders using a machine learning based-model, which was discriminated between the responder and the non-responders in this treatment. Methods and Results: We used the retrospective databases of 23 ICM patients and 23 non-ICM patients undergoing autologous myoblast patch transplantation to develop machine learning models to discriminate 3-year VAD free survival. Sixty-nine pre-transplantation clinical futures were selected to train the models. In ICM models, there were 4 VADs or deaths, and in non-ICM models, there were 10 VADs or deaths during the 3-year follow-up periods. Using these databases, we trained multiple machine learning models and evaluated the models with the leave-one-out method. In ICM, k nearest neighbor demonstrated the best performance, showing the accuracy was 95.7%, and the AUC was 0.95. The features associated with 3-year VAD free survival in ICM were NYHA classification, cardiac index, and left ventricular ejection fraction. In non-ICM, k nearest neighbor demonstrated the best performance among the trained classifiers, demonstrating the accuracy was 95.7%, and the AUC was 0.93. The features associated with 3-year VAD free survival in non-ICM were pulmonary capillary wedge pressure, pulmonary vascular resistance, and albumin. Conclusion: We found the features associated with 3-year VAD free survival in autologous myoblast patch transplantation in ICM and non-ICM. Focusing on these features may facilitate optimal candidate selection in ICM and non-ICM for regenerative therapy.

BackgroundThe large number of SARS-Cov-2 cases during the COVID-19 global pandemic has burdened healthcare systems and created a shortage of resources and services. In recent years, mortality prediction models have shown a potential in alleviating this issue; however, these models are susceptible to biases in specific subpopulations with different risks of mortality, such as patients with prior history of smoking. The current study aims to develop a machine learning-based mortality prediction model for COVID-19 patients that have a history of smoking in the Iranian population.MethodsA retrospective study was conducted across six medical centers between 18 and 2020 and 15 March 2022, comprised of 678 CT scans and laboratory-confirmed COVID-19 patients that had a history of smoking. Multiple machine learning models were developed using 10-fold cross-validation. The target variable was in-hospital mortality and input features included patient demographics, levels of care, vital signs, medications, and comorbidities. Two sets of models were developed for at-admission and post-admission predictions. Subsequently, the top five prediction models were selected from at-admission models and post-admission models and their probabilities were calibrated.ResultsThe in-hospital mortality rate for smoker COVID-19 patients was 20.1%. For “at admission” models, the best-calibrated model was XGBoost which yielded an accuracy of 87.5% and F1 score of 86.2%. For the “post-admission” models, XGBoost also outperformed the rest with an accuracy of 90.5% and F1 score of 89.9%. Active smoking was among the most important features in patients’ mortality prediction.ConclusionOur machine learning-based mortality prediction models have the potential to be adapted for improving the management of smoker COVID-19 patients and predicting patients’ chance of survival.

One of core technologies for prognostics is to predict failures before they occur and estimate time to failure (TTF) by using built-in predictive models. The predictive model could be either physics-based model or machine learning-based model. Machine learning-based predictive modeling is an emerging application of machine learning to machinery maintenance. Accurate TTF estimation could help performing predictive action “just-in-time”. However, the developed predictive models sometimes fail to provide a precise TTF estimate. To address this issue, we propose a Particle Filtering (PF)-based method to estimate TTF. After introducing the PF-based algorithm, we present the implementation along with the experimental results obtained from a case study of Auxiliary Power Unit (APU) prognostics. To our best knowledge, this is the first application of PF-based method to APU prognostic. The results demonstrated that the PF-based method is useful for estimating TTF for predictive maintenance and it greatly improved TTF estimation precision for APU prognostics.

Background and study aims The aim of this study was to develop a machine learning-based model to classify bowel cleansing quality and to test this model in comparison to a pixel analysis model and assessments by four colon capsule endoscopy (CCE) readers.Methods A pixel analysis and a machine learning-based model with four cleanliness classes (unacceptable, poor, fair and good) were developed to classify CCE videos. Cleansing assessments by four CCE readers in 41 videos from a previous study were compared to the results both models yielded in this pilot study.Results The machine learning-based model classified 47 % of the videos in agreement with the averaged classification by CCE readers, as compared to 32 % by the pixel analysis model. A difference of more than one class was detected in 12 % of the videos by the machine learning-based model and in 32 % by the pixel analysis model, as the latter tended to overestimate cleansing quality. A specific analysis of unacceptable videos found that the pixel analysis model classified almost all of them as fair or good, whereas the machine learning-based model identified five out of 11 videos in agreement with at least one CCE reader as unacceptable.Conclusions The machine learning-based model was superior to the pixel analysis in classifying bowel cleansing quality, due to a higher sensitivity to unacceptable and poor cleansing quality. The machine learning-based model can be further improved by coming to a consensus on how to classify cleanliness of a complete CCE video, by means of an expert panel.

Background Patients classified as high bleeding risk (HBR) undergoing percutaneous coronary intervention (PCI) face a significantly higher incidence of net adverse clinical events (NACE) compared to non-HBR patients. Existing risk assessment models, such as the CRUSADE and TIMI scores, do not adequately address the unique risks faced by the HBR population. There is an urgent need for a precise and comprehensive predictive model tailored to PCI-HBR patients to guide clinical decision-making and improve patient outcomes. Methods This study aimed to develop a machine learning-based predictive model for long-term NACE in PCI-HBR patients. We utilized data from the Prognostic Analysis and an Appropriate Antiplatelet Strategy for Patients with Percutaneous Coronary Intervention and High Bleeding Risk (PPP-PCI) registry database. Feature selection and interpretation were performed using a SHAP (SHapley Additive exPlanations) model based on Recursive Feature Elimination (RFE). Model construction and evaluation were conducted using four algorithms: logistic regression, random forest, gradient boosting, and XGBoost. Results A total of 1512 PCI-HBR patients were included in the study. The XGBoost model demonstrated the highest predictive performance, achieving an area under the receiver operating characteristic curve (AUC) of 0.85. The SHAP model identified 24 significant variables contributing to the prediction of NACE, including clinical parameters, laboratory findings, and echocardiographic data. Conclusions Our machine learning-based model offers a promising tool for predicting long-term NACE in PCI-HBR patients. The model’s high predictive accuracy and interpretability have the potential to enhance clinical decision-making and improve patient care. Further validation in larger, diverse populations is warranted to confirm these findings.SHAP summary plot. AUC plot.

IntroductionInternational guidelines recommend single or hypo-fraction for palliative radiotherapy (PRT). However, due to the inherent difficulty in accurately predicting life expectancy, the customization of end-of-life care for individual patients poses a significant challenge. This study aims to develop a machine learning-based mortality prediction model for PRT tailored to life expectancy.MethodsA retrospective analysis encompassed 318 patients who expired after receiving PRT for advanced cancer from March 2013 to July 2023. Various model algorithms employing 22 variables were used to predict mortality within 30 days from initiation of PRT: extra trees, random forest, light gradient boosting machine (LightGBM), and extreme gradient boosting (XGBoost). We evaluated each model's performance using accuracy, precision, recall, specificity, and F1 score.ResultsA total of 302 patients were analyzed after excluding those with missing data. The <30-day mortality group had significantly worse Eastern Cooperative Oncology Group (ECOG) status, lower albumin, higher neutrophil‑to‑lymphocyte ratio, and lower lymphocyte count (all p<0.05). No significant multicollinearity was observed. Among all models, LightGBM showed the best performance (accuracy: 0.725, F1-score: 0.720). A minimal variable model (MVM) using the top eight features plus sex achieved comparable performance to the full variable model (FVM), with improved recall and reduced complexity.ConclusionWe developed a machine learning-based mortality prediction model for tailoring PRT to life expectancy. The model identified clinically meaningful predictors that reflect patient condition and tumor burden. The MVM demonstrated a comparable performance to the FVM, suggesting its potential utility as an interpretable and practical clinical decision support tool for individualized end-of-life care planning.

Antenatal depression (AND), occurring during pregnancy, is associated with severe outcomes. However, there is a lack of objective and universally applicable prediction methods for AND in clinical practice. We leveraged sociodemographic and pregnancy-related data to develop and validate a machine learning-based AND prediction model. Data from 20,950 pregnant women form 3 hospitals were used and divided into training and test sets. AND was defined as an EPDS score of 10 or above. Using machine learning, we selected 34 characteristic variables and divided them into three categories based on clinical practice: Base Variables, General Variables, and Obstetric Variables. Based on this classification, we constructed four different AND random forest prediction models: the Base Model, the Base+General Model, the Base+Obstetric Model, and the Full Model. The AUC range in the test set was 0.687-0.710. The Base+General Model achieved the best performance with an AUC of 0.710 (95% CI: 0.693-0.710) in predicting AND risk during the late pregnancy period. The AUC of the Base Model was only 0.022 lower than that of the top-performing model, indicating its solid foundation for early AND screening. We have only analyzed the dataset from two eastern cities, and have not yet validated our models in an external dataset. Machine learning-based prediction models offer the capability to anticipate the risk of AND across different pregnancy stages. This enables the earlier and more accurate identification of pregnant women who may be at risk, facilitating timely interventions for improving outcomes for both mothers and their offspring.

To propose a machine learning-based method for predicting the trajectories during manual acupuncture manipulation (MAM), aiming to improve the precision and consistency of acupuncture practitioner' operation and provide the real-time suggestions on MAM error correction. Computer vision technology was used to analyze the hand micromotion when holding needle during acupuncture, and provide a three-dimensional coordinate description method of the index finger joints of the holding hand. Focusing on the 4 typical motions of MAM, a machine learning-based MAM trajectory predictive model was designed. By integrating the changes of phalangeal joint angle and hand skeletal information of acupuncture practitioner, the motion trajectory of the index finger joint was predicted accurately. Besides, the roles of machine learning-based MAM trajectory predictive model in the skill transmission of acupuncture manipulation were verified by stratified randomized controlled trial. The performance of MAM trajectory predictive model, based on the long short-term memory network (LSTM), obtained the highest stability and precision, up to 98%. The learning effect was improved when the model applied to the skill transmission of acupuncture manipulation. The machine learning-based MAM predictive model provides acupuncture practitioner with precise action prediction and feedback. It is valuable and significant for the inheritance and error correction of manual operation of acupuncture.

This study aimed to develop and validate a machine learning-based predictive model for individualized assessment and management of chemotherapy-induced myelosuppression (CIM) in patients with colorectal cancer (CRC). A total of 450 patients with CRC undergoing chemotherapy were retrospectively enrolled in the training cohort, and an additional 150 patients were included for external validation. Feature selection was performed using the Least Absolute Shrinkage and Selection Operator (LASSO) regression. Three machine learning algorithms [Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XGBoost)] were applied to construct predictive models. Model performance was assessed using multiple metrics, including accuracy, area under the receiver operating characteristic curve (AUC), F1 score, sensitivity, and specificity. The SHapley Additive exPlanations (SHAP) method was employed to rank and interpret the importance of predictive features. In the training cohort, 52.4% of patients developed CIM. The feature selection process identified 19 significant variables that were incorporated into the predictive models. Both LR and RF demonstrated optimal performance, with an AUC of 0.83 and an accuracy of 0.76 in the training set. In the test set, RF continued to outperform other models, achieving an AUC of 0.77 and an accuracy of 0.71. External validation confirmed the robustness of the RF model, which achieved an AUC of 0.93 (95% CI: 0.89-0.97), an accuracy of 0.89, a sensitivity of 0.86, and a specificity of 0.92. SHAP analysis revealed that the most important predictors included hematological parameters, nutritional risk score (NRS2002), and a history of radiotherapy. The RF-based machine learning model demonstrated high accuracy and strong external validation capability for predicting the risk of CIM in CRC patients.

BACKGROUNDSevere dengue children with critical complications have been attributed to high mortality rates, varying from approximately 1% to over 20%. To date, there is a lack of data on machine-learning-based algorithms for predicting the risk of in-hospital mortality in children with dengue shock syndrome (DSS).AIMTo develop machine-learning models to estimate the risk of death in hospitalized children with DSS.METHODSThis single-center retrospective study was conducted at tertiary Children’s Hospital No. 2 in Viet Nam, between 2013 and 2022. The primary outcome was the in-hospital mortality rate in children with DSS admitted to the pediatric intensive care unit (PICU). Nine significant features were predetermined for further analysis using machine learning models. An oversampling method was used to enhance the model performance. Supervised models, including logistic regression, Naïve Bayes, Random Forest (RF), K-nearest neighbors, Decision Tree and Extreme Gradient Boosting (XGBoost), were employed to develop predictive models. The Shapley Additive Explanation was used to determine the degree of contribution of the features.RESULTSIn total, 1278 PICU-admitted children with complete data were included in the analysis. The median patient age was 8.1 years (interquartile range: 5.4-10.7). Thirty-nine patients (3%) died. The RF and XGboost models demonstrated the highest performance. The Shapley Addictive Explanations model revealed that the most important predictive features included younger age, female patients, presence of underlying diseases, severe transaminitis, severe bleeding, low platelet counts requiring platelet transfusion, elevated levels of international normalized ratio, blood lactate and serum creatinine, large volume of resuscitation fluid and a high vasoactive inotropic score (> 30).CONCLUSIONWe developed robust machine learning-based models to estimate the risk of death in hospitalized children with DSS. The study findings are applicable to the design of management schemes to enhance survival outcomes of patients with DSS.

Septic patients in the intensive care unit (ICU) often develop sepsis-associated delirium (SAD), which is strongly associated with poor prognosis. The aim of this study is to develop a machine learning-based model for the early prediction of SAD. Patient data were extracted from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database and the eICU Collaborative Research Database (eICU-CRD). The MIMIC-IV data were divided into a training set and an internal validation set, while the eICU-CRD data served as an external validation set. Feature variables were selected using least absolute shrinkage and selection operator regression, and prediction models were built using logistic regression, support vector machines, decision trees, random forests, extreme gradient boosting (XGBoost), k-nearest neighbors and naive Bayes methods. The performance of the models was evaluated in the validation set. The model was also applied to a group of patients who were not assessed or could not be assessed for delirium. The MIMIC-IV and eICU-CRD databases included 14,620 and 1723 patients, respectively, with a median time to diagnosis of SAD of 24 and 30 h. Compared with Non-SAD patients, SAD patients had higher 28-days ICU mortality rates and longer ICU stays. Among the models compared, the XGBoost model had the best performance and was selected as the final model (internal validation area under the receiver operating characteristic curves (AUROC) = 0.793, external validation AUROC = 0.701). The XGBoost model outperformed other models in predicting SAD. The establishment of this predictive model allows for earlier prediction of SAD compared to traditional delirium assessments and is applicable to patients who are difficult to assess with traditional methods.