Predicting Postpartum Hemorrhage Using Clinical Features Extracted With Large Language Models

Electronic health records (EHRs) with clinical decision support tools are now ubiquitous in healthcare organizations. Clinical foundation models (CFMs) pretrained on large-scale, heterogeneous structured EHR data have emerged as a powerful approach to improve predictive performance and generalizability. Meanwhile, large language models (LLMs) pretrained on broad data sources are being applied to an expanding range of healthcare tasks. However, it remains unclear whether generalist LLMs can match specialized CFMs for disease risk prediction using structured clinical data. We compared CFMs (Med-BERT, CLMBR) against fine-tuned generalist LLMs (Mistral, LLaMA-2/3/3.1), a clinical LLM (Me-LLaMA), and LLM-generated embeddings paired with simple classifiers (using DeepSeek, Qwen3, and GPT-OSS) on two disease risk prediction tasks: heart failure risk among diabetic patients (DHF) and pancreatic cancer diagnosis (PaCa). Evaluations spanned multi-site EHR data, claims data, and an open-source single-institution benchmark (EHRSHOT). Performance was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). On larger EHR and claims cohorts (>30,000 patients), fine-tuned CFMs outperformed fine-tuned LLMs by a small but statistically significant margin (<1% AUROC). The clinical LLM performed comparably to generalist LLMs despite being smaller. On the open-source PaCa cohort (3,810 patients, 199 cases), LLMs achieved slightly higher AUROCs that were not statistically significant (LLaMA-3.1-70B 86.1% vs. Med-BERT 85.3%, p=0.27), but CFMs achieved significantly higher AUPRC (Med-BERT 55.9% vs. LLaMA-3.1-70B 41.1%, p=0.001). Notably, LLM-generated trajectory embeddings paired with logistic regression or a simple MLP, without any LLM fine-tuning, achieved the best overall performance, with AUROC exceeding 90% (Qwen3) and AUPRC reaching 66% (GPT-OSS 20B). LLM-generated embeddings with lightweight classifiers outperformed both fine-tuned CFMs and fine-tuned LLMs on AUROC and AUPRC. While these results demonstrate the potential of generalist models to match or surpass specialized CFMs, their substantially greater computational cost and variable AUPRC performance in the fine-tuning setting warrant caution. We provide a reproducible evaluation framework and codebase to support continued benchmarking.

BackgroundTreatment of sepsis requires prompt initiation of appropriate empiric antibiotics. Coverage for Gram-negative bacilli (GNB) is decided based on risk factors for resistant organisms, as documented in both structured and unstructured data in the electronic health record. We hypothesized that adding large language models (LLMs) that can decode clinical narratives would improve performance of deep learning models in predicting antimicrobial resistance in GNB sepsis.MethodsThis retrospective cohort analysis included all adult patients who met CDC sepsis criteria admitted to 10 BJC hospitals from 1/2018 to 12/2023. Sepsis episodes were classified as community-onset (< 48hrs from hospitalization) or hospital-onset. GNB were stratified as ceftriaxone-susceptible (SS), ceftriaxone-resistant but cefepime-susceptible (RS), and ceftriaxone- and cefepime-resistant (RR). Culture negative or non-GNB sepsis were labeled SS. Deep learning models (XGBoost, DeepAMR) were developed using all available structured data (e.g., demographics, vitals, labs). LLMs (BioGPT, Medical-LLaMa3-8B) were developed on full or partial history and physical exam notes. Models were assessed using area under the receiver operating characteristic curve (AUROC) and area under precision-recall curve (AUPRC).ResultsA total of 29,517 patients were included, for 23,949 community-onset and 25,101 hospital-onset sepsis episodes with RS+RR GNB in 9% and 17%, respectively (Table 1). Relevant features of structured data models included history of prior infections and resistant organisms. DeepAMR was the best performing deep learning model with AUROC 0.85/AUPRC 0.40 and AUROC 0.81/AUPRC 0.50 for community- and hospital-onset sepsis, respectively (Table 2). The best performing LLM was Medical-LLaMa3-8B on full notes (Figure 1), with AUROC 0.74/AUPRC 0.21 and AUROC 0.67/AUPRC 0.30 for community- and hospital-onset sepsis, respectively. Combining deep learning and LLMs did not significantly improve the prediction of resistant infections compared to deep learning alone.ConclusionStructured medical data provided the highest prediction of resistant GNB infections in sepsis. More analysis is needed to understand why the unstructured data did not improve model predictive performance.DisclosuresMaria Cristina Vazquez Guillamet, MD, Bionano: Stocks/Bonds (Public Company)|Charisma: Stocks/Bonds (Public Company)|Ocugen: Stocks/Bonds (Public Company)

Introduction: Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other. Method: This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and as if required, arbitration by a third researcher. Results: There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35–1.00, F1-score 0.16–0.99, area under the precision-recall curve (AUPRC) 0.51–1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27. Conclusion: A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

Introduction: Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other. Method: This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and as if required, arbitration by a third researcher. Results: There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35–1.00, F1-score 0.16–0.99, area under the precision-recall curve (AUPRC) 0.51–1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27. Conclusion: A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

Hypoxemia is a common complication associated with anesthesia in painless gastroscopy. With the aging of the social population, the number of cases of hypoxemia among middle-aged and elderly patients is increasing. However, tools for predicting hypoxemia in middle-aged and elderly patients are lacking. In this study, we investigated the risk factors for hypoxemia in middle-aged and elderly outpatients undergoing painless gastroscopy based on machine learning and constructed a risk prediction model. In this retrospective study, we included the data on 1,348 outpatients undergoing painless gastroscopy. In total, 26 characteristic variables, including demographic information, past medical history, and clinical data of the patients were included, and BorutaShap was used for feature selection. Five machine learning algorithm models, including logistic regression (LR), support vector machine (SVM), random forest (RF), extreme gradient boosting (XGB), and light gradient boosting machine (LightGBM), were selected. The best models were selected based on the area under the receiver operating characteristic curve (AUROC). Model feature importance was explained and analyzed using Shapley Additive Explanations (SHAP). The endpoint event of this study was considered to be hypoxemia during the procedure, defined as at least one occurrence of pulse oxygen saturation below 90% without probe misalignment or interference from the beginning of anesthesia induction to the end of painless gastroscopy. In the final cohort of 984 patients, 11% of patients (108/984) experienced hypoxemia during the painless gastroscopy procedure. The AUROCs of the five models were as follows: Logistic Regression (AUROC = 0.893, 95CI: 0.881–0.899), SVM (AUROC = 0.855, 95CI: 0.812–0.884), Random Forest (AUROC = 0.914, 95CI: 0.889–0.924), XGB (AUROC = 0.902, 95CI: 0.865–0.919), and LightGBM (AUROC = 0.891, 95CI: 0.847–0.917). Regarding the explanation of the importance of SHAP features, preoperative variables (baseline SpO2, body mass index, and micrognathia) and intraoperative variables (operating time of gastroscopy, induction dose of etomidate and propofol mixture, append anesthetic, cough, and repeated pharyngeal irritation) significantly contributed to the model. We identified eight potential risk factors related to the occurrence of hypoxemia in middle-aged and elderly patients undergoing painless gastroscopy, based on machine learning feature engineering. Among the five machine learning algorithms, RF exhibited the best predictive performance in the internal test set and had a certain degree of generalization ability in the external validation set, which indicated that the RF model was more suitable for the data framework of this study. This model was more likely to enhance the accuracy of hypoxemia prediction in middle-aged and elderly patients undergoing painless gastroscopy, and thus, it is suitable for assisting anesthesiologists in clinical decision-making.

BackgroundAcute kidney disease (AKD) affects more than half of critically ill elderly patients with acute kidney injury (AKI), which leads to worse short-term outcomes.ObjectiveWe aimed to establish 2 machine learning models to predict the risk and prognosis of AKD in the elderly and to deploy the models as online apps.MethodsData on elderly patients with AKI (n=3542) and AKD (n=2661) from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database were used to develop 2 models for predicting the AKD risk and in-hospital mortality, respectively. Data collected from Xiangya Hospital of Central South University were for external validation. A bootstrap method was used for internal validation to obtain relatively stable results. We extracted the indicators within 24 hours of the first diagnosis of AKI and the fluctuation range of some indicators, namely delta (day 3 after AKI minus day 1), as features. Six machine learning algorithms were used for modeling; the area under the receiver operating characteristic curve (AUROC), decision curve analysis, and calibration curve for evaluating; Shapley additive explanation (SHAP) analysis for visually interpreting; and the Heroku platform for deploying the best-performing models as web-based apps.ResultsFor the model of predicting the risk of AKD in elderly patients with AKI during hospitalization, the Light Gradient Boosting Machine (LightGBM) showed the best overall performance in the training (AUROC=0.844, 95% CI 0.831-0.857), internal validation (AUROC=0.853, 95% CI 0.841-0.865), and external (AUROC=0.755, 95% CI 0.699–0.811) cohorts. In addition, LightGBM performed well for the AKD prognostic prediction in the training (AUROC=0.861, 95% CI 0.843-0.878), internal validation (AUROC=0.868, 95% CI 0.851-0.885), and external (AUROC=0.746, 95% CI 0.673-0.820) cohorts. The models deployed as online prediction apps allowed users to predict and provide feedback to submit new data for model iteration. In the importance ranking and correlation visualization of the model’s top 10 influencing factors conducted based on the SHAP value, partial dependence plots revealed the optimal cutoff of some interventionable indicators. The top 5 factors predicting the risk of AKD were creatinine on day 3, sepsis, delta blood urea nitrogen (BUN), diastolic blood pressure (DBP), and heart rate, while the top 5 factors determining in-hospital mortality were age, BUN on day 1, vasopressor use, BUN on day 3, and partial pressure of carbon dioxide (PaCO2).ConclusionsWe developed and validated 2 online apps for predicting the risk of AKD and its prognostic mortality in elderly patients, respectively. The top 10 factors that influenced the AKD risk and mortality during hospitalization were identified and explained visually, which might provide useful applications for intelligent management and suggestions for future prospective research.

BackgroundA predictive model for risk of Mycoplasma pneumoniae (MP)-related hepatitis in MP pneumonia pediatric patients can improve treatment selection and therapeutic effect. However, currently, no predictive model is available.MethodsThree hundred seventy-four pneumonia pediatric patients with/without serologically-confirmed MP infection and ninety-three health controls were enrolled. Logistic regressions were performed to identify the determinant variables and develop predictive model. Predictive performance and optimal diagnostic threshold were evaluated using area under the receiver operating characteristic curve (AUROC). Stratification analysis by age and MP-IgM titer was used to optimize model’s clinical utility. An external validation set, including 84 MP pneumonia pediatric patients, was used to verify the predictive efficiency. After univariate analysis to screen significant variables, monocyte count (MO), erythrocyte distribution width (RDW) and platelet count (PLT) were identified as independent predictors in multivariate analysis.ResultsWe constructed MRP model: MO [^109/L] × 4 + RDW [%] – PLT [^109/L] × 0.01. MRP achieved an AUROC of 0.754 and the sensitivity and specificity at cut-off value 10.44 were 71.72 and 61.00 %, respectively in predicting MP-related hepatitis from MP pneumonia. These results were verified by the external validation set, whereas it merely achieved an AUROC of 0.540 in pneumonia without MP infection. The AUROC of MRP was 0.812 and 0.787 in infants and toddlers (0–36 months) and low MP-IgM titer subgroup (1:160–1:320), respectively. It can achieve an AUROC of 0.804 in infants and toddler with low MP-IgM titer subgroup.ConclusionsMRP is an effective predictive model for risk of MP-related hepatitis in MP pneumonia pediatric patients, especially infants and toddlers with low MP-IgM titer.

BackgroundFetal alcohol syndrome (FAS) is a lifelong developmental disability that occurs among individuals with prenatal alcohol exposure (PAE). With improved prediction models, FAS can be diagnosed or treated early, if not completely prevented.ObjectiveIn this study, we sought to compare different machine learning algorithms and their FAS predictive performance among women who consumed alcohol during pregnancy. We also aimed to identify which variables (eg, timing of exposure to alcohol during pregnancy and type of alcohol consumed) were most influential in generating an accurate model.MethodsData from the collaborative initiative on fetal alcohol spectrum disorders from 2007 to 2017 were used to gather information about 595 women who consumed alcohol during pregnancy at 5 hospital sites around the United States. To obtain information about PAE, questionnaires or in-person interviews, as well as reviews of medical, legal, or social service records were used to gather information about alcohol consumption. Four different machine learning algorithms (logistic regression, XGBoost, light gradient-boosting machine, and CatBoost) were trained to predict the prevalence of FAS at birth, and model performance was measured by analyzing the area under the receiver operating characteristics curve (AUROC). Of the total cases, 80% were randomly selected for training, while 20% remained as test data sets for predicting FAS. Feature importance was also analyzed using Shapley values for the best-performing algorithm.ResultsOverall, there were 20 cases of FAS within a total population of 595 individuals with PAE. Most of the drinking occurred in the first trimester only (n=491) or throughout all 3 trimesters (n=95); however, there were also reports of drinking in the first and second trimesters only (n=8), and 1 case of drinking in the third trimester only (n=1). The CatBoost method delivered the best performance in terms of AUROC (0.92) and area under the precision-recall curve (AUPRC 0.51), followed by the logistic regression method (AUROC 0.90; AUPRC 0.59), the light gradient-boosting machine (AUROC 0.89; AUPRC 0.52), and XGBoost (AUROC 0.86; AURPC 0.45). Shapley values in the CatBoost model revealed that 12 variables were considered important in FAS prediction, with drinking throughout all 3 trimesters of pregnancy, maternal age, race, and type of alcoholic beverage consumed (eg, beer, wine, or liquor) scoring highly in overall feature importance. For most predictive measures, the best performance was obtained by the CatBoost algorithm, with an AUROC of 0.92, precision of 0.50, specificity of 0.29, F1 score of 0.29, and accuracy of 0.96.ConclusionsMachine learning algorithms were able to identify FAS risk with a prediction performance higher than that of previous models among pregnant drinkers. For small training sets, which are common with FAS, boosting mechanisms like CatBoost may help alleviate certain problems associated with data imbalances and difficulties in optimization or generalization.

Background and AimsThe 12-lead electrocardiogram (ECG) remains a cornerstone of cardiac diagnostics, yet existing artificial intelligence (AI) solutions for automated interpretation often lack generalizability, remain closed source, and are primarily trained using supervised learning (SL), which requires extensive labelled datasets and may limit adaptability across diverse clinical settings. Self-supervised learning (SSL) can potentially overcome these limitations by learning robust representations from unlabelled data. To address these challenges, this study developed and compared two open-source foundational ECG models: DeepECG-SL, a supervised multilabel ECG model, and DeepECG-SSL, a self-supervised model.MethodsBoth models were trained on over 1 million ECGs using a standardized preprocessing pipeline and automated free-text extraction from ECG reports to predict 77 cardiac conditions. DeepECG-SSL leveraged unlabelled data through self-supervised contrastive learning and masked lead modelling before fine-tuning for downstream tasks, while DeepECG-SL was trained directly on labelled diagnostic data in an end-to-end fashion. Performance was evaluated across seven private, multilingual healthcare systems and four public ECG repositories, with assessment of fairness by age and sex, and investigation of privacy vulnerabilities as well as memory and compute requirements.ResultsDeepECG-SSL achieved micro-averaged area under the receiver operating characteristic curves (AUROCs) across all 77 cardiac conditions for ECG interpretation of 0.990 [95% confidence interval (CI): 0.990, 0.990] on the internal dataset (MHI-ds), 0.981 (95% CI: 0.981, 0.981) on external public datasets (UKB, CLSA, MIMIC-IV and PTB), and 0.983 (95% CI: 0.983, 0.983) on external private datasets (UW, UCSF, JGH, NYP, MGH, CSH and CHUM), while DeepECG-SL demonstrated AUROCs of 0.992 (95% CI: 0.992, 0.992), 0.980 (95% CI: 0.980, 0.980), and 0.983 (95% CI: 0.983, 0.984), respectively. Fairness analyses revealed minimal disparities (true-positive rate and false-positive rate difference <0.1) across age and sex groups for both models. DeepECG-SSL demonstrated superior performance on limited-data digital biomarker tasks, with the largest improvements in long QT syndrome (LQTS) genotype classification (AUROC 0.931 vs 0.850, P = .026, n = 127 ECGs) and 5 year atrial fibrillation risk prediction (AUROC 0.742 vs 0.734, P < 0.001, n = 132 050 ECGs), while achieving superior performance in left ventricular ejection fraction ≤40% classification (AUROC 0.926 vs 0.917, P < 0.001, n = 25 252 ECGs) and comparable performance in LQTS detection (AUROC 0.767 vs 0.735, P = 0.117, n = 934 ECGs).ConclusionsThis study establishes SSL as a promising paradigm for ECG analysis, particularly in settings with limited annotated data, enhancing accessibility, generalizability, and fairness in AI-driven cardiac diagnostics. By releasing model weights, preprocessing tools, and validation code, this work aims to support robust, data-efficient AI diagnostics across diverse clinical environments and questions.

We aimed to estimate the rate of kidney function decline over 10 years in the general population and develop a machine learning model to predict it. We used the JMDC database from 2012 to 2021, which includes company employees and their family members in Japan, where annual health checks are mandated for people aged 40-74 years. We estimated the slope (average change) of estimated glomerular filtration rate (eGFR) over a period of 10 years. Then, using the annual health-check results and prescription claims for the first five years from 2012 to 2016 as predictor variables, we developed an XGBoost model, evaluated its prediction performance with the root mean squared error (RMSE), R2, and area under the receiver operating characteristic curve (AUROC) for rapid decliners (defined as the slope <-3 ml/min/1.73 m2/year) using 5-fold cross validation, and compared these indicators with those of (i) the simple application of the eGFR slope from 2012 to 2016 and (ii) the adjusted linear regression model. We included 126,424 individuals (mean age, 45.2 years; male, 82.4%; mean eGFR, 79.0 ml/min/1.73 m2 in 2016). The mean slope was -0.89 (standard deviation, 0.96) ml/min/1.73 m2/year. The predictive performance of the XGBoost model (RMSE, 0.78; R2, 0.35; and AUROC, 0.89) was better than that of either the simple application of the eGFR slope from 2012 to 2016 (RMSE, 1.94; R2, -3.03; and AUROC, 0.79) or the adjusted linear regression model (RMSE, 0.81; R2, 0.30; and AUROC, 0.87). We estimated the rate of kidney function decline over 10 years in the general population as well as demonstrated that application of machine learning to annual health-check and claims data provides better predictive performance compared to traditional methods.

<sec> <title>BACKGROUND</title> Early prediction of gestational diabetes mellitus (GDM) is critical for improving maternal health outcomes. However, predictive models are often challenged by limited early-pregnancy samples, severe class imbalance in datasets, and complex interrelationships among clinical features. </sec> <sec> <title>OBJECTIVE</title> This study aimed to develop and evaluate a unified dual-dimensional enhancement framework that integrates data augmentation and feature engineering to improve early GDM prediction performance by addressing data imbalance and leveraging medical prior knowledge. </sec> <sec> <title>METHODS</title> We proposed a framework combining Generative Adversarial Network (GAN)-based data augmentation with Large Language Model (LLM)-inspired feature engineering. GAN sampling was used to generate clinically plausible synthetic minority class samples to mitigate data imbalance. The LLM was guided to organize features into domains (e.g., basic demographics, metabolic syndrome, core liver biomarkers) and generate higher-order composite features, integrating medical prior knowledge. Machine learning models were subsequently developed, and interpretability analyses were performed using SHAP to identify key predictors. </sec> <sec> <title>RESULTS</title> The Random Forest model enhanced by TVAE-based feature augmentation demonstrated the best performance. On the test dataset, it achieved a recall of 0.7559, an accuracy of 0.8444, and an area under the receiver operating characteristic curve (AUROC) of 0.8873. SHAP analysis identified the following five features as the most influential predictors: fasting blood glucose, the composite feature (fasting blood glucose + triglycerides) × pre-pregnancy BMI, activated partial thromboplastin time, leukocyte count, and neutrophil count. </sec> <sec> <title>CONCLUSIONS</title> The proposed dual-dimensional enhancement framework effectively alleviates data limitations and captures complex feature interactions in early GDM prediction. This strategy not only improves model performance, particularly in recall, but also provides interpretable biological evidence to support rapid clinical screening, stratified management, and early intervention in pregnancy. </sec>

Objective: To assess the performance of continuous glucose monitoring (CGM) metrics in predicting development of gestational diabetes mellitus (GDM) and adverse pregnancy outcomes. Methods: In a prospective observational study, CGM data were collected from 760 pregnant participants. Predictive CGM-based models were built using logistic and elastic net regression. Predictive performance was assessed by the area under the receiver operating characteristic curve (AUROC). Results: The AUROCs of using 2nd trimester % time &amp;gt;140 mg/dL (TA140) and week 13-14 TA140 in predicting GDM were 0.81 and 0.74, respectively. The AUROCs for predicting large for gestational age births (LGA) and hypertensive disorders of pregnancy (HDP) using 2nd trimester TA140 were both 0.58. Adding multiple predictors to the models did not meaningfully improve AUROC. When matching the specificity of OGTT, a model using TA140 in weeks 13-14 achieved similar sensitivity to OGTT in predicting HDP (13% vs. 10%, respectively) and LGA (6% for both) (Figure). Conclusion: CGM metrics such as TA140 predicted GDM with high AUROCs as early as 13-14 weeks’ gestation. CGM metrics at 13-14 weeks were statistically similar to OGTT at 24-28 weeks in predicting APOs although sensitivity was low for both. CGM could potentially be used as an alternative to OGTT and/or an early screening tool for elevated hyperglycemia during gestation. Disclosure Z. Li: None. R. Beck: Consultant; Insulet Corporation. Research Support; Insulet Corporation. Consultant; Tandem Diabetes Care, Inc. Research Support; Tandem Diabetes Care, Inc. Consultant; Beta Bionics, Inc. Research Support; Beta Bionics, Inc., Dexcom, Inc., Bigfoot Biomedical, Inc. Consultant; Novo Nordisk. Research Support; Novo Nordisk, Eli Lilly and Company. Consultant; embecta, Vertex Pharmaceuticals Incorporated, Hagar, Ypsomed AG, Sanofi, Zucara Therapeutics, Sequel. C. Durnwald: Consultant; Dexcom, Inc. Research Support; Dexcom, Inc. A.L. Carlson: Research Support; Medtronic, Insulet Corporation, Tandem Diabetes Care, Inc., Eli Lilly and Company, Sanofi. Other Relationship; Novo Nordisk. Advisory Panel; MannKind Corporation. Research Support; Dexcom, Inc. E. Norton: None. R.M. Bergenstal: Other Relationship; Abbott. Research Support; Arkray Marketing. Consultant; Ascensia Diabetes Care, Bigfoot Biomedical, Inc., CeQur. Other Relationship; Dexcom, Inc., Eli Lilly and Company. Consultant; embecta, Hygieia. Research Support; Insulet Corporation. Consultant; MannKind Corporation. Other Relationship; Medtronic, Novo Nordisk. Consultant; Onduo LLC, Roche Diabetes Care. Other Relationship; Sanofi. Research Support; Tandem Diabetes Care, Inc. Other Relationship; UnitedHealth Group. Consultant; Vertex Pharmaceuticals Incorporated, Zealand Pharma A/S. M.L. Johnson: Research Support; Abbott, Lilly Diabetes, National Institute of Diabetes and Digestive and Kidney Diseases, Novo Nordisk, Medtronic, Dexcom, Inc., Leona M. and Harry B. Helmsley Charitable Trust. Consultant; Jaeb Center for Health Research. Research Support; Patient-Centered Outcomes Research Institute, Jaeb Center for Health Research. S.M. Dunnigan: Research Support; Jaeb Center for Health Research, Patient-Centered Outcomes Research Institute, National Institute of Diabetes and Digestive and Kidney Diseases. M. Banfield: None. K.J. Krumwiede: None. J. Sibayan: None. P. Calhoun: None. Funding Study funding was provided by The Leona M. and Harry B. Helmsley and UnitedHealth Group. Dexcom provided the devices used in the study.

Introduction Hypertrophic cardiomyopathy (HCM) is a common inherited cardiovascular disease, typically transmitted in an autosomal dominant manner. Although pathogenic variants in sarcomere-encoding genes are key determinants of disease onset and progression, genetic testing has faced hurdles in terms of human and financial resources, necessitating efficient prioritization. The Mayo HCM Genotype Predictor Score (Mayo score) and the Toronto HCM Genotype Score (Toronto score), both derived from clinical and echocardiographic variables, have showed acceptable performance estimating the pre-test probability of genotype positivity. While electrocardiogram (ECG) is a low-cost and minimally invasive modality and pathogenic variants are associated with electrophysiological abnormalities, ECG parameters were not incorporated into these models. Purpose This study aimed to develop an improved predictive model for genotype positivity in HCM by integrating ECG parameters. Methods We retrospectively analysed 507 unrelated HCM patients from a Japanese multicentre cohort. Genotype positivity was defined as the presence of pathogenic or likely pathogenic (P/LP) variants in definitive sarcomeric genes. Candidate ECG variables were selected via univariate analysis and stepwise selection based on Akaike’s information criterion in a multivariable logistic regression model. A novel point-based Mayo-ECG score was developed and internally validated using bootstrap resampling. Predictive performance was assessed by the area under the receiver operating characteristic curve (AUROC) and net reclassification improvement (NRI). Results Overall, 162 (32.0%) were genotype positive. P/LP variants were most frequently identified in MYBPC3 (n = 74), followed by MYH7 (n = 62), and the other genes. Genotype-positive patients had a higher prevalence of atrial fibrillation, intraventricular conduction disturbance (QRS duration ≥ 120 msec or bundle branch block), prolonged QT interval, left axis deviation, lower Sokolow-Lyon index and less frequent T-wave inversion in the precordial leads compared to genotype-negative patients. The Mayo-ECG score (Figure 1) stratified genotype positivity from 7.6% (score ≤ -1) to 90.9% (score ≥ 4) (Figure 2A). Bootstrap resampling demonstrated the Mayo-ECG score (AUROC, 0.81 [95% CI, 0.77-0.85]) outperformed the Mayo score (AUROC, 0.76 [95% CI, 0.72-0.81]; p &amp;lt; 0.001, NRI: 0.21 [95% CI, 0.13-0.29]; p = 0.007) and the Toronto score (AUROC, 0.75 [95% CI, 0.70-0.79]; p = 0.009, NRI: 0.17 [95% CI, 0.06-0.27]; p &amp;lt; 0.001) (Figure 2B), with good calibration (Brier score: 0.156, max calibration error: 0.024). Conclusions Genotype-positive and genotype-negative HCM patients exhibited distinct ECG characteristics. The Mayo-ECG score significantly enhances genotype prediction in HCM by incorporating ECG parameters, outperforming conventional models. Its simplicity and improved accuracy support its use for optimizing genetic screening in HCM.Novel Mayo-ECG score Performance of Mayo-ECG score

e23161 Background: Data retrieval is challenging in clinical research and traditional methods for data collection are often time-consuming and may be error-prone. Large Language Models (LLMs) have shown zero-shot capabilities in converting unstructured clinical text into structured data. These technologies could support the retrieval stage of clinical trials by leveraging the information reported in Electronic Health Records (EHRs) without relying any longer on manual curation. APOLLO 11 Consortium (NCT05550961) is a multicentric Italian trial which leverages a federated infrastructure for the analysis of advanced lung cancer patient data across Italy. Methods: We conducted a pilot study using Llama 3.1 8B on 358 Non-Small Cell Lung Cancer patients from the IRCCS Istituto Nazionale dei Tumori, leader of the APOLLO 11 Consortium. Anonymized EHRs have been analyzed within the LLM pipeline for feature extraction by Wiest et al. A combination of zero/few shot prompting techniques both in English and Italian languages was used. We selected smoking, histology, PD-L1 and staging as multiclass variables and bone/brain/liver metastases as binary variables. The ground truth collection involved a first Manual Data Entry (1-MDE) and a final full-revised MDE (2-MDE). The LLM accuracy was calculated only for the comparison LLM vs 2-MDE. In addition, we calculated the percentage of Missing Information (% MI) in 1-MDE, 2-MDE and LLM extraction. Results: Compared to 2-MDE, LLM achieved feature-specific accuracies of 0.78 for PD-L1, 0.85 for BONE METASTASIS, 0.83 for BRAIN METASTASIS, 0.89 for LIVER METASTASIS and 0.96 for TUMOUR STAGING. For smoking and staging, LLM extraction also reduced % MI relative to 1-MDE (Table 1). Only for PD-L1, we further analyzed the 12.8% of MI and found that 91.3% resulted from hallucinations (i.e., PD-L1 was misclassified as missing). Evaluations using English prompts confirmed the pipeline’s adaptability and high tasks accuracy. Conclusions: This study confirms the feasibility of LLMs for data retrieval in clinical trials demonstrating strong performance across diverse clinical features with minimal prompt optimization. LLMs could assist clinicians and data entry personnel in the 1-MDE process, streamlining initial data structuring and saving time. The 2-MDE step can remain as a quality check to address any discrepancies. Further improvements could focus on prompt optimization and integrating human feedback to reduce hallucination rates. Clinical trial information: NCT05550961 . %MI in 1-MDE, 2-MDE and LLM extraction. Accuracy refers only to LLM vs 2-MDE. Histology and metastasis sites were collected only in 2-MDE. NA = not available. Smoking PD-L1 Histology Bone Met Brain Met Liver Met T N M Stage % MI 1-MDE 6.4 8.9 NA NA NA NA 22.5 22.5 23.11 98.3 % MI 2-MDE 6.6 3 0 0 0 0 0 0 0 0 % MI LLM 2.7 12.8 10.3 0 0 0 0 0 0 6.9 % accuracy (LLM vs 2-MDE) 67 78 91 85 83 89 39 52 70 96

BackgroundMachine learning (ML) models for clinical prediction are often developed independently of downstream value, interpretability, and decision-making needs. As a result, model outputs may be difficult to act upon in practice, especially when trade-offs between various clinical aspects are not explicitly addressed.PurposeThis paper presents a cost-aware prediction (CAP) framework designed to support 1-year mortality prediction for heart failure (HF) patients to identify those eligible for home care based on high predicted mortality risk. By leveraging cost-benefit analysis and large language model (LLM) agents, the framework facilitates clinical interpretability and helps stakeholders navigate real-world trade-offs in ML-based decision support.MethodsFirst, we developed an ML model predicting 1-year mortality using electronic health records including all patients with a first in-hospital HF diagnosis between 2017-2023 (N=30,021, 22% mortality) from Region Västra Götaland, Sweden (total population 1.8 million people). Second, we introduced clinical impact projection (CIP) curves to visualise important cost dimensions – quality of life and healthcare provider expenses. These costs were further divided into treatment and error costs, to assess the consequences of both correct and incorrect predictions. Finally, we developed four LLM agents to generate individual patient-specific descriptions explaining the certainty of the prediction, cost-benefit considerations, and risk reduction strategies. The system was evaluated by clinicians for decision support value.ResultsFor the ML predictive modelling, the eXtreme gradient boosting model achieved the best performance, with an area under the receiver operating characteristic curve (AUROC) of 0.804 (95%; confidence interval (CI) 0.792-0.816), area under the precision-recall curve (AUPRC) of 0.529 (95% CI 0.502-0.558) and a Brier score of 0.135 (95% CI 0.130-0.140) shown in Figure 1. The CIP cost curves provided a population-level overview of cost composition across decision thresholds. LLM-generated cost-benefit descriptions enabled prediction analysis at individual patient-levels. An illustrative example can be found in Figure 2. The system was generally well received according to the evaluation with clinicians. However, clinical feedback emphasised the need to strengthen the technical accuracy for speculative tasks.ConclusionsAn ML algorithm was developed to predict 1-year mortality risks. Further, the CAP framework utilises LLM agents to integrate this ML classifier outcomes and cost-benefit analysis, to support clinical decision-making by more transparent and interpretable communication. While receiving positive clinical feedback, future work is needed for fine-tuning of LLM narratives for optimal accuracy and usefulness.