Predictive structured\u2013unstructured interactions in EHR models: A case study of suicide prediction

To examine the value of unstructured electronic health record (EHR) data (free-text notes) in identifying a set of geriatric syndromes. Retrospective analysis of unstructured EHR notes using a natural language processing (NLP) algorithm. Large multispecialty group. Older adults (N=18,341; average age 75.9, 58.9% female). We compared the number of geriatric syndrome cases identified using structured claims and structured and unstructured EHR data. We also calculated these rates using a population-level claims database as a reference and identified comparable epidemiological rates in peer-reviewed literature as a benchmark. Using insurance claims data resulted in a geriatric syndrome prevalence ranging from 0.03% for lack of social support to 8.3% for walking difficulty. Using structured EHR data resulted in similar prevalence rates, ranging from 0.03% for malnutrition to 7.85% for walking difficulty. Incorporating unstructured EHR notes, enabled by applying the NLP algorithm, identified considerably higher rates of geriatric syndromes: absence of fecal control (2.1%, 2.3 times as much as structured claims and EHR data combined), decubitus ulcer (1.4%, 1.7 times as much), dementia (6.7%, 1.5 times as much), falls (23.6%, 3.2 times as much), malnutrition (2.5%, 18.0 times as much), lack of social support (29.8%, 455.9 times as much), urinary retention (4.2%, 3.9 times as much), vision impairment (6.2%, 7.4 times as much), weight loss (19.2%, 2.9 as much), and walking difficulty (36.34%, 3.4 as much). The geriatric syndrome rates extracted from structured data were substantially lower than published epidemiological rates, although adding the NLP results considerably closed this gap. Claims and structured EHR data give an incomplete picture of burden related to geriatric syndromes. Geriatric syndromes are likely to be missed if unstructured data are not analyzed. Pragmatic NLP algorithms can assist with identifying individuals at high risk of experiencing geriatric syndromes and improving coordination of care for older adults.

ObjectiveThis study uses electronic health record (EHR) data to predict 12 common cancer symptoms, assessing the efficacy of machine learning (ML) models in identifying symptom influencers.Materials and MethodsWe analyzed EHR data of 8156 adults diagnosed with cancer who underwent cancer treatment from 2017 to 2020. Structured and unstructured EHR data were sourced from the Enterprise Data Warehouse for Research at the University of Iowa Hospital and Clinics. Several predictive models, including logistic regression, random forest (RF), and XGBoost, were employed to forecast symptom development. The performances of the models were evaluated by F1-score and area under the curve (AUC) on the testing set. The SHapley Additive exPlanations framework was used to interpret these models and identify the predictive risk factors associated with fatigue as an exemplar.ResultsThe RF model exhibited superior performance with a macro average AUC of 0.755 and an F1-score of 0.729 in predicting a range of cancer-related symptoms. For instance, the RF model achieved an AUC of 0.954 and an F1-score of 0.914 for pain prediction. Key predictive factors identified included clinical history, cancer characteristics, treatment modalities, and patient demographics depending on the symptom. For example, the odds ratio (OR) for fatigue was significantly influenced by allergy (OR = 2.3, 95% CI: 1.8-2.9) and colitis (OR = 1.9, 95% CI: 1.5-2.4).DiscussionOur research emphasizes the critical integration of multimorbidity and patient characteristics in modeling cancer symptoms, revealing the considerable influence of chronic conditions beyond cancer itself.ConclusionWe highlight the potential of ML for predicting cancer symptoms, suggesting a pathway for integrating such models into clinical systems to enhance personalized care and symptom management.

Antidepressant selection is largely a trial-and-error process. We used electronic health record (EHR) data and artificial intelligence (AI) to predict response to four antidepressants classes (SSRI, SNRI, bupropion, and mirtazapine) 4 to 12 weeks after antidepressant initiation. The final data set comprised 17,556 patients. Predictors were derived from both structured and unstructured EHR data and models accounted for features predictive of treatment selection to minimize confounding by indication. Outcome labels were derived through expert chart review and AI-automated imputation. Regularized generalized linear model (GLM), random forest, gradient boosting machine (GBM), and deep neural network (DNN) models were trained and their performance compared. Predictor importance scores were derived using SHapley Additive exPlanations (SHAP). All models demonstrated similarly good prediction performance (AUROCs ≥ 0.70, AUPRCs ≥ 0.68). The models can estimate differential treatment response probabilities both between patients and between antidepressant classes for the same patient. In addition, patient-specific factors driving response probabilities for each antidepressant class can be generated. We show that antidepressant response can be accurately predicted from real-world EHR data with AI modeling, and our approach could inform further development of clinical decision support systems for more effective treatment selection.

Depression and anxiety are prevalent mental health conditions among individuals with type 2 diabetes mellitus (T2DM), who exhibit unique vulnerabilities and etiologies. However, existing approaches fail to fully utilize regional heterogeneous electronic health record (EHR) data. Integrating this data can provide a more comprehensive understanding of depression and anxiety in T2DM patients, leading to more personalized treatment strategies. This study aims to develop and validate a deep learning model, the Regional EHR for Depression and Anxiety Prediction Model (REDAPM), using regional EHR data to predict depression and anxiety in patients with T2DM. A case-control development and validation study was conducted using regional EHR data from the Nanjing Health Information Center (NHIC). Two retrospective, matched (1:3) datasets were constructed from the full cohort for the model's internal and external validation. These two datasets were selected from the NHIC data of 2020 and 2022, respectively. The REDAPM incorporates both structured and unstructured EHR data, capturing the temporal dependency of clinical events. The performance of REDAPM was compared to a set of baseline models, evaluated using the area under the receiver operating characteristic curve (ROC-AUC) and the area under the precision-recall curve (PR-AUC). Subgroup, ablation, and interpretation analyses were conducted to identify relevant clinical features available from EHRs. The internal and external validation datasets comprised 24,724 and 34,340 patients, respectively. The REDAPM outperformed baseline models in both datasets, achieving ROC-AUC scores of 0.9029±0.008 and 0.7360±0.005, and PR-AUC scores of 0.8124±0.011 and 0.5504±0.009. Ablation and subgroup experiments confirmed the significant contribution of patients' medical history text to the model's performance. Integrated gradient score analysis identified the predictive importance of other mental disorders. The REDAPM effectively leverages the heterogeneous characteristics of regional EHR data, demonstrating strong predictive performance for depression onset in diabetic patients. It also shows potential for discovering significant clinical features, indicating considerable promise for clinical utility.

2043 Background: Electronic health records (EHR) are used for retrospective cancer outcomes analysis. Sites and timing of recurrence are not captured in structured EHR data. Novel computerized methods are necessary to use unstructured longitudinal EHR data for large scale studies. Methods: We previously developed a neural network-based NLP algorithm to identify no recurrence vs. metastatic recurrence cases by analyzing physician notes, pathology and radiology reports in Stanford’s breast cancer database, Oncoshare (Cohort A). To validate this algorithm for local vs. distant recurrence, we identified a distinct Oncoshare cohort (Cohort B). Cases were manually curated for longitudinal development of local or distant recurrence and metastatic sites. A two-sided t-test was used to compare mean probabilities between local and distant recurrence cases. Next, we combined cases in Cohorts A and B to train and validate a novel NLP classifier that identifies metastatic site. The combined cohort was randomly divided into training and validation sets. Sensitivity and specificity were calculated for the NLP algorithm’s ability to detect metastatic sites compared to manual curation. Results: In Cohort B: 350 metastatic cases were identified. Mean probability for local and distant recurrence was 0.43 and 0.79, respectively and differed significantly for patients with local vs. distant recurrence (p<0.01). In Cohorts A and B: 632 metastatic cases were used for determination of sites. Sensitivity and specificity were highest for detection of peritoneal metastasis followed by liver, lung, skin, bone and central nervous system (table). Conclusions: This NLP algorithm is a scalable tool that uses unstructured EHR data to capture breast cancer recurrence, distinguishing local from distant recurrence and identifying metastatic site. This method may facilitate analysis of large datasets and correlation of outcomes with metastatic site. [Table: see text]

Background Statins are guideline-recommended medications that reduce cardiovascular events in patients with diabetes. Yet, statin use is concerningly low in this high-risk population. Identifying reasons for statin nonuse, which are typically described in unstructured electronic health record data, can inform targeted system interventions to improve statin use. We aimed to leverage a deep learning approach to identify reasons for statin nonuse in patients with diabetes. Methods and Results Adults with diabetes and no statin prescriptions were identified from a multiethnic, multisite Northern California electronic health record cohort from 2014 to 2020. We used a benchmark deep learning natural language processing approach (Clinical Bidirectional Encoder Representations from Transformers) to identify statin nonuse and reasons for statin nonuse from unstructured electronic health record data. Performance was evaluated against expert clinician review from manual annotation of clinical notes and compared with other natural language processing approaches. Of 33 461 patients with diabetes (mean age 59±15 years, 49% women, 36% White patients, 24% Asian patients, and 15% Hispanic patients), 47% (15 580) had no statin prescriptions. From unstructured data, Clinical Bidirectional Encoder Representations from Transformers accurately identified statin nonuse (area under receiver operating characteristic curve [AUC] 0.99 [0.98-1.0]) and key patient (eg, side effects/contraindications), clinician (eg, guideline-discordant practice), and system reasons (eg, clinical inertia) for statin nonuse (AUC 0.90 [0.86-0.93]) and outperformed other natural language processing approaches. Reasons for nonuse varied by clinical and demographic characteristics, including race and ethnicity. Conclusions A deep learning algorithm identified statin nonuse and actionable reasons for statin nonuse in patients with diabetes. Findings may enable targeted interventions to improve guideline-directed statin use and be scaled to other evidence-based therapies.

The increase in hospitalizations of patients with noncommunicable diseases remains a key challenge for healthcare systems. A significant portion of these hospitalizations is potentially preventable with timely outpatient monitoring and effective primary care. Predictive models based on data from health information systems (HIS) and electronic health records (EHRs) make it possible to identify patients at increased risk of hospitalization and improve their management strategies. The article systematizes and summarizes international experience in using hospitalization risk prognostic models, developed on the basis of HIS and EHR data. We made a systematic review of publications presented in Russian and foreign databases (Scopus, PubMed, BMC Health Services Research, BMJ Research, Google Scholar, Elibrary, Oxford Academic, Wiley Online Library) for the period 1993-2023. The analysis included studies that used EHR and HIS data to develop and/or validate prognostic models of hospitalization risk. The information is systematized by following predictor categories: diseases and comorbidities, prescriptions, interaction with healthcare organizations, sociodemographic and laboratory parameters. The final analysis included 14 studies describing 28 models for predicting the hospitalization risk. Most studies used logistic regression. Machine learning methods (gradient boosting, random forest, and Bayesian ensemble models) were used in some studies. The categories with the highest prognostic value were "diseases and comorbidities" (42,3%), "medication prescriptions" (23,6%), and "interaction with health organizations" (19,8%). Including temporal characteristics (frequency and recency of hospitalizations, visits to primary care facilities, and laboratory tests) increased the model accuracy. Predictive models based on EHR and HIS data demonstrate high discriminatory power and enable the assessment of hospitalization risk at the primary care level. Machine learning methods in some studies demonstrated the possibility of a more in-depth analysis of the relationships between predictors and improved prediction accuracy. A promising area of research is the further development, external validation, and adaptation of models using expanded EHR and HIS data sets for its use in outpatient monitoring and preventing noncommunicable diseases.

For research involving big data, researchers must accurately identify patients with ocular diseases or phenotypes of interest. Reliance on administrative billing codes alone for this purpose is limiting. To develop a method to accurately identify the presence or absence of ocular conditions of interest using electronic health record (EHR) data. This study is a retrospective analysis of the EHR data of patients (n = 122 339) in the Sight Outcomes Research Collaborative Ophthalmology Data Repository who received eye care at participating academic medical centers between August 1, 2012, and August 31, 2017. An algorithm that searches structured and unstructured (free-text) EHR data for conditions of interest was developed and then tested to determine how well it could detect the presence or absence of exfoliation syndrome (XFS). The algorithm was trained to search for evidence of XFS among a sample of patients with and without XFS (n = 200) by reviewing International Classification of Diseases, Ninth Revision or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-9 or ICD-10) billing codes, the patient's problem list, and text within the ocular examination section and unstructured (free-text) data in the EHR. The likelihood that each patient had XFS was estimated using logistic least absolute shrinkage and selection operator (LASSO) regression. The EHR data of all patients were run through the algorithm to generate an XFS probability score for each patient. The algorithm was validated with review of EHRs by glaucoma specialists. Positive predictive value (PPV) and negative predictive value (NPV) of the algorithm were computed as the proportion of patients correctly classified with XFS or without XFS. This study included 122 339 patients, with a mean (SD) age of 52.4 (25.1) years. Of these patients, 69 002 (56.4%) were female and 99 579 (81.4%) were white. The algorithm assigned a less than 10% probability of XFS for 121 085 patients (99.0%) as well as an XFS probability score of more than 75% for 543 patients (0.4%), more than 90% for 353 patients (0.3%), and more than 99% for 83 patients (0.07%). Validated by glaucoma specialists, the algorithm had a PPV of 95.0% (95% CI, 89.5%-97.7%) and an NPV of 100% (95% CI, 91.2%-100%). When there was ICD-9 or ICD-10 billing code documentation of XFS, in 86% or 96% of the records, respectively, evidence of XFS was also recorded elsewhere in the EHR. Conversely, when there was clinical examination or free-text evidence of XFS, it was documented with ICD-9 codes only approximately 40% of the time and even less often with ICD-10 codes. The algorithm developed, tested, and validated in this study appears to be better at identifying the presence or absence of XFS in EHR data than the conventional approach of assessing only billing codes; such an algorithm may enhance the ability of investigators to use EHR data to study patients with ocular diseases.

Objective: Using methylation characteristics of human genes to construct machine learning predictive models for screening cervical cancer and precancerous lesions. Methods: Human DNA methylation detection was performed on 224 cervical exfoliated cell specimens from the Cancer Hospital of the Chinese Academy of Medical Sciences, Tianjin Central Hospital of Gynecology Obstetrics, Xinmi Maternal and Child Health Hospital of Henan Province, West China Second Affiliated Hospital of Sichuan University, and Heping Hospital Affiliated to Changzhi Medical College collected during April 2014 and March 2015. The hypermethylated gene fragments related to cervical cancer were selected by high-density, high-association, and hypermethylated gene fragment screening and the LASSO regression algorithm. Taking cervical intraepithelial neoplasia grade 2 (CIN2) or more severe lesions as the research outcome, machine learning predictive models based on the random forest (RF), naive Bayes (NB), and support vector machine (SVM) algorithm, respectively, were constructed. A total of 144 outpatient specimens were used as the training set and 80 cervical exfoliated cell specimens from women participating in the cervical cancer screening program were used as the test set to verify the predictive models. Using histological diagnosis results as the gold standard, the detection efficacy for CIN2 or more severe lesions of the three machine learning predictive models were compared with that of the human papilloma virus (HPV) detection and cytological diagnosis. Results: In the training set of 144 cases, there were 34 cases of HPV positivity, with a positive rate of 23.61%. Cytologically, there were 37 cases diagnosed as no intraepithelial lesion or malignancy (NILM), and 107 cases diagnosed as atypical squamous cells of undetermined significance (ASC-US) or above. Histologically, there were 28 cases without cervical intraepithelial neoplasia or benign cervical lesions, 31 cases of CIN1, 18 cases of CIN2, 31 cases of CIN3, and 36 cases of squamous cell carcinoma. Seven hypermethylated gene fragments were selected from 45 genes, and three machine learning prediction models based on the RF, NB, and SVM algorithm, respectively, were constructed. In the validation set of 80 cases, there were 28 cases of HPV positivity, with a positive rate of 35.00%. Cytologically, there were 65 cases diagnosed as NILM and 15 cases as ASC-US or above. Histologically, there were 39 cases without cervical intraepithelial neoplasia or benign cervical lesions, 10 cases of CIN1, 10 cases of CIN2, 11 cases of CIN3, and 10 cases of squamous cell carcinoma. In the validation set, the area under the curve (AUC) values of the RF model, NB model, SVM model, HPV detection, and cytological diagnosis of CIN2 or above were 0.90, 0.88, 0.82, 0.68, and 0.45, respectively. The DeLong test showed that there was no statistically significant difference in the AUC values between the RF, NB, and SVM models (all P＞0.05), and the AUC values of the RF and NB models were higher than that of HPV detection (both P＜0.01), and the AUC values of the RF, NB, and SVM models were higher than that of cytological diagnosis (all P＜0.01). Compared with the NB model, the sensitivity of the RF model was similar (80.65% vs. 77.42%), but the specificity of the NB model was much higher than that of the RF model (93.88% vs. 73.47%). Conclusion: Among the machine learning prediction models for cervical cancer and precancerous lesions constructed based on human DNA methylation, the NB model has good predictive performance for CIN2 and above lesions, and may be used for screening of cervical cancer and precancerous lesions.

BackgroundRetention in HIV care is essential to end the HIV epidemic and improve individual patient outcomes, yet 1/3 of people living with HIV in Maryland are not consistently retained in care. Predictive modeling using electronic health record (EHR) data is a promising strategy to identify patients at risk for lapses in care; however, existing efforts remain limited. Regional analyses using statewide health systems can uniquely inform local strategies by accounting for demographic, clinical, and structural variations. This study aimed to develop predictive models utilizing comprehensive EHR data from the University of Maryland Medical System (UMMS) to identify people living with HIV who are at risk of lapsing in care.Figure 1.Receiver Operating Characteristic (ROC) Curve demonstrating the performance of the Random Forest model in predicting lapses in HIV care. The model achieved an area under the curve (AUC) of 0.91, with high recall (98%) and precision (81%), indicating strong predictive capability for identifying patients at risk of lapsing in HIV care.Figure 2.SHAP summary illustrating the impact of key features on the Random Forest model's predictions of HIV care lapses. Each dot represents a patient encounter, with dot colors indicating feature value magnitude (high in red, low in blue).MethodsUtilizing EHR data from UMMS including HIV-related prescriptions, laboratory tests and clinical visits from adults receiving antiretroviral therapy from 1/2016-6/2024, we identified 8,518 patients with 205,633 encounters. Consolidating multiple encounters within 10 days as one encounter, and including only encounters preceding the last recorded encounter for both lapsed and non-lapsed groups, resulted in 4,735 patients. Of those, 2,667 (56%) had lapses - defined as not having an HIV-related clinical encounter within 12 months. We used a Random Forest classifier with extensive hyperparameter tuning via randomized search cross-validation. Model performance was assessed via the performance matrices and SHAP values for feature importance.ResultsThe Random Forest model demonstrated an AUC of 0.91 with 98% recall for lapses, and 81% precision (Figure 1). Feature importance analysis highlighted significant predictors, including time on treatment, age, BMI, alcohol use, employment status, racial demographics, and comorbidities such as hypertension and diabetes (Figure 2).ConclusionPredictive modeling using EHR data from a comprehensive statewide health system can effectively identify patients at risk for lapses in HIV care. Clinically actionable predictors, such as treatment duration, demographics, and specific health conditions, provide practical insights that can guide interventions to enhance patient retention and ultimately improve health outcomes for people living with HIV.DisclosuresAll Authors: No reported disclosures

Deep learning approaches have tremendous potential to improve the predictive power of traditional suicide prediction models to detect and predict intentional self-harm (ISH). Existing research is limited by a general lack of consistent performance and replicability across sites. We aimed to validate a deep learning approach used in previous research to detect and predict ISH using clinical note text and evaluate its generalizability to other academic medical centers. We extracted clinical notes from electronic health records (EHRs) of 1,538 patients with International Classification of Diseases codes for ISH and 3,012 matched controls without ISH codes. We evaluated the performance of two traditional bag-of-words models (i.e., Naïve Bayes, Random Forest) and two convolutional neural network (CNN) models including randomly initialized (CNNr) and pre-trained Word2Vec initialized (CNNw) weights to detect ISH within 24 hours of and predict ISH from clinical notes 1-6 months before the first ISH event. In detecting concurrent ISH, both CNN models outperformed bag-of-words models with AUCs of.99 and F1 scores of 0.94. In predicting future ISH, the CNN models outperformed Naïve Bayes models with AUCs of 0.81-0.82 and F1 scores of 0.61-.64. We demonstrated that leveraging EHRs with a well-defined set of ISH ICD codes to train deep learning models to detect and predict ISH using clinical note text is feasible and replicable at more than one institution. Future work will examine this approach across multiple sites under less controlled settings using both structured and unstructured EHR data.

Objective: To evaluate the continuity and completeness of electronic health record (EHR) data, and the concordance of select clinical outcomes and baseline comorbidities between EHR and linked claims data, from three healthcare delivery systems in Taiwan. Methods: We identified oral hypoglycemic agent (OHA) users from the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), which was linked to the National Health Insurance Research Database (NHIRD), from June 2011 to December 2016. A secondary evaluation involved two additional EHR databases. We created consecutive 90-day periods before and after the first recorded OHA prescription and defined patients as having continuous EHR data if there was at least one encounter or prescription in a 90-day interval. EHR data completeness was measured by dividing the number of encounters in the NTUH-iMD by the number of encounters in the NHIRD. We assessed the concordance between EHR and claims data on three clinical outcomes (cardiovascular events, nephropathy-related events, and heart failure admission). We used individual comorbidities that comprised the Charlson comorbidity index to examine the concordance of select baseline comorbidities between EHRs and claims. Results: We identified 39,268 OHA users in the NTUH-iMD. Thirty-one percent (n = 12,296) of these users contributed to the analysis that examined data continuity during the 6-month baseline and 24-month follow-up period; 31% (n = 3,845) of the 12,296 users had continuous data during this 30-month period and EHR data completeness was 52%. The concordance of major cardiovascular events, nephropathy-related events, and heart failure admission was moderate, with the NTU-iMD capturing 49–55% of the outcome events recorded in the NHIRD. The concordance of comorbidities was considerably different between the NTUH-iMD and NHIRD, with an absolute standardized difference >0.1 for most comorbidities examined. Across the three EHR databases studied, 29–55% of the OHA users had continuous records during the 6-month baseline and 24-month follow-up period. Conclusion: EHR data continuity and data completeness may be suboptimal. A thorough evaluation of data continuity and completeness is recommended before conducting clinical and translational research using EHR data in Taiwan.

Hospital-induced delirium is one of the most common and costly iatrogenic conditions, and its incidence is predicted to increase as the population of the United States ages. An academic and clinical interdisciplinary systems approach is needed to reduce the frequency and impact of hospital-induced delirium. The long-term goal of our research is to enhance the safety of hospitalized older adults by reducing iatrogenic conditions through an effective learning health system. In this study, we will develop models for predicting hospital-induced delirium. In order to accomplish this objective, we will create a computable phenotype for our outcome (hospital-induced delirium), design an expert-based traditional logistic regression model, leverage machine learning techniques to generate a model using structured data, and use machine learning and natural language processing to produce an integrated model with components from both structured data and text data. This study will explore text-based data, such as nursing notes, to improve the predictive capability of prognostic models for hospital-induced delirium. By using supervised and unsupervised text mining in addition to structured data, we will examine multiple types of information in electronic health record data to predict medical-surgical patient risk of developing delirium. Development and validation will be compliant to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Work on this project will take place through March 2024. For this study, we will use data from approximately 332,230 encounters that occurred between January 2012 to May 2021. Findings from this project will be disseminated at scientific conferences and in peer-reviewed journals. Success in this study will yield a durable, high-performing research-data infrastructure that will process, extract, and analyze clinical text data in near real time. This model has the potential to be integrated into the electronic health record and provide point-of-care decision support to prevent harm and improve quality of care. DERR1-10.2196/48521.

IntroductionThe ability to utilize the electronic health record (EHR) without compromising the doctor-patient relationship (DPR) is an essential skill of all physicians and trainees, yet little time is spent on educating or assessing learners on needed techniques. To address this gap, we developed a conventional OSCE station coupled with a simulated patient chart within the Epic program in order to assess our students' skills utilizing the EHR during a patient encounter.MethodsOf third-year medical students, 119 were given full access to the patient's simulated chart 24 hours in advance of their OSCE to review clinical data. During an in-person OSCE with a standardized patient (SP), students performed a focused history and physical, using the EHR to verify allergies and medications. Students completed an electronic patient note graded by faculty. SPs evaluated the students on communication and interpersonal skills with specific rubric elements. Faculty graded the students' notes to evaluate their expression of clinical reasoning in the assessment and plan.ResultsTraining SPs and faculty to assess students on EHR skills was feasible. After implementation of a comprehensive curriculum focused on EHR and DPR, there was a significant difference on EHR-related communication skills (M = 76.4, SD = 17.6) versus (M = 37, SD = 28.9) before curriculum enhancement t (117.9) = −12.4, p <.001.DiscussionThe EHR OSCE station provided a standardized method of assessing students' EHR skills during a patient encounter. Challenges still exist in the technological requirements to develop and deliver cases in today's EHR platform.

BackgroundOpioid use disorder (OUD) remains a critical public health crisis in the United States. Despite widespread policy and clinical interventions, early identification of individuals at risk for developing OUD remains challenging due to limitations in traditional screening approaches and a lack of individualized risk stratification methods. Machine learning (ML) methods offer an opportunity to develop timely, high-performing, and explainable predictive models that can enhance OUD prevention strategies in clinical settings.ObjectiveThis study aims to develop and validate an ML model using electronic health record (EHR) data to predict the 3-month risk of incident OUD among adults initiating opioid therapy and to stratify patients into clinically actionable risk groups.MethodsThis prognostic modeling study used 2017‐2022 OneFlorida+ EHR data to develop and validate ML algorithms predicting 3-month incident OUD risk. We included 182,083 adults (≥18 y) without cancer, overdose, or OUD or hospice history who received ≥1 outpatient, noninjectable opioid prescription. Using 183 predictors measured in sequential 3-month intervals, we developed an elastic net, least absolute shrinkage and selection operator, gradient boosting machine (GBM), and random forest models on randomly split training, testing, and validation sets. Model performance was assessed using C-statistics, predictive values, and number needed to evaluate, with patients stratified into risk deciles for clinical applicability. Model explainability was assessed using Shapley additive explanations, and fairness was evaluated using standard metrics. We externally validated the best-performing model using an independent cohort from the 2018‐2020 UPMC (formerly University of Pittsburgh Medical Center) health system.ResultsIn the validation sample (n=60,694), GBM (C-statistics=0.879, 95% CI 0.874‐0.884) and elastic net (C-statistics=0.872, 95% CI 0.867‐0.877) outperformed least absolute shrinkage and selection operator (C-statistics=0.846, 95% CI 0.840‐0.851) and random forest (C-statistics=0.798, 95% CI 0.792‐0.804), with GBM model requiring the fewest predictors (n=75) for predicting 3-month incident OUD. Using the GBM algorithm to predict the subsequent 3-month OUD risk, the top decile subgroup had a positive predictive value of 3.26%, a negative predictive value of 99.8%, and a number needed to evaluate of 31. The top decile (n=6696) captured ~68% of patients with OUD. Shapley additive explanations analysis identified age, number of outpatient visits, history of back and other pain conditions, comorbidity burden, and opioid prescribing patterns as the strongest predictors of incident OUD. Fairness assessment showed an acceptable false negative rate parity across race, age, and sex. In external validation on the UPMC cohort, the GBM model maintained good discrimination (C-statistics=0.756, 95% CI 0.750‐0.762) and effective risk stratification.ConclusionsAn ML algorithm predicting incident OUD derived from OneFlorida+ EHR data performed well in external validation with data using UPMC. The algorithm might be valuable for incident OUD risk prediction and stratification across health systems, with potential to inform early intervention.