Cognitive HSE Risk Prediction and Notification Tool Based on Natural Language Processing

This study aimed to develop and apply natural language processing (NLP) algorithms to identify recurrent atrial fibrillation (AF) episodes following rhythm control therapy initiation using electronic health records (EHRs). We included adults with new-onset AF who initiated rhythm control therapies (ablation, cardioversion, or antiarrhythmic medication) within two US integrated healthcare delivery systems. A code-based algorithm identified potential AF recurrence using diagnosis and procedure codes. An automated NLP algorithm was developed and validated to capture AF recurrence from electrocardiograms, cardiac monitor reports, and clinical notes. Compared with the reference standard cases confirmed by physicians' adjudication, the F-scores, sensitivity, and specificity were all above 0.90 for the NLP algorithms at both sites. We applied the NLP and code-based algorithms to patients with incident AF (n=22 970) during the 12 months after initiating rhythm control therapy. Applying the NLP algorithms, the percentages of patients with AF recurrence for sites 1 and 2 were 60.7% and 69.9% (ablation), 64.5% and 73.7% (cardioversion), and 49.6% and 55.5% (antiarrhythmic medication), respectively. In comparison, the percentages of patients with code-identified AF recurrence for sites 1 and 2 were 20.2% and 23.7% for ablation, 25.6% and 28.4% for cardioversion, and 20.0% and 27.5% for antiarrhythmic medication, respectively. When compared with a code-based approach alone, this study's high-performing automated NLP method identified significantly more patients with recurrent AF. The NLP algorithms could enable efficient evaluation of treatment effectiveness of AF therapies in large populations and help develop tailored interventions.

Development and Validation of a Natural Language Processing Tool to Identify Injuries in Infants Associated With Abuse.

2043 Background: Electronic health records (EHR) are used for retrospective cancer outcomes analysis. Sites and timing of recurrence are not captured in structured EHR data. Novel computerized methods are necessary to use unstructured longitudinal EHR data for large scale studies. Methods: We previously developed a neural network-based NLP algorithm to identify no recurrence vs. metastatic recurrence cases by analyzing physician notes, pathology and radiology reports in Stanford’s breast cancer database, Oncoshare (Cohort A). To validate this algorithm for local vs. distant recurrence, we identified a distinct Oncoshare cohort (Cohort B). Cases were manually curated for longitudinal development of local or distant recurrence and metastatic sites. A two-sided t-test was used to compare mean probabilities between local and distant recurrence cases. Next, we combined cases in Cohorts A and B to train and validate a novel NLP classifier that identifies metastatic site. The combined cohort was randomly divided into training and validation sets. Sensitivity and specificity were calculated for the NLP algorithm’s ability to detect metastatic sites compared to manual curation. Results: In Cohort B: 350 metastatic cases were identified. Mean probability for local and distant recurrence was 0.43 and 0.79, respectively and differed significantly for patients with local vs. distant recurrence (p<0.01). In Cohorts A and B: 632 metastatic cases were used for determination of sites. Sensitivity and specificity were highest for detection of peritoneal metastasis followed by liver, lung, skin, bone and central nervous system (table). Conclusions: This NLP algorithm is a scalable tool that uses unstructured EHR data to capture breast cancer recurrence, distinguishing local from distant recurrence and identifying metastatic site. This method may facilitate analysis of large datasets and correlation of outcomes with metastatic site. [Table: see text]

IntroductionDetailed, precise information on a pharmaceutical’s projected therapeutic use is required for horizon scanning. Inferring an estimated indication from trial protocols is a key skill of horizon scanners. The International Horizon Scanning Initiative (IHSI) database utilizes semi-automated data collection. This pilot aimed to verify that the extraction of relevant word sets to generate an estimated indication could be semi-automated.Methods Ten drugs approved in Europe in 2021 were selected as the pilot test set. The test set included drugs approved for the treatment of rare diseases (n=4), haemato-oncology (n=3), and non-oncology conditions (n=3). Eight of the drugs were approved based on phase III trials. The assessment comprised a review of the pivotal trial that supported product registration for these drugs. We undertook a comparison between a human curator and a natural language processing (NLP) algorithm in generating granular tags relating to key aspects of the drugs’ estimated indication (stage of disease, patient-specific subgroup, and place in treatment).ResultsIn 50 percent of cases, the NLP accurately tagged a word or word set related to stage of disease, patient-specific subgroup, or place in treatment, which was also tagged by human curators. In 50 percent of cases, the NLP did not identify words or word sets tagged by human curators. Where relevant, the NLP successfully tagged the same word sets relating to stage of disease for all drugs in the test set. The same word sets relating to patient-specific subgroup were successfully tagged for three drugs in the set. NLP successfully tagged word sets relating to place in treatment for two drugs.ConclusionsThe NLP algorithm is successful in extracting relevant word sets, which can be used to generate an estimated indication in an automated or semi-automated process. The pilot highlighted that further testing is required to advance the sensitivity of the algorithm. Further piloting exploring both unsupervised and supervised modeling approaches (named entity recognition and deep neural networks, respectively) is planned.

e19062 Background: The annual incidence of venous thromboembolism (VTE) is roughly 50-fold increased post allogeneic hematopoietic cell transplant (HSCT). Timely detection of VTE events has a significant impact on morbidity and mortality from these events. Manual chart review is commonly used for detection of VTE events; however, this is labor intensive and often not feasible. The International Classification of Diseases, 10th revision (ICD-10) has attempted to codify VTE more easily but has high false positive and false negative rates. Natural language processing (NLP) algorithms have been designed to improve detection of VTE events in electronic medical record systems over the past several years. This study aims to compare the accuracy of our institution’s NLP algorithm to the ICD-10 in detecting extremity VTE/pulmonary embolism (PE) events in hematopoietic stem cell transplant (HSCT) patients. Methods: This study was a retrospective analysis of adult patients who had allogeneic HSCT at MD Anderson between 2016-2020. Patient charts were assessed by three different methods for acute VTE events within 100 days of HSCT (index date). ICD-10 codes and our institution’s NLP algorithm were used separately to identify these VTE events. Afterwards, manual chart review was performed for confirmation. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for the ICD-10, NLP, and the NLP+ICD combination. Results: A total of 1,391 patient charts were analyzed in our study. There were 129 (9.3%) VTE events detected via manual chart review. Three methodologies were used for comparison: the NLP algorithm alone, the ICD code alone, and detection by NLP or ICD. The NLP algorithm alone had the best specificity (98.9%) and PPV (89.4%) of the three methods. The ICD-10 code had a statistically significant decrease in ability to detect acute VTE events compared to the NLP (p<0.001) (Table). Of the 19 VTE events not identified by NLP, all were found in radiology or vascular laboratory reports. Conclusions: Our study shows the overall superior performance of the NLP algorithm, as it had higher sensitivity, specificity, PPV, and NPV compared to the ICD-10, as well as a better specificity and PPV compared to the combination of both methods. Future refinement of the NLP algorithm and combination with other detection methodologies will allow for more accurate detection of VTE in large cohorts. [Table: see text]

The prevalence of patients who are Incapacitated with No Evident Advance Directives or Surrogates (INEADS) remains unknown because such data are not routinely captured in structured electronic health records. This study sought to develop and validate a natural language processing (NLP) algorithm to identify information related to being INEADS from clinical notes. We used a publicly available dataset of critical care patients from 2001 through 2012 at a United States academic medical center, which contained 418,393 relevant clinical notes for 23,904 adult admissions. We developed 17 subcategories indicating reduced or elevated potential for being INEADS, and created a vocabulary of terms and expressions within each. We used an NLP application to create a language model and expand these vocabularies. The NLP algorithm was validated against gold standard manual review of 300 notes and showed good performance overall (F-score = 0.83). More than 80% of admissions had notes containing information in at least one subcategory. Thirty percent (n = 7,134) contained at least one of five social subcategories indicating elevated potential for being INEADS, and <1% (n = 81) contained at least four, which we classified as high likelihood of being INEADS. Among these, n = 8 admissions had no subcategory indicating reduced likelihood of being INEADS, and appeared to meet the definition of INEADS following manual review. Among the remaining n = 73 who had at least one subcategory indicating reduced likelihood of being INEADS, manual review of a 10% sample showed that most did not appear to be INEADS. Compared with the full cohort, the high likelihood group was significantly more likely to die during hospitalization and within four years, to have Medicaid, to have an emergency admission, and to be male. This investigation demonstrates potential for NLP to identify INEADS patients, and may inform interventions to enhance advance care planning for patients who lack social support.

ObjectivesUrinary tract infection (UTI) is common in home care but not easily captured with standard assessment. This study aimed to examine the value of nursing notes in detecting UTI signs and symptoms in home care. DesignThe study developed a natural language processing (NLP) algorithm to automatically identify UTI-related information in nursing notes. Setting and ParticipantsHome care visit notes (n = 1,149,586) and care coordination notes (n = 1,461,171) for 89,459 patients treated in the largest nonprofit home care agency in the United States during 2014. MeasuresWe generated 6 categories of UTI-related information from literature and used the Unified Medical Language System (UMLS) to identify a preliminary list of terms. The NLP algorithm was tested on a gold standard set of 300 clinical notes annotated by clinical experts. We used structured Outcome and Assessment Information Set data to extract the frequency of UTI-related emergency department (ED) visits or hospitalizations and explored time-patterns in documentation of UTI-related information. ResultsThe NLP system achieved very good overall performance (F measure = 0.9, 95% CI: 0.87-0.93) based on the test results obtained by using the notes for patients admitted to the ED or hospital due to UTI. UTI-related information was significantly more prevalent (P < .01 for all the tests) in home care episodes with UTI-related ED admission or hospitalization vs the general patient population; 81% of home care episodes with UTI-related hospitalization or ED admission had at least 1 category of UTI-related information vs 21.6% among episodes without UTI-related hospitalization or ED admission. Frequency of UTI-related information documentation increased in advance of UTI-related hospitalization or ED admission, peaking within a few days before the event. Conclusions and ImplicationsInformation in nursing notes is often overlooked by stakeholders and not integrated into predictive modeling for decision-making support, but our findings highlight their value in early risk identification and care guidance. Health care administrators should consider using NLP to extract clinical data from nursing notes to improve early detection and treatment, which may lead to quality improvement and cost reduction.

Rare life-threatening conditions, such as multisystemic hereditary transthyretin amyloidosis (ATTRv) polyneuropathy, are often underdiagnosed or diagnosed late in the disease course, although early diagnosis is crucial for treatment success. Red flag symptoms have been identified, but manual screening of multidisciplinary medical records on this set of symptoms is time-consuming. This study aimed to validate a Natural Language Processing (NLP) algorithm to perform such a search in an automated manner, in order to improve early diagnosis and treatment. A novel state-of-the-art NLP procedure was applied to extract red flag symptoms from patients' electronic medical records and to select patients at risk for ATTRv polyneuropathy for further clinical review. Accuracy of the algorithm was assessed through comparison with a manual standard on a random sample of 300 patients. Out of a retrospective sample of 1015 patients, the NLP algorithm yielded 128 patients with three or more red flag symptoms of which 69 patients were considered eligible for genetic testing after clinical review. High accuracy was found in the detection of red flag symptoms, with F1 scores between 0.88 and 0.98. A relative increase of 48.6% in genetic testing, to identify patients with a rare disease earlier, was demonstrated. An NLP algorithm, after clinical validation, offers a valid and accurate tool to detect red flag symptoms in medical records across multiple disciplines, supporting better screening for patients with rare diseases. This opens the door to further NLP applications, facilitating rapid diagnosis and early treatment of rare diseases.

BackgroundAccurate identification of new diagnoses of human papillomavirus–associated cancers and precancers is an important step toward the development of strategies that optimize the use of human papillomavirus vaccines. The diagnosis of human papillomavirus cancers hinges on a histopathologic report, which is typically stored in electronic medical records as free-form, or unstructured, narrative text. Previous efforts to perform surveillance for human papillomavirus cancers have relied on the manual review of pathology reports to extract diagnostic information, a process that is both labor- and resource-intensive. Natural language processing can be used to automate the structuring and extraction of clinical data from unstructured narrative text in medical records and may provide a practical and effective method for identifying patients with vaccine-preventable human papillomavirus disease for surveillance and research.ObjectiveThis study's objective was to develop and assess the accuracy of a natural language processing algorithm for the identification of individuals with cancer or precancer of the cervix and anus.MethodsA pipeline-based natural language processing algorithm was developed, which incorporated machine learning and rule-based methods to extract diagnostic elements from the narrative pathology reports. To test the algorithm’s classification accuracy, we used a split-validation study design. Full-length cervical and anal pathology reports were randomly selected from 4 clinical pathology laboratories. Two study team members, blinded to the classifications produced by the natural language processing algorithm, manually and independently reviewed all reports and classified them at the document level according to 2 domains (diagnosis and human papillomavirus testing results). Using the manual review as the gold standard, the algorithm’s performance was evaluated using standard measurements of accuracy, recall, precision, and F-measure.ResultsThe natural language processing algorithm’s performance was validated on 949 pathology reports. The algorithm demonstrated accurate identification of abnormal cytology, histology, and positive human papillomavirus tests with accuracies greater than 0.91. Precision was lowest for anal histology reports (0.87, 95% CI 0.59-0.98) and highest for cervical cytology (0.98, 95% CI 0.95-0.99). The natural language processing algorithm missed 2 out of the 15 abnormal anal histology reports, which led to a relatively low recall (0.68, 95% CI 0.43-0.87).ConclusionsThis study outlines the development and validation of a freely available and easily implementable natural language processing algorithm that can automate the extraction and classification of clinical data from cervical and anal cytology and histology.

To examine the value of unstructured electronic health record (EHR) data (free-text notes) in identifying a set of geriatric syndromes. Retrospective analysis of unstructured EHR notes using a natural language processing (NLP) algorithm. Large multispecialty group. Older adults (N=18,341; average age 75.9, 58.9% female). We compared the number of geriatric syndrome cases identified using structured claims and structured and unstructured EHR data. We also calculated these rates using a population-level claims database as a reference and identified comparable epidemiological rates in peer-reviewed literature as a benchmark. Using insurance claims data resulted in a geriatric syndrome prevalence ranging from 0.03% for lack of social support to 8.3% for walking difficulty. Using structured EHR data resulted in similar prevalence rates, ranging from 0.03% for malnutrition to 7.85% for walking difficulty. Incorporating unstructured EHR notes, enabled by applying the NLP algorithm, identified considerably higher rates of geriatric syndromes: absence of fecal control (2.1%, 2.3 times as much as structured claims and EHR data combined), decubitus ulcer (1.4%, 1.7 times as much), dementia (6.7%, 1.5 times as much), falls (23.6%, 3.2 times as much), malnutrition (2.5%, 18.0 times as much), lack of social support (29.8%, 455.9 times as much), urinary retention (4.2%, 3.9 times as much), vision impairment (6.2%, 7.4 times as much), weight loss (19.2%, 2.9 as much), and walking difficulty (36.34%, 3.4 as much). The geriatric syndrome rates extracted from structured data were substantially lower than published epidemiological rates, although adding the NLP results considerably closed this gap. Claims and structured EHR data give an incomplete picture of burden related to geriatric syndromes. Geriatric syndromes are likely to be missed if unstructured data are not analyzed. Pragmatic NLP algorithms can assist with identifying individuals at high risk of experiencing geriatric syndromes and improving coordination of care for older adults.

Although metabolic dysfunction-associated steatotic liver disease (MASLD) and MASLD with increased alcohol intake (MetALD) are identified as clinical entities; tools to identify patients from electronic health records (EHRs) to perform large outcome studies are lacking. In this retrospective study of participants from the Veterans Analysis of Liver Disease cohort assembled from 1/1/2013 to 12/31/2022, a rule-based natural language processing (NLP) algorithm searched EHRs for imaging evidence of hepatic steatosis. This was combined with identification of cardiometabolic risk factors and harmful alcohol use. Algorithm-derived diagnoses of MASLD, MetALD, alcohol-associated steatotic liver disease (ALD), and no steatotic liver disease (SLD) were validated using a blinded review of randomly selected charts. Among 817,657 eligible veterans, SLD was present in over half (n = 438,209, 53.5%), including MASLD in 299,259 (36.5%), 99,163 with MetALD (12.1%), and 38,552 (4.7%) with ALD. The NLP algorithm had a high correlation with steatosis on chart review, with a κ of 0.86 (95% CI 0.82-0.90), sensitivity of 0.96, and specificity of 0.90. Classification of MASLD, MetALD, ALD, and no SLD by the algorithm also showed high correlation with chart review, with a κ of 0.87 (95% CI 0.82-0.91). This algorithm identified 299,259 (36.5%) of the study cohort with MASLD, compared with 23,218 patients (2.8%) identified using International Classification of Diseases-9/10 codes. An algorithm combining rule-based NLP with cardiometabolic risk factors and alcohol use from EHRs accurately identifies and classifies SLD and can be applied in large epidemiologic studies of SLD in the Veterans Health Administration.

PurposeWe aimed to develop a Natural Language Processing (NLP) algorithm to extract cognitive scores from electronic health records (EHR) data and compare them with cognitive function recorded by Centers for Medicare & Medicaid Services (CMS)-mandated clinical assessments in nursing homes and home health visits.Patients and MethodsWe identified a cohort of Medicare beneficiaries who had either the Minimum Data Set (MDS) or Outcome and Assessment Information Set (OASIS) linked to EHR data from the Research Patient Data Registry (Mass General Brigham, Boston, MA) from 2010 to 2019. We applied an NLP approach to identify the Montreal Cognitive Assessment (MoCA) and the Mini-Mental State Examination (MMSE) scores from unstructured clinician notes in EHR. Using the NLP-extracted MoCA or MMSE scores from EHR, we compared mean differences of extracted MoCA or MMSE by cognition status determined by MDS (impaired vs intact cognition) and OASIS (severe impairment vs intact cognition) data, respectively.ResultsOur study cohort had 7419 patients who had MDS (19.7%) or OASIS (80.3%) assessments, with a mean age of 80 (SD=7) years and 60% female. In EHR, the NLP algorithm extracted cognitive test scores with 97% accuracy (95% CI: 92–99%) for MoCA and 100% accuracy (95% CI: 84–100%) for MMSE. In MDS, the mean difference in extracted MoCA was −5.6 (95% CI: −8.7, −2.4, p=0.0008), and the mean difference in extracted MMSE was −7.9 (95% CI: −12.4, −3.5, p=0.0012). In OASIS, the mean difference in extracted MoCA and extracted MMSE was −4.8 (95% CI: −9.1, −0.6, p=0.0006) and −4.5 (95% CI: −9.5, −0.5, p=0.0182), respectively.ConclusionWe developed an NLP algorithm to accurately extract cognitive scores from unstructured EHR, and these extracted cognitive scores were well correlated with cognition function recorded in CMS-mandated clinical assessments. This could help researchers identify patients with various degrees of cognitive impairment in EHR-based research.

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

Centuries of biological knowledge are contained in the massive body of scientific literature, written for human-readability but too big for any one person to consume. Large-scale mining of information from the literature is necessary if biology is to transform into a data-driven science. A computer can handle the volume but cannot make sense of the language. This paper reviews and discusses the use of natural language processing (NLP) and machine-learning algorithms to extract information from systematic literature. NLP algorithms have been used for decades, but require special development for application in the biological realm due to the special nature of the language. Many tools exist for biological information extraction (cellular processes, taxonomic names, and morphological characters), but none have been applied life wide and most still require testing and development. Progress has been made in developing algorithms for automated annotation of taxonomic text, identification of taxonomic names in text, and extraction of morphological character information from taxonomic descriptions. This manuscript will briefly discuss the key steps in applying information extraction tools to enhance biodiversity science.

Financial stress, one of the social determinants, is common among cancer patients because of high out-ofpocket costs for treatment, as well as indirect costs. The National Academy of Medicine (NAM) has advised providers to recognize and discuss cost concerns with patients in order to enhance shared decision-making for treatment and exploration of financial assistant programs. However, financial stress is rarely assessed in clinical practice or research, thus, under-coded and under-documented in clinical practice. Natural language processing (NLP) offers great potential that can automatically extract and process data on financial stress from clinical free text existing in the patient electronic health record (EHR). We developed and evaluated an NLP approach to identify financial stress from clinical narratives for patients with prostate cancer. Of 4,195 eligible prostate cancer patients, we randomly sampled 3,138 patients (75%) as a training dataset (150,990 documents) to develop a financial stress lexicon and NLP algorithms iteratively. The remaining 1,057 patients (25%) were used as a test dataset (55,516 documents) to evaluate the NLP algorithm performance. The common terms representing financial stress were "financial concerns," "unable to afford," "insurance issue," "unemployed," and "financial assistance." Negations were used to exclude false mentions of financial stress. Applying both pre- and post-negation, the NLP algorithm identified 209 patients (6.0%) from the training sample and 66 patients (6.2%) with 161 notes from the test sample as having documented financial stress. Two independent domain experts manually reviewed all 161 notes with NLP identified positives and randomly selected 161 notes with NLP-identified negatives, the NLP algorithm yielded 0.86 for precision, 1 for recall, and 0.9.2 for F-score. Financial stress information is not commonly documented in the EHR, neither in structured format nor in clinical narratives. However, natural language processing can accurately extract financial stress information from clinical notes when such narrative information is available.