Identify diabetic retinopathy-related clinical concepts and their attributes using transformer-based natural language processing methods

Diabetic retinopathy (DR) is a leading cause of blindness in American adults. If detected early, DR can be treated to preventing further damage causing blindness, therefore, early detection is very important for the treatment of DR. There is an increasing interest in developing Al technologies to help early detection of DR using electronic health records (EHR). The detailed diagnoses information documented in image reports is a valuable resource that could help detect lesions from the medical image, thus helping early detection of DR. In this study, we examined two state-of-the-art transformer-based natural language processing models, including BERT and RoBERTa, to extract DR-related concepts from clinical narratives. We identified four different categories of DR-related clinical concepts including lesions, eye parts, laterality, and severity, developed annotation guidelines, annotated a DR-corpus of 536 image reports, and trained four transformer-based NLP models for clinical concept extraction. The experimental results show that the BERT model pretrained with the MIMIC III dataset achieved the best strict/lenient F1-score of 0.9503 and 0.9645, respectively.

447 Background: Mortality from bladder cancer (BC) increases exponentially once it invades the muscle. At the population level, accurate delineation of these patients is challenging. Methods: To develop and validate a natural language processing (NLP) model for automatically identifying muscle-invasive BC (MIBC) patients, aiding in population-based BC research. All patients with a CPT code for transurethral resection of bladder tumor (TURBT) (N = 76,060) were selected from the Department of Veterans Affairs (VA) Corporate Data Warehouse database. A sample of 600 patients (with 2,337 full-text notes) who had TURBT and confirmed pathology results were selected for NLP model development (500 patients) and validation (100 patients). Muscle-invasion (yes/no), unknown, or no cancer, were confirmed by detailed chart review of pathology notes. The NLP performance was assessed by calculating the sensitivity, positive predictive value (PPV), and overall accuracy at the individual note and patient levels. Results: In the validation cohort, the NLP model had overall accuracy of 88% and 92% at the note and patient levels. Specifically, PPV and specificity for predicting muscle-invasion on note level were 83% and 70%, respectively. The model classified non-muscle invasive BC (NMIBC) with 98% sensitivity at both the note and patient levels. Although the sensitivity for MIBC was 70% for note-level determination, the sensitivity was 86% when evaluated at the patient level. When applying the model to 71,200 patients VA-wide, the model classified 13,642 (19%) as having MIBC and 47,595 (66%) as NMIBC. The NLP model was able to identify invasion status for 96% TURBT patients at the population level. Inherent limitations include relatively small training set given the size of the VA population. Conclusions: This NLP model for identifying muscle-invasion at the population level had high accuracy. The NLP model may be a practical and accurate tool for efficiently identifying BC invasion status and may potentially aid in population-based BC research in the VA.

To develop and evaluate an internally validated natural language processing (NLP) model to determine guideline adherence of antibiotic decisions in the emergency department (ED) for skin and soft tissue infections (SSTIs). This cross-sectional pilot study developed and applied a random forest (RF)/NLP model to classify clinical narratives of patients with skin infections as requiring either methicillin-resistant Staphylococcus aureus (MRSA) or non-MRSA antibiotic coverage. Conducted at a quaternary care children's hospital with an annual volume of 50,000 ED visits, our study included patients aged 1 to 18 years presenting for SSTIs to the ED from July 1, 2018, through June 30, 2023. Main outcomes included the NLP model's sensitivity, specificity, and receiver operator characteristic (ROC) curve compared to gold standard manual physician review of the medical records. A total of 1588 patients were part of the training data set, with an additional 423 patients utilized for validation. The RF model achieved an area under the curve (AUC) of 0.99, 97.0% sensitivity, and 94.9% specificity. In the validation data set, the model achieved 96.6% sensitivity and 90.1% specificity. Performance remained strong despite absent/missing patient history in certain narratives. The NLP model demonstrated that automated analysis of clinical narratives for determining guideline adherence is feasible. Despite missing data in narratives, the model's high performance suggests potential for broader application. Given the rise in antibiotic-resistant infections and the role of judicious antibiotic use, developing automated systems through NLP can significantly contribute to health care delivery and patient safety. The methodology of this study provides a feasible, sustainable path for similar applications in the emergency setting.

Ontology learning from unstructured text has become a critical task for knowledge-driven applications in Big Data and Artificial Intelligence. While significant advances have been made in the automatic extraction of concepts and relations using neural and Transformer-based models, the generation of formal Description Logic axioms required for constructing logically consistent and computationally tractable ontologies remains largely underexplored. This paper puts forward a novel pipeline for automated axiom generation through schema mapping. Our paper introduces three key innovations: a deterministic mapping framework that guarantees logical consistency (unlike stochastic Large Language Models); guaranteed formal consistency verified by OWL reasoners (unaddressed by prior statistical methods); and a transparent, scalable bridge from neural extractions to symbolic logic, eliminating manual post-processing. Technically, the pipeline builds upon the outputs of a Transformer-based fusion model for joint concept and relation extraction. We then map lexical relational phrases to formal ontological properties through a lemmatization-based schema alignment step. Entity typing and hierarchical induction are then employed to infer class structures, as well as domain and range constraints. Using RDFLib and structured data processing, we transform the extracted triples into both assertional (ABox) and terminological (TBox) axioms expressed in Description Logic. Experimental evaluation on benchmark datasets (Conll04 and NYT) demonstrates the efficacy of the approach, with expert validation showing high acceptance rates (>95%) and reasoners confirming zero inconsistencies. The pipeline thus establishes a reliable, scalable foundation for automated ontology learning, advancing the field from extraction to formally verifiable knowledge base construction.

Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction

Natural language processing with deep learning for medical adverse event detection from free-text medical narratives: A case study of detecting total hip replacement dislocation

Understanding treatment patterns and effectiveness for patients with metastatic prostate cancer (mPCa) is dependent on accurate assessment of metastatic status. The objective was to develop a natural language processing (NLP) model for identifying patients with mPCa and evaluate the model's performance against chart-reviewed data and an International Classification of Diseases (ICD) 9/10 code-based method. In total, 139,057 radiology reports on 6,211 unique patients from the Department of Veterans Affairs were used. The gold standard was metastases by detailed chart review of radiology reports. NLP performance was assessed by sensitivity, specificity, positive predictive value, negative predictive value, and date of metastases detection. Receiver operating characteristic curves was used to assess model performance. When compared with chart review, the NLP model had high sensitivity and specificity (85% and 96%, respectively). The NLP model was able to predict patient-level metastasis status with a sensitivity of 91% and specificity of 81%, whereas sensitivity and specificity using ICD9/10 billing codes were 73% and 86%, respectively. For the NLP model, date of metastases detection was exactly concordant and within < 1 week in 55% and 58% of patients, compared with 8% and 17%, respectively, using the ICD9/10 billing codes method. The area under the curve for the NLP model was 0.911. A limitation is the NLP model was developed on the basis of a subset of patients with mPCa and may not be generalizable to all patients with mPCa. This population-level NLP model for identifying patients with mPCa was more accurate than using ICD9/10 billing codes when compared with chart-reviewed data. Upon further validation, this model may allow for efficient population-level identification of patients with mPCa.

Relation extraction is a natural language processing (NLP) task to extract semantic relations between entities from unstructured texts, and is an important research content of constructing knowledge graph. Based on the documents issued by the China Banking and Insurance Regulatory Commission, we construct a knowledge graph in the field of financial regulation through relation extraction, which helps bank staff to realize intelligent query of regulatory documents and locate business-related regulatory requirements quickly and accurately. This paper proposes the Financial Regulation BERT (FR-BERT) model for relation extraction in the financial regulation field, aiming at the dataset's characteristics of rich semantic information of target entities, clear keywords and long texts. FR-BERT uses the BERT model to obtain sentence vector information containing each word vector, then obtains the target entity vectors and keyword vector by locating the sentence vector information, and obtains the text vector by sending the whole sentence vector information into Bi-directional Long Short-Term Memory (BiLSTM). Finally, the model uses the above vectors for classification information. FR-BERT integrates target entity vectors, keyword vector, and text vector output by BiLSTM to achieve relation extraction between entities. Compared with other relation extraction models selected in this paper, the experimental results show that FR-BERT performs better such as Macro-F1 on the relation extraction task in the financial regulation field, which verifies the effectiveness of the method.

108 Background: Classification of disease response is an essential task in cancer research and needs to be done at scale. Automating this process can improve efficiency in the generation of real-world evidence, potentially leading to better patient outcomes. We aim to develop and evaluate Natural Language Processing (NLP) models for this task. Methods: Using 6203 computed tomography (CT) and 1358 magnetic resonance imaging (MRI) reports from 587 patients with lung cancer of all stages seen at the National Cancer Centre Singapore (NCCS), we trained four NLP models (BioBERT, RadBERT-RoBERTA, BioClinicalBERT, GatorTron) to classify the reports into one of four categories: no evidence of disease, stable disease, partial response or disease progression. Model output was compared against human-curated ground truth and performance was evaluated by accuracy. Results: Of the 4 models, GatorTron performed the best (accuracy = 97.1%), followed by RadBERT-RoBERTA (accuracy = 96.2%), BioBERT (accuracy = 94.2%), with BioClinicalBERT being last (accuracy = 90.4%). NLP Model runtimes for the dataset were relatively short, with BioBERT and BioClinicalBERT taking 3 minutes per epoch, RadBERT-RoBERTA taking 6 minutes per epoch, and GatorTron taking 10 minutes per epoch on a single central processing unit (CPU). Conclusions: We have demonstrated the effectiveness of NLP models for classifying disease responses in radiology reports of lung cancer patients. This has the potential to help derive progression-free survival for real-world evidence generation.

Current research on knowledge graph construction is focused chiefly on general-purpose fields, whereas constructing knowledge graphs in vertically segmented professional fields faces numerous difficulties. To solve the problems of complex relation types of domain entities, the lack of a large amount of annotated corpus, and the difficulty of extraction, this study proposed a method for constructing domain-annotated datasets based on publicly available texts on the web, which integrates remote supervision and semi-supervision. For the relational triad extraction of a given core entity (an entity lexicon defined semi-automatically by experts), an inflated gate attention network structure for increasing the perceptual field of the model is proposed. In addition, a relational extraction model, Ro-DGANet, was designed based on this structure, incorporating the idea of a probability graph. The Ro-DGANet model was experimentally evaluated on the publicly available Chinese datasets LIC2019 and CHIP2020 and compared with the mainstream relation extraction models, achieving the best results with F1 values of 82.99% and 66.39%, respectively. Finally, the Ro-DGANet model was applied to the relation extraction task of equipment components in industrial scenarios and to the relation extraction task of core knowledge points of programming languages. The analysis results show that the proposed method is applicable to open relation extraction among core entities in different domains with reliable performance and portability.

Transformer architectures are highly expressive because they use self-attention mechanisms to encode long-range dependencies in the input sequences. In this paper, we present a literature review on Transformer-based (TB) models, providing a detailed overview of each model in comparison to the Transformer’s standard architecture. This survey focuses on TB models used in the field of Natural Language Processing (NLP) for textual-based tasks. We begin with an overview of the fundamental concepts at the heart of the success of these models. Then, we classify them based on their architecture and training mode. We compare the advantages and disadvantages of popular techniques in terms of architectural design and experimental value. Finally, we discuss open research, directions, and potential future work to help solve current TB application challenges in NLP.

In secondary analysis of electronic health records, a crucial task consists in correctly identifying the patient cohort under investigation. In many cases, the most valuable and relevant information for an accurate classification of medical conditions exist only in clinical narratives. Therefore, it is necessary to use natural language processing (NLP) techniques to extract and evaluate these narratives. The most commonly used approach to this problem relies on extracting a number of clinician-defined medical concepts from text and using machine learning techniques to identify whether a particular patient has a certain condition. However, recent advances in deep learning and NLP enable models to learn a rich representation of (medical) language. Convolutional neural networks (CNN) for text classification can augment the existing techniques by leveraging the representation of language to learn which phrases in a text are relevant for a given medical condition. In this work, we compare concept extraction based methods with CNNs and other commonly used models in NLP in ten phenotyping tasks using 1,610 discharge summaries from the MIMIC-III database. We show that CNNs outperform concept extraction based methods in almost all of the tasks, with an improvement in F1-score of up to 26 and up to 7 percentage points in area under the ROC curve (AUC). We additionally assess the interpretability of both approaches by presenting and evaluating methods that calculate and extract the most salient phrases for a prediction. The results indicate that CNNs are a valid alternative to existing approaches in patient phenotyping and cohort identification, and should be further investigated. Moreover, the deep learning approach presented in this paper can be used to assist clinicians during chart review or support the extraction of billing codes from text by identifying and highlighting relevant phrases for various medical conditions.

Relations extraction is a widely researched topic in nature language processing. However, most of the work in the literature concentrate on the methods that are dealing with single relation between two named entities. In the task of multiple relations extraction, traditional statistic-based methods have difficulties in selecting features and improving the performance of extraction model. In this paper, we presented formal definitions of multiple entities and multiple relations and put forward three labeling methods which were used to label entity categories, relation categories and relation conditions. We also proposed a novel relation extraction model which is based on dynamic long short-term memory network. To train our model, entity feature, entity position feature and part of speech feature are used together. These features are used to describe complex relations and improve the performance of relation extraction model. In the experiments, we classified the corpus into three sets which are composed of 0–20 words, 20–35 words and 35+ words sentences. On conll04.corp, the final precision, recall rate and F-measure reached 72.9, 70.8 and 67.9% respectively.

&lt;span lang="EN-US"&gt;In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, &lt;a name="_Hlk158992630"&gt;&lt;/a&gt;bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. &lt;/span&gt;

Medical educators pay increasing attention to the potential utility of narrative data for assessment, but lack of efficient and standardized ways of interpreting the data have limited its use. Natural language processing (NLP) algorithms could provide new insights for using narrative data within assessment especially identifying at-risk or low-performing students. Assessment data were reviewed from 16 cohorts of medical students from the University of Cincinnati College of Medicine (graduating classes of 2006-2022). A T-score Average (TSA) was calculated for each student based on clerkship assessment data. Narrative data from core clerkship evaluations in responses to the prompt "any opportunities for improvement" were utilized for analysis. The narrative data and calculated TSA were then used to train and test 4 NLP models with the goal of utilizing NLP to identify at-risk students as defined by the bottom 10% average TSA score. Based on typical NLP methods, the developed NLP models all performed adequately in identifying student performance with an overall accuracy of 0.8 across all 4 models. However, none of the NLP models was able to identify students within the bottom 10% of performance. During this process, we uncovered the presence of "copy/paste" behavior, a previously undocumented phenomenon within narrative data where preceptors duplicated comments from 1 student to the next. Training NLP models including "copy/paste" comments improved NLP ability to identify students within the bottom and top 10% of performance. NLP models were unable to accurately identify at-risk students. Model accuracy increased with the inclusion of "copy/paste" comments, indicating there may be some discriminatory functionality within aspects of narrative data beyond keywords such as lexical diversity and word quantity. Future work will explore how these other aspects of narrative data associate with performance and utilize more advanced large language models of analysis.Based on typical natural language processing (NLP) methods, the tested NLP models performed adequately in identifying student performance with an overall accuracy of 0.8 across all models. However, none of the NLP models was able to identify students within the bottom 10% of performance, revealing widespread "copy/paste" behavior in evaluations that may limit narrative data's discriminatory utility.