An Improved Nested Named-Entity Recognition Model for Subject Recognition Task under Knowledge Base Question Answering

Current work in named entity recognition (NER) uses either cross entropy (CE) or conditional random fields (CRF) as the objective/loss functions to optimize the underlying NER model. Both of these traditional objective functions for the NER problem generally produce adequate performance when the data distribution is balanced and there are sufficient annotated training examples. But since NER is inherently an imbalanced tagging problem, the model performance under the low-resource settings could suffer using these standard objective functions. Based on recent advances in area under the ROC curve (AUC) maximization, we propose to optimize the NER model by maximizing the AUC score. We give evidence that by simply combining two binary-classifiers that maximize the AUC score, significant performance improvement over traditional loss functions is achieved under low-resource NER settings. We also conduct extensive experiments to demonstrate the advantages of our method under the low-resource and highly-imbalanced data distribution settings. To the best of our knowledge, this is the first work that brings AUC maximization to the NER setting. Furthermore, we show that our method is agnostic to different types of NER embeddings, models and domains. The code of this work is available at https://github.com/dngu0061/NER-AUC-2T.

We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. The biggest challenge of distantly-supervised NER is that the distant supervision may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this paper, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextualized augmentations created by pre-trained language models to improve the generalization ability of the NER model. On three benchmark datasets, our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.

The biggest problem confronted by named entity recognition (NER) in practical applications is that the number of labeled corpora in most application domains is small due to the high labeling cost; Only a few domains have a large number of well labeled corpora. The lack of labeled corpora is the biggest problem that restricts the application of NER in reality. To address the above problems, we propose T-NER, a model based on transfer learning, in this paper. Structurally, T-NER is of multi-model fusion architecture: the multi-dimensional and multi-level fusion of the sequence learning models BiLSTM and BiGRU is performed; In terms of method, T-NER is a transfer learning based on parameter sharing, which uses the model to train the richly labeled corpora in the source domain, and shares the model parameters of the trained corpora in the source domain to the target domain, so as to transfer the knowledge of the source domain to the target domain. The implementation of model parameter sharing is based on the loop structure of T_NER. After the corpus in the source domain is trained, its model parameters will return to the initial part of the model by utilizing the loop structure of the model to participate in the training of the target domain. In the testing process, the training speed and training results of T_NER in small sample experiments are better than the base model, which shows the strong superiority of the T_NER model in the NER of small samples; In large sample experiments, the effect of the T_NER model is not inferior to the base model. As far as the current experimental results are concerned, the T_NER model can realize the training of small-sample and fast transferable NER model.

Developing robust and reliable models for Named Entity Recognition (NER) in the Russian language presents significant challenges due to the linguistic complexity of Russian and the limited availability of suitable training datasets. This study introduces a semi-automated methodology for building a customized Russian dataset for NER specifically designed for literary purposes. The paper provides a detailed description of the methodology employed for collecting and proofreading the dataset, outlining the pipeline used for processing and annotating its contents. A comprehensive analysis highlights the dataset’s richness and diversity. Central to the proposed approach is the use of a voting system to facilitate the efficient elicitation of entities, enabling significant time and cost savings compared to traditional methods of constructing NER datasets. The voting system is described theoretically and mathematically to highlight its impact on enhancing the annotation process. The results of testing the voting system with various thresholds show its impact in increasing the overall precision by 28% compared to using only the state-of-the-art model for auto-annotating. The dataset is meticulously annotated and thoroughly proofread, ensuring its value as a high-quality resource for training and evaluating NER models. Empirical evaluations using multiple NER models underscore the dataset’s importance and its potential to enhance the robustness and reliability of NER models in the Russian language.

Named entity recognition (NER) plays a vital role in extracting critical medical entities from health care records, facilitating applications such as clinical decision support and data mining. Developing robust NER models for low-resource languages, such as Estonian, remains a challenge due to the scarcity of annotated data and domain-specific pretrained models. Large language models (LLMs) have proven to be promising in understanding text from any language or domain. This study addresses the development of medical NER models for low-resource languages, specifically Estonian. We propose a novel approach by generating synthetic health care data and using LLMs to annotate them. These synthetic data are then used to train a high-performing NER model, which is applied to real-world medical texts, preserving patient data privacy. Our approach to overcoming the shortage of annotated Estonian health care texts involves a three-step pipeline: (1) synthetic health care data are generated using a locally trained GPT-2 model on Estonian medical records, (2) the synthetic data are annotated with LLMs, specifically GPT-3.5-Turbo and GPT-4, and (3) the annotated synthetic data are then used to fine-tune an NER model, which is later tested on real-world medical data. This paper compares the performance of different prompts; assesses the impact of GPT-3.5-Turbo, GPT-4, and a local LLM; and explores the relationship between the amount of annotated synthetic data and model performance. The proposed methodology demonstrates significant potential in extracting named entities from real-world medical texts. Our top-performing setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction. These results indicate a strong performance in recognizing certain entity types while highlighting the complexity of extracting procedures. This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method's applicability to other domains and languages.

The rise of chronic diseases and pandemics, such as COVID-19 has emphasized the need for effective patient data processing while ensuring privacy through anonymization and de-identification of protected health information. Anonymized data facilitates research without compromising patient confidentiality. This paper introduces expert small artificial intelligence (AI) models developed using the large language model (LLM)-in-the-loop methodology to meet the demand for domain-specific de-identification of named entity recognition (NER) models. These models overcome the privacy risks associated with LLMs used through application programming interfaces by eliminating the need to transmit or store sensitive data. More importantly, they consistently outperform LLMs in de-identification tasks, offering superior performance and reliability. Our de-identification NER models, developed in eight languages—English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic—achieved F1-macro score averages of 0.931, 0.960, 0.955, 0.937, 0.930, 0.963, 0.957, and 0.922, respectively. These results establish our de-identification NER models as the most accurate healthcare anonymization solutions, surpassing existing small models and even general-purpose LLMs, such as GPT-4o. While Part I of this series introduced the LLM-in-the-loop methodology for biomedical document translation, this second paper showcases its success in developing cost-effective expert small NER models in de-identification tasks. Our findings lay the groundwork for future healthcare AI innovations, including biomedical entity and relation extraction, demonstrating the value of specialized models for domain-specific challenges.

Many clinically significant health conditions in older adults are underreported or only recorded in unstructured health records. These records, however, contain valuable information for patient care and prognosis. This study utilized 10.6 million free-text entries from the electronic health records of 102,525 patients aged 50–80 across various care settings in Finland from 2010 to 2022. A deep learning-based natural language processing model was employed to perform named entity recognition (NER) to identify falls, incontinence, loneliness, and mobility limitations from the free-text entries. The performance of the NER models was evaluated by precision, recall and F1 scores. Diagnostic codes for incontinence and falls were collected for comparisons. Cox regression models were used to assess the predictive value of the identified conditions for all-cause mortality. The NER models demonstrated excellent performance with recall, precision and F1 scores greater than 0.80 across the health conditions. Compared to diagnostic codes, NER identified greater numbers of falls (31987 vs 4090) and incontinence (7059 vs 3873) onsets and yielded greater hazard ratios for all-cause mortality: 1.31 vs 1.04 for falls and 1.99 vs 0.65 for incontinence. Deep learning-based NER models present new opportunities to identify vulnerable patients in free text health records.

Named Entity Recognition (NER) is a core task of NLP. State-of-art supervised NER models rely heavily on a large amount of high-quality annotated data, which is quite expensive to obtain. Various existing ways have been proposed to reduce the heavy reliance on large training data, but only with limited effect. In this paper, we propose a novel way to make full use of the weakly-annotated texts in encyclopedia pages for exactly unsupervised NER learning, which is expected to provide an opportunity to train the NER model with no manually-labeled data at all. Briefly, we roughly divide the sentences of encyclopedia pages into two parts simply according to the density of inner url links contained in each sentence. While a relatively small number of sentences with dense links are used directly for training the NER model initially, the left sentences with sparse links are then smartly selected for gradually promoting the model in several self-training iterations. Given the limited number of sentences with dense links for training, a data augmentation method is proposed, which could generate a lot more training data with the help of the structured data of encyclopedia to greatly augment the training effect. Besides, in the iterative self-training step, we propose to utilize a graph model to help estimate the labeled quality of these sentences with sparse links, among which those with the highest labeled quality would be put into our training set for updating the model in the next iteration. Our empirical study shows that the NER model trained with our unsupervised learning approach could perform even better than several state-of-art models fully trained on newswires data.

Named entity recognition (NER) in Chinese electronic medical records (EMRs) has become an important task of clinical natural language processing (NLP). However, limited studies have been performed on the clinical NER study in Chinese EMRs. Furthermore, when end-to-end neural network models have improved clinical NER performance, medical knowledge dictionaries such as various disease association dictionaries, which provide rich information of medical entities and relations among them, are rarely utilized in NER model. In this study, we investigate the problem of NER in Chinese EMRs and propose a clinical neural network NER model enhanced with medical knowledge attention by combining the entity mention information contained in external medical knowledge bases with EMR context together. Experimental results on the manually labeled dataset demonstrated that the proposed method can achieve better performance than the previous methods in most cases.

Intent classification (IC) and Named Entity Recognition (NER) are arguably the two main components needed to build a Natural Language Understanding (NLU) engine, which is a main component of conversational agents. The IC and NER components are closely intertwined and the entities are often connected to the underlying intent. Current research has primarily focused to model IC and NER as two separate units, which results in error propagation, and thus, sub-optimal performance. In this paper, we propose a simple yet effective novel framework for NLU where the parameters of the IC and the NER models are jointly trained in a consolidated parameter space. Text semantic representations are obtained from popular pre-trained contextual language models, which are fine-tuned for our task, and these parameters are propagated to other deep neural layers in our framework leading to a faithful unified modelling of the IC and NER parameters. The overall framework results in a faithful parameter sharing when the training is underway, leading to a more coherent learning. Experiments on two public datasets, ATIS and SNIPS, show that our model outperforms other methods by a noticeable margin. On the SNIPS dataset, we obtain a 1.42% improvement in NER in terms of the F1 score, and 1% improvement in intent accuracy score. On ATIS, we achieve 1.54% improvement in intent accuracy score. We also present qualitative results to showcase the effectiveness of our model.

Recent advances in deep neural networks (DNNs) have enabled us to achieve reliable named entity recognition (NER) models without handcrafting features. However, these are also some obstacles imposed by using those machine learning methods, in need of a large amount of manually labeled data. To avoid such limitations, we could replace human annotation with distant supervision, however there remain a technical challenge on the error label issue caused by ignoring the entities that are not included in the vocabulary, which should be addressed to achieve the effective NER model. Then, we propose a novel back-labeling approach and integrate it into a tagging scheme, especially, we apply this scheme to handle the NER task in traditional Chinese medicine (TCM) field. In addition, we discuss how to use distant supervision methods to achieve better performance of the NER model. We conduct some experiments and verify that our scheme can effectively improve the entity recognition on the basis of distant supervision.

Medical named entity recognition via lattice-enhanced transfer learning with random attention

Threat intelligence named entity recognition techniques based on few-shot learning

Aiming at the problem of insufficient labeled data in the medical field, a Named Entity Recognition model that introduces counterfactual mechanism to enhance vocabulary is proposed in this article. With the idea of Semi-Supervised Learning, the model aims at a small amount of labeled data, builds a counterfactual vocabulary generator that captures more dependencies to enhance medical data by introducing and improving the counterfactual mechanism in the structural causal model. Further more, a vocabulary information fusion recognizer is constructed to verify the effectiveness of the data. The recognizer integrates character feature embedding, vocabulary information feature embedding in training data, and position feature embedding. While achieving medical vocabulary enhancement, it also solves the problem of inaccurate entity recognition and improves the accuracy of entity recognition. Through comparative and ablation experiments, it is shown that the named entity recognition model with counterfactual mechanism in this paper achieved an F1 score of 84.67% and 86.15% on the CCKS2019 and CCKS2020 datasets, respectively, which were 0.22%–4.57% and 0.315–5.04% higher than other related models, and 3.82% and 3.86% higher than traditional counterfactual generators, respectively, proving the effectiveness of the model.

Tomato leaf pests and diseases pose a significant threat to the yield and quality of Q6 tomatoes, highlighting the necessity for comprehensive studies on effective control methods. Current control measures predominantly rely on experience and manual observation, hindering the integration of multi-source data. To address this, we integrated information resources related to tomato leaf pests and diseases from agricultural standards documents, knowledge websites, and relevant literature. Guided by domain experts, we preprocessed this data to construct a sample set. We utilized the Named Entity Recognition (NER) model ALBERT-BiLSTM-CRF to conduct end-to-end knowledge extraction experiments, which outperformed traditional models such as 1DCNN-CRF and BiLSTM-CRF, achieving a recall rate of 95.03%. The extracted knowledge was then stored in the Neo4j graph database, effectively visualizing the internal structure of the knowledge graph. We developed a digital diagnostic system for tomato leaf pests and diseases based on the knowledge graph, enabling graphical management and visualization of pest and disease knowledge. The constructed knowledge graph offers insights for controlling tomato leaf pests and diseases and provides new research directions for pest control in other crops.