AUC Maximization for Low-Resource Named Entity Recognition

The biggest problem confronted by named entity recognition (NER) in practical applications is that the number of labeled corpora in most application domains is small due to the high labeling cost; Only a few domains have a large number of well labeled corpora. The lack of labeled corpora is the biggest problem that restricts the application of NER in reality. To address the above problems, we propose T-NER, a model based on transfer learning, in this paper. Structurally, T-NER is of multi-model fusion architecture: the multi-dimensional and multi-level fusion of the sequence learning models BiLSTM and BiGRU is performed; In terms of method, T-NER is a transfer learning based on parameter sharing, which uses the model to train the richly labeled corpora in the source domain, and shares the model parameters of the trained corpora in the source domain to the target domain, so as to transfer the knowledge of the source domain to the target domain. The implementation of model parameter sharing is based on the loop structure of T_NER. After the corpus in the source domain is trained, its model parameters will return to the initial part of the model by utilizing the loop structure of the model to participate in the training of the target domain. In the testing process, the training speed and training results of T_NER in small sample experiments are better than the base model, which shows the strong superiority of the T_NER model in the NER of small samples; In large sample experiments, the effect of the T_NER model is not inferior to the base model. As far as the current experimental results are concerned, the T_NER model can realize the training of small-sample and fast transferable NER model.

In the subject recognition (SR) task under Knowledge Base Question Answering (KBQA), a common method is by training and employing a general flat Named-Entity Recognition (NER) model. However, it is not effective and robust enough in the case that the recognized entity could not be strictly matched to any subjects in the Knowledge Base (KB). Compared to flat NER models, nested NER models show more flexibility and robustness in general NER tasks, whereas it is difficult to employ a nested NER model directly in an SR task. In this paper, we take advantage of features of a nested NER model and propose an Improved Nested NER Model (INNM) for the SR task under KBQA. In our model, each question token is labeled as either an entity token, a start token, or an end token by a modified nested NER model based on semantics. Then, entity candidates would be generated based on such labels, and an approximate matching strategy is employed to score all subjects in the KB based on string similarity to find the best-matched subject. Experimental results show that our model is effective and robust to both single-relation questions and complex questions, which outperforms the baseline flat NER model by a margin of 3.3% accuracy on the SimpleQuestions dataset and a margin of 11.0% accuracy on the WebQuestionsSP dataset.

Developing robust and reliable models for Named Entity Recognition (NER) in the Russian language presents significant challenges due to the linguistic complexity of Russian and the limited availability of suitable training datasets. This study introduces a semi-automated methodology for building a customized Russian dataset for NER specifically designed for literary purposes. The paper provides a detailed description of the methodology employed for collecting and proofreading the dataset, outlining the pipeline used for processing and annotating its contents. A comprehensive analysis highlights the dataset’s richness and diversity. Central to the proposed approach is the use of a voting system to facilitate the efficient elicitation of entities, enabling significant time and cost savings compared to traditional methods of constructing NER datasets. The voting system is described theoretically and mathematically to highlight its impact on enhancing the annotation process. The results of testing the voting system with various thresholds show its impact in increasing the overall precision by 28% compared to using only the state-of-the-art model for auto-annotating. The dataset is meticulously annotated and thoroughly proofread, ensuring its value as a high-quality resource for training and evaluating NER models. Empirical evaluations using multiple NER models underscore the dataset’s importance and its potential to enhance the robustness and reliability of NER models in the Russian language.

Evaluation of clinical named entity recognition methods for Serbian electronic health records

In the era of information explosion, named entity recognition (NER) has attracted widespread attention in the field of natural language processing, as it is fundamental to information extraction. Recently, methods of NER based on representation learning, e.g., character embedding and word embedding, have demonstrated promising recognition results. However, existing models only consider partial features derived from words or characters while failing to integrate semantic and syntactic information, e.g., capitalization, inter-word relations, keywords, and lexical phrases, from multilevel perspectives. Intuitively, multilevel features can be helpful when recognizing named entities from complex sentences. In this study, we propose a novel attentive multilevel feature fusion (AMFF) model for NER, which captures the multilevel features in the current context from various perspectives. It consists of four components to, respectively, capture the local character-level (CL), global character-level (CG), local word-level (WL), and global word-level (WG) features in the current context. In addition, we further define document-level features crafted from other sentences to enhance the representation learning of the current context. To this end, we introduce a novel context-aware attentive multilevel feature fusion (CAMFF) model based on AMFF, to fully leverage document-level features from all the previous inputs. The obtained multilevel features are then fused and fed into a bidirectional long short-term memory (BiLSTM)-conditional random field (CRF) network for the final sequence labeling. Extensive experiments on four benchmark datasets demonstrate that our proposed AMFF and CAMFF models outperform a set of state-of-the-art baseline methods and the features learned from multiple levels are complementary.

Medical named entity recognition via lattice-enhanced transfer learning with random attention

Named entity recognition (NER) in Chinese electronic medical records (EMRs) has become an important task of clinical natural language processing (NLP). However, limited studies have been performed on the clinical NER study in Chinese EMRs. Furthermore, when end-to-end neural network models have improved clinical NER performance, medical knowledge dictionaries such as various disease association dictionaries, which provide rich information of medical entities and relations among them, are rarely utilized in NER model. In this study, we investigate the problem of NER in Chinese EMRs and propose a clinical neural network NER model enhanced with medical knowledge attention by combining the entity mention information contained in external medical knowledge bases with EMR context together. Experimental results on the manually labeled dataset demonstrated that the proposed method can achieve better performance than the previous methods in most cases.

The objective was to study the dynamics of B-type natriuretic peptide (BNP) and its relationship with hemodynamic parameters during on-pump coronary artery bypass grafting (CABG), and to evaluate the informativeness of the biomarker as a predictor of myocardial dysfunction.Materials and methods. The study involved 127 patients aged 59 [54–66.75] years with ischemic heart disease who underwent CABG. The BNP blood level was determined in the operating room at stages: I – before surgery (BNP1 ), II – at the end of surgery (BNP2 ). Hemodynamic parameters were analyzed at the same stages. Correlation analysis, logistic regression with the calculation of the odds ratio (OR) and 95% confidence interval (95% CI) and ROC analysis with the calculation of the area under the ROC curve (AUC) were used.Results. BNP1 blood level was 49 [25.6–91.6], BNP2 – 90 [47.8–140.2] pg/ml (p < 0.0001). BNP1 correlated with central venous pressure (CVP) at stage I (rho = 0.212; p = 0.017) and with pulmonary artery wedge pressure (PAWP) at stage II (rho = 0.204; p = 0.045). BNP2 correlated with PAWP at stage II (rho = 0.204; p = 0.045). BNP1 > 52.1 pg/ml was the predictor of ICU length of stay > 24 hours (OR 1.0290, 95% CI 1.0154– 1.0427, p < 0.0001, AUC 0.775), BNP1 > 71 pg/ml was the predictor of inotropic index > 5 c. u. (OR 1.0076, 95% CI 1.0015–1.0138, p = 0.014, AUC 0.705) and BNP1 > 90.8 pg/ml was the predictor of vasoactive inotropic index > 10 c. u. (OR 1.0070, 95% CI 1.0014–1.0126, p = 0.013, AUC 0.727). BNP2 > 67.5 pg/ml was the predictor of ICU length of stay > 24 hours (OR 1.0179, 95% CI 1.0073–1.0287, p < 0.0009, AUC 0.763), BNP2 > 94.3 pg/ml was the predictor of inotropic index > 5 c. u. (OR 1.0063, 95% CI 1.0010–1.0117, p = 0.020, AUC 0.713), BNP2 > 144 pg/ml was the predictor of intra-aortic balloon pumping (OR 1.0037, 95% CI 1 .0000–1.0074, p = 0.048, AUC 0.854), BNP2 > 159 pg/ml was the predictor of vasoactive inotropic index > 10 c. u. (OR 1.0072, 95% CI 1.0006–1.0139, p = 0.033, AUC 0.729) and BNP2 > 161 pg/ml was the predictor of early mortality in the ICU (OR 1.0040, 95% CI 1, 0000-1.0080, p = 0.049, AUC 0.845). Conclusion. In 78.7% of patients undergoing on-pump CABG, BNP blood level does not exceed the upper limit of normal; by the end of surgery, the biomarker level increases by 32.9 [17.7–62.0] pg/ml. Before and at the end of surgery, BNP values are weakly correlated with CVP and PAWP and do not correlate with other hemodynamic parameters. Before surgery, BNP blood level in the range of 52.1–90.8 pg/ml are predictors of ICU stay > 24 hours (AUC 0.775), inotropic scale > 5 (AUC 0.705) and vasoactive-inotropic scale > 10 c. u. (AUC 0.727). At the end of surgery, BNP > 67.5 pg/ml is associated with an ICU stay > 24 hours (AUC 0.763), and BNP > 90.4 pg/ml is associated with inotropic scale > 5 c. u. (AUC 0.713). The BNP, increased to 144.0–161.0 pg/ml, indicates severe myocardial dysfunction, including hemodynamic support with intra-aortic balloon pumping (AUC 0.854), vasoactive-inotropic scale > 10 c. u. (AUC 0.729) and the risk of early mortality in the ICU (AUC 0.845).

Named entity recognition (NER) plays a vital role in extracting critical medical entities from health care records, facilitating applications such as clinical decision support and data mining. Developing robust NER models for low-resource languages, such as Estonian, remains a challenge due to the scarcity of annotated data and domain-specific pretrained models. Large language models (LLMs) have proven to be promising in understanding text from any language or domain. This study addresses the development of medical NER models for low-resource languages, specifically Estonian. We propose a novel approach by generating synthetic health care data and using LLMs to annotate them. These synthetic data are then used to train a high-performing NER model, which is applied to real-world medical texts, preserving patient data privacy. Our approach to overcoming the shortage of annotated Estonian health care texts involves a three-step pipeline: (1) synthetic health care data are generated using a locally trained GPT-2 model on Estonian medical records, (2) the synthetic data are annotated with LLMs, specifically GPT-3.5-Turbo and GPT-4, and (3) the annotated synthetic data are then used to fine-tune an NER model, which is later tested on real-world medical data. This paper compares the performance of different prompts; assesses the impact of GPT-3.5-Turbo, GPT-4, and a local LLM; and explores the relationship between the amount of annotated synthetic data and model performance. The proposed methodology demonstrates significant potential in extracting named entities from real-world medical texts. Our top-performing setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction. These results indicate a strong performance in recognizing certain entity types while highlighting the complexity of extracting procedures. This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method's applicability to other domains and languages.

Many clinically significant health conditions in older adults are underreported or only recorded in unstructured health records. These records, however, contain valuable information for patient care and prognosis. This study utilized 10.6 million free-text entries from the electronic health records of 102,525 patients aged 50–80 across various care settings in Finland from 2010 to 2022. A deep learning-based natural language processing model was employed to perform named entity recognition (NER) to identify falls, incontinence, loneliness, and mobility limitations from the free-text entries. The performance of the NER models was evaluated by precision, recall and F1 scores. Diagnostic codes for incontinence and falls were collected for comparisons. Cox regression models were used to assess the predictive value of the identified conditions for all-cause mortality. The NER models demonstrated excellent performance with recall, precision and F1 scores greater than 0.80 across the health conditions. Compared to diagnostic codes, NER identified greater numbers of falls (31987 vs 4090) and incontinence (7059 vs 3873) onsets and yielded greater hazard ratios for all-cause mortality: 1.31 vs 1.04 for falls and 1.99 vs 0.65 for incontinence. Deep learning-based NER models present new opportunities to identify vulnerable patients in free text health records.

We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. The biggest challenge of distantly-supervised NER is that the distant supervision may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this paper, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextualized augmentations created by pre-trained language models to improve the generalization ability of the NER model. On three benchmark datasets, our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.

Named Entity Recognition (NER) is a core task of NLP. State-of-art supervised NER models rely heavily on a large amount of high-quality annotated data, which is quite expensive to obtain. Various existing ways have been proposed to reduce the heavy reliance on large training data, but only with limited effect. In this paper, we propose a novel way to make full use of the weakly-annotated texts in encyclopedia pages for exactly unsupervised NER learning, which is expected to provide an opportunity to train the NER model with no manually-labeled data at all. Briefly, we roughly divide the sentences of encyclopedia pages into two parts simply according to the density of inner url links contained in each sentence. While a relatively small number of sentences with dense links are used directly for training the NER model initially, the left sentences with sparse links are then smartly selected for gradually promoting the model in several self-training iterations. Given the limited number of sentences with dense links for training, a data augmentation method is proposed, which could generate a lot more training data with the help of the structured data of encyclopedia to greatly augment the training effect. Besides, in the iterative self-training step, we propose to utilize a graph model to help estimate the labeled quality of these sentences with sparse links, among which those with the highest labeled quality would be put into our training set for updating the model in the next iteration. Our empirical study shows that the NER model trained with our unsupervised learning approach could perform even better than several state-of-art models fully trained on newswires data.

Intent classification (IC) and Named Entity Recognition (NER) are arguably the two main components needed to build a Natural Language Understanding (NLU) engine, which is a main component of conversational agents. The IC and NER components are closely intertwined and the entities are often connected to the underlying intent. Current research has primarily focused to model IC and NER as two separate units, which results in error propagation, and thus, sub-optimal performance. In this paper, we propose a simple yet effective novel framework for NLU where the parameters of the IC and the NER models are jointly trained in a consolidated parameter space. Text semantic representations are obtained from popular pre-trained contextual language models, which are fine-tuned for our task, and these parameters are propagated to other deep neural layers in our framework leading to a faithful unified modelling of the IC and NER parameters. The overall framework results in a faithful parameter sharing when the training is underway, leading to a more coherent learning. Experiments on two public datasets, ATIS and SNIPS, show that our model outperforms other methods by a noticeable margin. On the SNIPS dataset, we obtain a 1.42% improvement in NER in terms of the F1 score, and 1% improvement in intent accuracy score. On ATIS, we achieve 1.54% improvement in intent accuracy score. We also present qualitative results to showcase the effectiveness of our model.

Text mining is increasingly used to manage the accelerating pace of the biomedical literature. Many text mining applications depend on accurate named entity recognition (NER) and normalization (grounding). While high performing machine learning methods trainable for many entity types exist for NER, normalization methods are usually specialized to a single entity type. NER and normalization systems are also typically used in a serial pipeline, causing cascading errors and limiting the ability of the NER system to directly exploit the lexical information provided by the normalization. We propose the first machine learning model for joint NER and normalization during both training and prediction. The model is trainable for arbitrary entity types and consists of a semi-Markov structured linear classifier, with a rich feature approach for NER and supervised semantic indexing for normalization. We also introduce TaggerOne, a Java implementation of our model as a general toolkit for joint NER and normalization. TaggerOne is not specific to any entity type, requiring only annotated training data and a corresponding lexicon, and has been optimized for high throughput. We validated TaggerOne with multiple gold-standard corpora containing both mention- and concept-level annotations. Benchmarking results show that TaggerOne achieves high performance on diseases (NCBI Disease corpus, NER f-score: 0.829, normalization f-score: 0.807) and chemicals (BioCreative 5 CDR corpus, NER f-score: 0.914, normalization f-score 0.895). These results compare favorably to the previous state of the art, notwithstanding the greater flexibility of the model. We conclude that jointly modeling NER and normalization greatly improves performance. The TaggerOne source code and an online demonstration are available at: http://www.ncbi.nlm.nih.gov/bionlp/taggerone zhiyong.lu@nih.gov Supplementary data are available at Bioinformatics online.

Power equipment maintenance work orders are vital in power equipment management because they contain detailed information such as equipment specifications, defect reports, and specific maintenance activities. However, due to limited research into automated information extraction, valuable operational and maintenance data remain underutilized. A key challenge is recognizing unstructured Chinese maintenance texts filled with specialized and abbreviated terms unique to the power sector. Existing named entity recognition (NER) solutions often fail to effectively manage these complexities. To tackle this, this paper proposes a NER model tailored to power equipment maintenance work orders. First, a dataset called power equipment maintenance work orders (PE-MWO) is constructed, which covers seven entity categories. Next, a novel position- and similarity-aware attention module is proposed, where an innovative position embedding method and attention score calculation are designed to improve the model's contextual understanding while keeping computational costs low. Further, with this module as the main body, combined with the BERT-wwm-ext and conditional random field (CRF) modules, an efficient NER model is jointly constructed. Finally, validated on the PE-MWO and five public datasets, our model shows high accuracy in recognizing power sector entities, outperforming comparative models on public datasets.