Grammar-Driven Text Segmentationfor Context Understanding of Myanmar Language

Current work in named entity recognition (NER) uses either cross entropy (CE) or conditional random fields (CRF) as the objective/loss functions to optimize the underlying NER model. Both of these traditional objective functions for the NER problem generally produce adequate performance when the data distribution is balanced and there are sufficient annotated training examples. But since NER is inherently an imbalanced tagging problem, the model performance under the low-resource settings could suffer using these standard objective functions. Based on recent advances in area under the ROC curve (AUC) maximization, we propose to optimize the NER model by maximizing the AUC score. We give evidence that by simply combining two binary-classifiers that maximize the AUC score, significant performance improvement over traditional loss functions is achieved under low-resource NER settings. We also conduct extensive experiments to demonstrate the advantages of our method under the low-resource and highly-imbalanced data distribution settings. To the best of our knowledge, this is the first work that brings AUC maximization to the NER setting. Furthermore, we show that our method is agnostic to different types of NER embeddings, models and domains. The code of this work is available at https://github.com/dngu0061/NER-AUC-2T.

Network representation aims to learn low-dimensional vector representations of network nodes while preserving the inherent properties of the network. For all its popularity, majority of the existing methods focus on exploitation of diverse information, including network topology and semantic information on nodes of network, and ignore their implicit semantics. For example, we all know the saying that birds of a feather flock together. More concretely, semantic information of one node can be influenced by its neighbors' semantic information. Furthermore, even two nodes are not directly connected, they may have similar implicit semantic information (i.e., high-order semantic proximity). Thus, they should be close in the represented vector space. To this end, we propose a Deep Semantic Network Representation approach (DSNR) in the self-translation framework from sequence to sequence. To excavate the implicit semantic information of nodes and capture the high-order semantic proximity, three key components make our approach effective, i.e., aggregation of nodes neighbors' semantic information and enhancement to the semantic feature representations of nodes by a deep autoencoder, integration of nodes semantic information in node identity sequence to generate node semantic sequence, and translation from node semantic sequence to node identity sequence to capture the high-order semantic proximity in an attention-enhanced seq2seq framework. Extensive experiments based on three real-world datasets have verified the effectiveness of our proposed approach11Code is available at https://github.com/DASE4/DSNR.

Deep language models, like ELMo, BERT and GPT, have achieved impressive results on several natural language tasks. These models are pretrained on large corpora of unlabeled general domain text and later supervisedly trained on downstream tasks. An optional step consists of finetuning the language model on a large intradomain corpus of unlabeled text, before training it on the final task. This aspect is not well explored in the current literature. In this work, we investigate the impact of this step on named entity recognition (NER) for Portuguese legal documents. We explore different scenarios considering two deep language architectures (ELMo and BERT), four unlabeled corpora and three legal NER tasks for the Portuguese language. Experimental findings show a significant improvement on performance due to language model finetuning on intradomain text. We also evaluate the finetuned models on two general-domain NER tasks, in order to understand whether the aforementioned improvements were really due to domain similarity or simply due to more training data. The achieved results also indicate that finetuning on a legal domain corpus hurts performance on the general-domain NER tasks. Additionally, our BERT model, finetuned on a legal corpus, significantly improves on the state-of-the-art performance on the LeNER-Br corpus, a Portuguese language NER corpus for the legal domain.

In domain-specific named entity recognition (NER), the out-of-vocabulary (OOV) problem arises due to linguistic features and rare vocabulary.OOV problem is particularly challenging in agglutinative languages such as Korean.The irregular decomposition of morphemes makes it difficult to represent all of them in language model dictionaries, resulting in poor NER performance.Subword tokenization which segments a word into atomic tokens that are no longer divided can be one of the possible solutions.In the construction industry, existing NER methods do not effective on housing defect complaints which contain many rare words, including jargon, slang, and typos.To address this challenge, we propose subword tokenization algorithms that can mitigate OOV problems based on considering linguistic features and pre-trained language models (PLMs).The primary objective of this study is to identify the optimal NER performance by comparing different subword tokenization methods depending on the language models used.For domain-specific NER, we defined and used 23 defectspecific named entity tags for dataset labelling.We then experimented with a total of three state-of-theart language models: one SentencePiece-based and two WordPiece-based subword tokenization models.The results demonstrate that the SentencePiece-based Korean Bidirectional Encoder Representations from Transformers (KoBERT) outperformed the two WordPiece-based language models (multilingual-BERT and Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA)) with an F1 score of 84.7%.The proposed method is expected to improve not only NER but also other downstream tasks that involve using Korean documents in the construction industry.

Methods for text segmentation from scene images

Semantic annotation for Web services represents an important approach that transforms Web services into Semantic Web services. This paper analyzes the distribution of implicit semantic information to enhance the understanding about Web services. After quantifying implicit semantic information in Web services, the PCA methodology is utilized for obtaining principal components of implicit semantic information. The experiment conducted on 11836 WSDL documentations shows that almost 97 percent of implicit semantic information is distributed in inputs and outputs. Thus, the evaluation of how much implicit semantic information can be described by semantic annotation is offered.

A complete inpatient electronic medical record contains a lot of information. In recent years, numerous researchers have carried out research on word segmentation of medical texts. Since the medical record text is written by free text, structuring the medical record text is an important part of the medical record intelligent analysis when these medical named entities are recognized. At present, named entity recognition methods are mainly divided into lexicographical and rule-based methods and machine learning-based methods. The machine learn-based approach takes the named entity recognition task as the annotation problem of sequence data, mainly considering the context information. The features commonly used in feature construction are contextual feature and dictionary features. BERT+LSTM+CRF were used to train the named entity recognition model. Open source CRF++ was adopted as the tool we relied on. We trained the LSTM+F model using the results of the original word segmentation and the information in the context as features. We carried out a 5-fold cross validation. The results showed that the overall F-1 score (MICRO-F) of named entity recognition reached 0.92, which confirmed that the model could accurately complete the task of medical named entity recognition.KeywordsWord segmentationNamed entity recognitionNatural language processing

A large number of web pages contain information about entities in lists where the lists are represented in textual form. Textual lists contain implicit records of entities. However, the field values of such records cannot easily be separated or extracted by automatic processes. This, therefore, remains a challenging research problem in the literature. Previous studies in the literature relied mainly on probabilistic graph-based models to capture the attributes and the likely structures of implicit records in a list. However, one of the important limitations of existing methods is that the structures of the records in input lists were implicitly encoded via training data which was manually created. This thesis aims to investigate novel techniques to acquire automatically information about entities from implicit records embedded in textual lists on the web. This thesis introduces a self-supervised learning framework which exploits both existing data in a knowledge base and the structural similarity between sequences in lists to build an extraction model automatically. In the proposed framework, initial labels for candidate field values are created and assigned to generate label sequences. Then, the structure of implicit records is captured via a graphical model to assign unmatched labels and rectify mismatched labels. As a result of which, the process of entity extraction from lists can be completely unsupervised and automated without user intervention. In order to attain that outcome, we address three substantive research problems that need to be solved in this thesis. Firstly, the text segments in input lists need to be assigned labels precisely so that their statistical information is then used to build an extraction model. However, previous studies have not considered completely both the format and content of field values when performing the text segmentation and assigning labels. By viewing the problem of assigning labels for text segments as the problem of membership checking in set theory, we identify and propose a dyadic representation of semantic relations between a text segment and an attribute by using its extensional and intensional representations. We incorporate those representations to define a novel format-enhanced labelling technique to assign labels for text segments. Secondly, the labels of identical concepts with differing sequences in an input list are often located in similar positions but the positions of the labels may vary somewhat in different sequences. However, until this point, there has been no information extraction system designed to capture the distribution of labels in differing positions in order to enhance extraction results. To capture the positional information of labels, we are proposing a proximity-based positional model, which is combined with a sequential model to improve the quality of the label-refinement phase in our framework. Thirdly, in order to reduce dependence on the overlap between knowledge bases and input lists, we exploit structural similarity between text segments and sequences in the input lists, and devise a structure-based similarity and data shifting-alignment technique to align text segments into groups before their labels are revised by a graphical model. By the proposed technique, we can reduce the dependency on the overlap between knowledge bases and input lists whilst maintaining high performance of extraction model. Initially experimental results demonstrate that our proposed techniques perform well when compared to the state-of-the-art method. We hope that the results presented in this thesis contribute to efforts on the extraction of information about entities in textual lists. Additionally, they contribute towards forthcoming research on the synthesis of information from different lists, and the provision of reasoning capacity by which to detect new relationships between entities drawn from raw lists on the web.

Task-driven infrared and visible image fusion via detail and semantic dual injection.

Named entity recognition (NER) from visual documents, such as invoices, receipts or business cards, is a critical task for visual document understanding. Most classical approaches use a sequence-based model (typically BiLSTM-CRF framework) without considering document structure. Recent work on graph-based model using graph convolutional networks to encode visual and textual features have achieved promising performance on the task. However, few attempts take geometry information of text segments (text in bounding box) in visual documents into account. Meanwhile, existing methods do not consider that related text segments which need to be merged to form a complete entity in many real-world situations. In this paper, we present GraphNEMR, a graph-based model that uses graph convolutional networks to jointly merge text segments and recognize named entities. By incorporating geometry information from visual documents into our model, richer 2D context information is generated to improve document representations. To merge text segments, we introduce a novel mechanism that captures both geometry information as well as semantic information based on pre-trained language model. Experimental results show that the proposed GraphNEMR model outperforms both sequence-based and graph-based SOTA methods significantly.

Named entity recognition (NER) from visual documents, such as invoices, receipts or business cards, is a critical task for visual document understanding. Most classical approaches use a sequence-based model (typically BiLSTM-CRF framework) without considering document structure. Recent work on graph-based model using graph convolutional networks to encode visual and textual features have achieved promising performance on the task. However, few attempts take geometry information of text segments (text in bounding box) in visual documents into account. Meanwhile, existing methods do not consider that related text segments which need to be merged to form a complete entity in many real-world situations. In this paper, we present GraphNEMR, a graph-based model that uses graph convolutional networks to jointly merge text segments and recognize named entities. By incorporating geometry information from visual documents into our model, richer 2D context information is generated to improve document representations. To merge text segments, we introduce a novel mechanism that captures both geometry information as well as semantic information based on pre-trained language model. Experimental results show that the proposed GraphNEMR model outperforms both sequence-based and graph-based SOTA methods significantly.

Named Entity Recognition (NER) on Clinical Electronic Medical Records (CEMR) is a fundamental step in extracting disease knowledge by identifying specific entity terms such as diseases, symptoms, etc. However, the state-of-the-art NER methods based on Long Short-Term Memory (LSTM) fail to fully exploit GPU parallelism under the massive medical records. Although a novel NER method based on Iterated Dilated CNNs (ID-CNNs) can accelerate network computing, it tends to ignore the word-order feature and semantic information of the current word. In order to enhance the performance of ID-CNNs-based models on NER tasks, an attention-based ID-CNNs-CRF model which combines word-order feature and local context is proposed. Firstly, Position Embedding is utilized to fuse word-order information. Secondly, ID-CNNs architecture is used to rapidly extract global semantic information. Simultaneously, the attention mechanism is employed to pay attention to the local context. Finally, we apply the CRF to obtain the optimal tag sequence. Experiments conducted on two CEMR datasets show that our model outperforms traditional ones. The F1-scores of 94.55% and 91.17% are obtained respectively on these two datasets and both are better than LSTMs-based models.

Contrastive fine-tuning for low-resource graph-level transfer learning

A massive number of biological entities, such as genes and mutations, are mentioned in the biomedical literature. The capturing of the semantic relatedness of biological entities is vital to many biological applications, such as protein-protein interaction prediction and literature-based discovery. Concept embeddings—which involve the learning of vector representations of concepts using machine learning models—have been employed to capture the semantics of concepts. To develop concept embeddings, named-entity recognition (NER) tools are first used to identify and normalize concepts from the literature, and then different machine learning models are used to train the embeddings. Despite multiple attempts, existing biomedical concept embeddings generally suffer from suboptimal NER tools, small-scale evaluation, and limited availability. In response, we employed high-performance machine learning-based NER tools for concept recognition and trained our concept embeddings, BioConceptVec, via four different machine learning models on ~30 million PubMed abstracts. BioConceptVec covers over 400,000 biomedical concepts mentioned in the literature and is of the largest among the publicly available biomedical concept embeddings to date. To evaluate the validity and utility of BioConceptVec, we respectively performed two intrinsic evaluations (identifying related concepts based on drug-gene and gene-gene interactions) and two extrinsic evaluations (protein-protein interaction prediction and drug-drug interaction extraction), collectively using over 25 million instances from nine independent datasets (17 million instances from six intrinsic evaluation tasks and 8 million instances from three extrinsic evaluation tasks), which is, by far, the most comprehensive to our best knowledge. The intrinsic evaluation results demonstrate that BioConceptVec consistently has, by a large margin, better performance than existing concept embeddings in identifying similar and related concepts. More importantly, the extrinsic evaluation results demonstrate that using BioConceptVec with advanced deep learning models can significantly improve performance in downstream bioinformatics studies and biomedical text-mining applications. Our BioConceptVec embeddings and benchmarking datasets are publicly available at https://github.com/ncbi-nlp/BioConceptVec.

Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a target Indian language. We employ a variant of the Teacher-Student model and optimize it jointly on the pseudo labels of the Teacher model and predictions on the generated weakly labeled data. We also present manually annotated test sets for three Indian languages: Hindi, Bengali, and Gujarati. We evaluate the performance of the proposed framework on the test sets of the three Indian languages. Empirical results show a minimum 10% performance improvement compared to the zero-shot transfer learning model on all languages. This indicates that weakly labeled data generated using the proposed annotation projection method in target Indian languages can complement well-annotated source language data to enhance performance. Our code is publicly available at https://github.com/aksh555/CL-NERIL.