Diversified Semantic Attention Model for Fine-Grained Entity Typing

Fine-grained entity typing aims to identify the semantic type of an entity in a particular plain text. It is an important task which can be helpful for a lot of natural language processing (NLP) applications. Most existing methods typically extract features separately from the entity mention and context words for type classification. These methods inevitably fail to model complex correlations between entity mentions and context words. They also neglect rich background information about these entities in knowledge bases (KBs). To address these issues, we take information from KBs into consideration to bridge entity mentions and their context together, and thereby propose Knowledge-Attention Neural Fine-Grained Entity Typing. Experimental results and case studies on real-world datasets demonstrate that our model significantly outperforms other state-of-the-art methods, revealing the effectiveness of incorporating KB information for entity typing. Code and data for this paper can be found at https://github.com/thunlp/KNET.

Entities play an important role in many natural language applications. Based on the Automatic content Extraction (ACE) conference, we study the extraction technologies of entity mentions in Chinese text. Compared to named entities, entity mentions have rich categories and complex structures, which bring great difficulty to the extraction task. To solve the above problems, we propose an unsupervised method to detect entity mentions and identify their categories in Chinese text, namely Un-MenEx. With the abundant data of Baidu Baike and Baidu search, Un-MenEx exploits a similarity calculation method to extract entity mentions in text, which solves the problem of identifying rare entity names difficultly and optimizes the mentions segmented wrongly. Moreover, Un-MenEx can meet the demand of processing massive data by reason of no manual annotation data. We conduct the experiments with the news text, and the experimental results show that this method has practical application value, and ensure the accuracy requirement.

Fine-grained Entity typing (FGET) is the task of assigning a fine-grained type from a hierarchy to entity mentions in the text. As the taxonomy of types evolves continuously, it is desirable for an entity typing system to be able to recognize novel types without additional training. This work proposes a zero-shot entity typing approach that utilizes the type description available from Wikipedia to build a distributed semantic representation of the types. During training, our system learns to align the entity mentions and their corresponding type representations on the known types. At test time, any new type can be incorporated into the system given its Wikipedia descriptions. We evaluate our approach on FIGER, a public benchmark entity tying dataset. Because the existing test set of FIGER covers only a small portion of the fine-grained types, we create a new test set by manually annotating a portion of the noisy training data. Our experiments demonstrate the effectiveness of the proposed method in recognizing novel types that are not present in the training data.

Extraction from raw text to a knowledge base of entities and fine-grained types is often cast as prediction into a flat set of entity and type labels, neglecting the rich hierarchies over types and entities contained in curated ontologies. Previous attempts to incorporate hierarchical structure have yielded little benefit and are restricted to shallow ontologies. This paper presents new methods using real and complex bilinear mappings for integrating hierarchical information, yielding substantial improvement over flat predictions in entity linking and fine-grained entity typing, and achieving new state-of-the-art results for end-to-end models on the benchmark FIGER dataset. We also present two new human-annotated datasets containing wide and deep hierarchies which we will release to the community to encourage further research in this direction: MedMentions, a collection of PubMed abstracts in which 246k mentions have been mapped to the massive UMLS ontology; and TypeNet, which aligns Freebase types with the WordNet hierarchy to obtain nearly 2k entity types. In experiments on all three datasets we show substantial gains from hierarchy-aware training.

Most work on knowledge base completion focuses on relations between entities, while entity types are also important knowledge. This paper addresses the problem of fine-grained entity typing for knowledge base completion. Context information plays a vital role in fine-grained entity typing, hence there is an urgent need to find ideal context representations. This paper presents a new approach CNNJM (convolutional neural network joint model) to learn the embeddings of the entities and their contextual information using convolutional neural network and correctly categorize the entities into their fine-grained type classes. We show that CNNJM outperforms state-of-art methods on a fine-grained entity typing benchmark.

Recently, there is an effort to extend fine-grained entity typing by using a richer and ultra-fine set of types, and labeling noun phrases including pronouns and nominal nouns instead of just named entity mentions. A key challenge for this ultra-fine entity typing task is that human annotated data are extremely scarce, and the annotation ability of existing distant or weak supervision approaches is very limited. To remedy this problem, in this paper, we propose to obtain training data for ultra-fine entity typing by using a BERT Masked Language Model (MLM). Given a mention in a sentence, our approach constructs an input for the BERT MLM so that it predicts context dependent hypernyms of the mention, which can be used as type labels. Experimental results demonstrate that, with the help of these automatically generated labels, the performance of an ultra-fine entity typing model can be improved substantially. We also show that our approach can be applied to improve traditional fine-grained entity typing after performing simple type mapping.

The task of Fine-grained Entity Type Classification (FETC) consists of assigning types from a hierarchy to entity mentions in text. Existing methods rely on distant supervision and are thus susceptible to noisy labels that can be out-of-context or overly-specific for the training sentence. Previous methods that attempt to address these issues do so with heuristics or with the help of hand-crafted features. Instead, we propose an end-to-end solution with a neural network model that uses a variant of cross-entropy loss function to handle out-of-context labels, and hierarchical loss normalization to cope with overly-specific ones. Also, previous work solve FETC a multi-label classification followed by ad-hoc post-processing. In contrast, our solution is more elegant: we use public word embeddings to train a single-label that jointly learns representations for entity mentions and their context. We show experimentally that our approach is robust against noise and consistently outperforms the state-of-the-art on established benchmarks for the task.

Relation Extraction (RE) is one of the fundamental tasks in Information Extraction. The goal of this task is to find the semantic relations between entity mentions in text. It has been shown in many previous work that the structure of the sentences (i.e., dependency trees) can provide important information/features for the RE models. However, the common limitation of the previous work on RE is the reliance on some external parsers to obtain the syntactic trees for the sentence structures. On the one hand, it is not guaranteed that the independent external parsers can offer the optimal sentence structures for RE and the customized structures for RE might help to further improve the performance. On the other hand, the quality of the external parsers might suffer when applied to different domains, thus also affecting the performance of the RE models on such domains. In order to overcome this issue, we introduce a novel method for RE that simultaneously induces the structures and predicts the relations for the input sentences, thus avoiding the external parsers and potentially leading to better sentence structures for RE. Our general strategy to learn the RE-specific structures is to apply two different methods to infer the structures for the input sentences (i.e., two views). We then introduce several mechanisms to encourage the structure and semantic consistencies between these two views so the effective structure and semantic representations for RE can emerge. We perform extensive experiments on the ACE 2005 and SemEval 2010 datasets to demonstrate the advantages of the proposed method, leading to the state-of-the-art performance on such datasets.

Fine-grained entity typing (FET) aims to assign proper semantic types to entity mentions according to their context, which is a fundamental task in various entity-leveraging applications. Current FET systems usually establish on large-scale weaklysupervised/distantly annotation data, which may contain abundant noise and thus severely hinder the performance of the FET task. Although previous studies have made great success in automatically identifying the noisy labels in FET, they usually rely on some auxiliary resources which may be unavailable in real-world applications (e.g., pre-defined hierarchical type structures, humanannotated subsets). In this paper, we propose a novel approach to automatically correct noisy labels for FET without external resources. Specifically, it first identifies the potentially noisy labels by estimating the posterior probability of a label being positive or negative according to the logits output by the model, and then relabel candidate noisy labels by training a robust model over the remaining clean labels. Experiments on two popular benchmarks prove the effectiveness of our method. Our source code can be obtained from https://github.com/CCIIPLab/DenoiseFET.

We present LOME, a system for performing multilingual information extraction. Given a text document as input, our core system identifies spans of textual entity and event mentions with a FrameNet (Baker et al., 1998) parser. It subsequently performs coreference resolution, fine-grained entity typing, and temporal relation prediction between events. By doing so, the system constructs an event and entity focused knowledge graph. We can further apply third-party modules for other types of annotation, like relation extraction. Our (multilingual) first-party modules either outperform or are competitive with the (monolingual) state-of-the-art. We achieve this through the use of multilingual encoders like XLM-R (Conneau et al., 2020) and leveraging multilingual training data. LOME is available as a Docker container on Docker Hub. In addition, a lightweight version of the system is accessible as a web demo.

In recent years, the amount of entities in large knowledge bases has been increasing rapidly. Such entities can help to bridge unstructured text with structured knowledge and thus be beneficial for many entity-centric applications. The key issue is to link entity mentions in text with entities in knowledge bases, where the main challenge lies in mention ambiguity. Many methods have been proposed to tackle this problem. However, most of the methods assume certain characteristics of the input mentions and documents, e.g., only named entities are considered. In this paper, we propose a context-aware approach to collective entity disambiguation of the input mentions in text with different characteristics in a consistent manner. We extensively evaluate the performance of our approach over 9 datasets and compare it with 14 state-of-the-art methods. Experimental results show that our approach outperforms the existing methods in most cases.

Abstractive single document summarization is considered as a challenging problem in the field of artificial intelligence and natural language processing. Meanwhile and specifically in the last two years, several deep learning summarization approaches were proposed that once again attracted the attention of researchers to this field.It is a well-known issue that deep learning approaches do not work well with small amounts of data. With some exceptions, this is, unfortunately, the case for most of the datasets available for the summarization task. Besides this problem, it should be considered that phonetic, morphological, semantic and syntactic features of the language are constantly changing over the time and unfortunately most of the summarization corpora are constructed from old resources. Another problem is the language of the corpora. Not only in the summarization field, but also in other fields of natural language processing, most of the corpora are only available in English. In addition to the above problems, license terms, and fees of the corpora are obstacles that prevent many academics and specifically non-academics from accessing these data.This work describes an open source framework to create an extendable multilingual corpus for abstractive single document summarization that addresses the above-mentioned problems. We describe a tool consisted of a scalable crawler and a centralized key-value store database to construct a corpus of an arbitrary size using a news aggregator service.

As one of the prominent research directions in the field of natural language processing (NLP), short-text similarity has been widely used in search recommendation and question-and-answer systems. Most of the existing short textual similarity models focus on considering semantic similarity while overlooking the importance of syntactic similarity. In this paper, we first propose an enhanced knowledge language representation model based on graph convolutional networks (KEBERT-GCN), which effectively uses fine-grained word relations in the knowledge base to assess semantic similarity and model the relationship between knowledge structure and text structure. To fully leverage the syntactic information of sentences, we also propose a computational model of constituency parse trees based on tree kernels (CPT-TK), which combines syntactic information, semantic features, and attentional weighting mechanisms to evaluate syntactic similarity. Finally, we propose a comprehensive model that integrates both semantic and syntactic information to comprehensively evaluate short-text similarity. The experimental results demonstrate that our proposed short-text similarity model outperforms the models proposed in recent years, achieving a Pearson correlation coefficient of 0.8805 on the STS-B dataset.

We study the problem of few-shot Fine-grained Entity Typing (FET), where only a few annotated entity mentions with contexts are given for each entity type. Recently, prompt-based tuning has demonstrated superior performance to standard fine-tuning in few-shot scenarios by formulating the entity type classification task as a ''fill-in-the-blank'' problem. This allows effective utilization of the strong language modeling capability of Pre-trained Language Models (PLMs). Despite the success of current prompt-based tuning approaches, two major challenges remain: (1) the verbalizer in prompts is either manually designed or constructed from external knowledge bases, without considering the target corpus and label hierarchy information, and (2) current approaches mainly utilize the representation power of PLMs, but have not explored their generation power acquired through extensive general-domain pre-training. In this work, we propose a novel framework for few-shot FET consisting of two modules: (1) an entity type label interpretation module automatically learns to relate type labels to the vocabulary by jointly leveraging few-shot instances and the label hierarchy, and (2) a type-based contextualized instance generator produces new instances based on given instances to enlarge the training set for better generalization. On three benchmark datasets, our model outperforms existing methods by significant margins. Code can be found at https://github.com/teapot123/Fine-Grained-Entity-Typing.

In recent years, due to the vigorous development of social network media, a large amount of social data can be obtained through social media. This trend has made it easier for researchers in natural language processing to obtain textual research materials, and has further stimulated the development of natural language processing. Emotion recognition is an important task in the field of natural language processing, and emotion recognition helps improve interaction in social. In this study, the emotion recognition of the content of twitter users’ posts and comments is based on the sentence level. We use large datasets to collect users’ posts and comments on Twitter for comparative analysis, and we attempt to understand the emotions expressed by users. Most of the recent studies focus on the analysis of vocabulary and syntactic features in sentences to learn the emotion of the sentence. However, the property of multi-feature may be conducive for promoting the performance of the model and may also be a disturbance of the training of the model. Our study is based on the latest of attention mechanism and XLNet to improve the recognition of emotions. We propose a novel emotion recognition model called Dynamic Weighted Attention with Multichannel Convolutional Neural Network (DACNN), which combines the multi-channel convolutional neural network and the attention mechanism of automatic weight adjustment to effectively improve the outcome of emotion recognition. In addition, we use the latest word embedding technology XLNet to obtain high-quality feature vectors of sentences. In the experiments, we compare the DACNN with the baselines on various datasets, and the results show that DACNN is better than the state-of-the-art methods in accuracy, precision, recall, and fl-score. The experimental results also prove that our attention mechanism and dynamic weighting can well promote the accuracy of emotion recognition.