Event extraction as machine reading comprehension with question-context bridging

Machine reading comprehension (MRC) is a challenging natural language processing (NLP) task. It has a wide application potential in the fields of question answering robots, human‐computer interactions in mobile virtual reality systems, etc. Recently, the emergence of pretrained models (PTMs) has brought this research field into a new era, in which the training objective plays a key role. The masked language model (MLM) is a self‐supervised training objective widely used in various PTMs. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking, entity masking, phrase masking, and span masking. In different MLMs, the length of the masked tokens is different. Similarly, in different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence. Thus, in MRC tasks with different answer lengths, whether the length of MLM is related to performance is a question worth studying. If this hypothesis is true, it can guide us on how to pretrain the MLM with a relatively suitable mask length distribution for MRC tasks. In this paper, we try to uncover how much of MLM’s success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in the MRC dataset. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple‐choice cloze task, and long multiple‐choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation.

The low-resource named entity recognition (NER) task faces the dual challenges of sparse annotation data and under-utilised syntactic structures. In this paper, we propose the Syntax-CLNP model, which solves the above problems through two core innovations: firstly, we design the Syntax-Aware Adapter, which dynamically injects dependent syntactic features by using differentiable syntactic projectors to realise semantic-syntactic collaborative characterisation; secondly, we propose the Contrastive Prototype Alignment (CPA) mechanism, which integrates contrastive learning and prototype network to strengthen the syntactic structure by gradient decoupling and enhance the syntactic structure through gradient decoupling. Prototype Alignment (CPA) mechanism is proposed, which integrates contrastive learning and prototype network, and strengthens intra-class feature compactness and inter-class separation through gradient decoupling to suppress noise interference. The experimental results show that on low resource datasets such as LingNER-40 (40 languages) and WikiANN, the F1 value of Syntax-CLNP is improved by 8.7% on average compared with the baseline model (p<0.01), the number of parameters increases by only 1% (111M), and the increase in inference latency is less than 4% (8.5ms), which achieves a balance between performance and efficiency. In the cross-language migration test, the model achieves 58.9% F1 in low-resource scenarios such as Tamil (<1k samples), verifying the generalisation capability. The study achieves the first end-to-end joint optimisation of syntactic tree and representation learning, providing an efficient solution for low-resource multilingual NER tasks.

For query-by-example spoken term detection (QbE-STD), generation of phone posteriorgrams requires labelled data which would be difficult for languages with low resources. One solution is to build models from rich resource languages and use them in the low resource scenario. However, phone classes are not language universal and alternate representation such as articulatory classes is explored. In this paper, we use articulatory information and their derivatives such as bottle-neck (BN) features (also referred to as articulatory BN features) for QbE-STD. We obtain Gaussian posteriorgrams of articulatory BN features in tandem with the acoustic parameters such as frequency domain linear prediction cepstral coefficients to perform the search. We compare the search performance of articulatory and phone BN features and show that articulatory BN features are a better representation. We also provide experimental results to show that low amounts (30 mins) of training data could be used to derive articulatory BN features.

Enhancing named entity recognition in low-resource scenarios using diffusion variational autoencoder and knowledge graphs

With the rapid growth of legal cases, professionals are under pressure to go through lengthy documents and grasp informative pieces of text in the limited time. Most of the existing techniques focus on simple legal information retrieval task, such as name or address of the prosecutor or the defendant, which can be easily accomplished with the help of handcrafted patterns or sequence labeling methods. Yet complicated texts always challenge such pattern-based methods and sequence labeling approaches. These texts state the same facts or describe the same events, but they do not share common or similar patterns. In this paper, we design a unified framework to extract legal information in various formats, including directly extracted information (a piece of span) and information that needs to be deduced. The framework follows the methodology to answer questions in machine reading comprehension (MRC) tasks. We treat the extraction fact labels as the counterpart of questions in MRC task and propose several strategies to represent them. We construct several datasets regarding different cases for training and testing. Our best strategy achieves up to 4% enhancement in F1 score on each dataset compared to the MRC baseline.

Machine reading comprehension(MRC), which employs computers to answer questions from given passages, is a popular research field. In natural language, a natural hierarchical representation can be seen: characters, words, phrases, sentences, paragraphs, and documents. Current studies have demonstrated that hierarchical information can help machines understand natural language. However, prior works focused on the overall performance of MRC tasks without considering hierarchical information. In addition, the noise problem still has not been adequately addressed, even though many researchers have adopted the technique of passage reranking. Thus, in this paper, focusing on noise information processing and the extraction of hierarchical information, we propose a model (PH-Model) with a passage reranking framework (P) and hierarchical neural network (H) for a Chinese multi-passage MRC task. PH-Model produces more precise answers by reducing noise information and extracting hierarchical information. Experimental results on the DuReader 2.0 dataset (a large scale real-world Chinese MRC dataset) show that PH-Model outperforms the ROUGE-L and BLEU-4 baseline by 18.24% and 24.17%, respectively.

Medical named entity recognition via lattice-enhanced transfer learning with random attention

Cross lingual transfer learning for sentiment analysis of Italian TripAdvisor reviews

Explainable few-shot learning with visual explanations on a low resource pneumonia dataset

Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.

Chinese electronic medical records named entity recognition (NER) is a core task in medical knowledge mining, which is usually viewed as a sequence labeling problem. Recent works introduce the machine reading comprehension (MRC) framework into this task and extract named entities in a question-answering manner, resulting in state-of-the-art performance. However, they extract entities of different types independently, ignoring the fact that entities presented in the context might highly correlate with each other. To address this issue, we extend the MRC-based model and propose Fusion Label Relations with MRC (FLR-MRC). The method implicitly models the relations between labels through graph attention networks and fuse label information with text for named entity recognition. Experimental results on the benchmark datasets CMeEE and CCKS2017-CNER demonstrate FLR-MRC outperform existing clinical NER methods, with F1-score reaching 0.6652 and 0.9101, respectively.KeywordsMedical named entity recognitionLabel relationsMachine reading comprehension

Machine Reading Comprehension(MRC), including a series of tasks that test the ability of models to understand natural language, has received quite a few attention in Natural Language Processing(NLP). Most existing works deal with MRC tasks by exploiting the expression capability of neural networks. Some of them have achieved impressive performance. Despite the rapid iteration of the models used, few work have focused on output layer and prediction method of answer span - also known as span extraction. In this paper, we focus on span extraction in the Question Answering(QA) task. A cross-sectional comparison of widely used span extraction methods is presented, with their strengths and weaknesses noted in detail. Furthermore, inspired by Faster R-CNN, we propose a brand new span extraction method. Experiment results show that our proposed method outperforms existing span extraction methods on both English and Chinese MRC tasks.

Active learning plays an important role in low-resource scenarios, i.e., when only a small amount of annotated instances is available. However, one does not know what is the best active learning strategy before actually testing a handful of strategies on a labeled set, which might not be viable in a real world low-resource scenario. Instead, it would be desirable to dynamically obtain the results from the best strategy on a given scenario, while using as little annotated resources as possible.In this paper, we present a novel application of prediction with expert advice to combine different query strategies as experts, giving a greater weight to those which select the most useful instances. We evaluated our approach in two Natural Language Processing (NLP) tasks: Part-of-Speech tagging (for English) and Named Entity Recognition (for Portuguese). Results show that our solution keeps up with the results of the best strategy in each scenario, nearly reaching fully supervised performance with only half of the annotated data.

Deep neural networks have achieved state-of-the-art performances on named entity recognition (NER) with sufficient training data, while they perform poorly in low-resource scenarios due to data scarcity. To solve this problem, we propose a novel data augmentation method based on pre-trained language model (PLM) and curriculum learning strategy. Concretely, we use the PLM to generate diverse training instances through predicting different masked words and design a task-specific curriculum learning strategy to alleviate the influence of noises. We evaluate the effectiveness of our approach on three datasets: CoNLL-2003, OntoNotes5.0, and MaScip, of which the first two are simulated low-resource scenarios, and the last one is a real low-resource dataset in material science domain. Experimental results show that our method consistently outperform the baseline model. Specifically, our method achieves an absolute improvement of 3.46% \(F_1\) score on the 1% CoNLL-2003, 2.58% on the 1% OntoNotes5.0, and 0.99% on the full of MaScip.

Speaker identification (SI) in texts aims to identify the speaker(s) for each utterance in texts. Previous studies divide SI into several sub-tasks (e.g., quote extraction, named entity recognition, gender identification, and coreference resolution). However, we are still far from solving these sub-tasks, making SI systems that rely on them seriously suffer from error propagation. End-to-end SI systems, on the other hand, are not limited by individual modules, but suffer from insufficient training data from the existing small-scale datasets. To make large end-to-end models possible, we design a new annotation guideline that regards SI as span extraction from the local context, and we annotate by far the largest SI dataset for Chinese named CSI based on eighteen novels. Viewing SI as a span selection task also introduces the possibility of applying existing strong extractive machine reading comprehension (MRC) baselines. Surprisingly, simply using such a baseline without human-annotated character names and carefully designed rules, we can already achieve performance comparable or better than those of previous state-of-the-art SI methods on all public SI datasets for Chinese. Furthermore, we show that our dataset can serve as additional training data for existing benchmarks, which leads to further gains (up to 6.5% in accuracy). Finally, using CSI as a clean source, we design an effective self-training paradigm to continuously leverage hundreds of unlabeled novels.