GF-Net: Improving machine reading comprehension with feature gates

BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance.ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.

Great advances have been made in closed-world Knowledge Graph Completion (KGC). But it still remains a challenge for open-world KGC. A recently proposed open-world KGC model called OWE found a method to map the text space embedding obtained from the entity name and description to a pre-trained graph embedding space, by which OWE can extend the embedding-based KGC models to the open world. However, OWE uses average aggregation to obtain the text representation, no matter the entity description is long or short. It uses much unnecessary textual information and may become unstable. In this paper, we propose an extension to OWE, which is named OWE-MRC, to extract short expressions for entities from long descriptions by using a Machine Reading Comprehension (MRC) model. After obtaining short descriptions for entities, OWE-MRC uses the extension method of OWE to extend the embedding-based KGC models to the open world. We have applied OWE-MRC to extend common KGC models, such as ComplEx and Graph Neural Networks (GNNs) based models, to perform open-world link prediction. Our experiments on two datasets FB20k and DBPedia50k indicate that (1) the MRC model can effectively extract meaningful short descriptions; (2) our OWE-MRC uses much less textual information than OWE, but achieves competitive performance on open-world link prediction. In addition, we have used OWE to extend the GNN-based model to the open world. And our extended GNN model has achieved significant improvements on open-world link prediction comparing to the state-of-the-art open-world KGC models.

Machine Reading Comprehension (MRC) is a challenging task in Natural Language Processing (NLP) field. It is a fast evolving field in the recent years because of the development of large-scale datasets and the deep learning technology. Even though, numerous MRC models have been developed for Question Answer (QA) system, the existing MRC models are not on par with reading comprehension. In order to assess and improve the performance of the projected datasets, an advanced level of evaluation metrics have been introduced. Therefore, this work has conducted a survey over the existing MRC models and evaluation metrics utilized in them to address the lack of MRC with human reading.

In this paper, we introduce a simple system Baidu submitted for MRQA (Machine Reading for Question Answering) 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. Our system is built on a framework of pretraining and fine-tuning, namely D-NET. The techniques of pre-trained language models and multi-task learning are explored to improve the generalization of MRC models and we conduct experiments to examine the effectiveness of these strategies. Our system is ranked at top 1 of all the participants in terms of averaged F1 score. Our codes and models will be released at PaddleNLP.

Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. Here, we demonstrate a simple yet effective method to attack MRC models and reveal the statistical biases in these models. We apply the method to the RACE dataset, for which the answer to each MRC question is selected from 4 options. It is found that several pre-trained language models, including BERT, ALBERT, and RoBERTa, show consistent preference to some options, even when these options are irrelevant to the question. When interfered by these irrelevant options, the performance of MRC models can be reduced from human-level performance to the chance-level performance. Human readers, however, are not clearly affected by these irrelevant options. Finally, we propose an augmented training method that can greatly reduce models' statistical biases.

Despite the remarkable progress on Machine Reading Comprehension (MRC) with the help of open-source datasets, recent studies indicate that most of the current MRC systems unfortunately suffer from weak robustness against adversarial samples. To address this issue, we attempt to take sentence syntax as the leverage in the answer predicting process which previously only takes account of phrase-level semantics. Furthermore, to better utilize the sentence syntax and improve the robustness, we propose a Syntactic Leveraging Network, which is designed to deal with adversarial samples by exploiting the syntactic elements of a question. The experiment results indicate that our method is promising for improving the generalization and robustness of MRC models against the influence of adversarial samples, with performance well-maintained.

In recent years, pre-trained language models, represented by the bidirectional encoder representations from transformers (BERT) model, have achieved remarkable success in machine reading comprehension (MRC). However, limited by the structure of BERT-based MRC models (for example, restrictions on word count), such models cannot effectively integrate significant features, such as syntax relations, semantic connections, and long-distance semantics between sentences, leading to the inability of the available models to better understand the intrinsic connections between text and questions to be answered based on it. In this paper, a multi-granularity syntax guidance (MgSG) module that consists of a “graph with dependence” module and a “graph with entity” module is proposed. MgSG selects both sentence and word granularities to guide the text model to decipher the text. In particular, syntactic constraints are used to guide the text model while exploiting the global nature of graph neural networks to enhance the model’s ability to construct long-range semantics. Simultaneously, named entities play an important role in text and answers and focusing on entities can improve the model’s understanding of the text’s major idea. Ultimately, fusing multiple embedding representations to form a representation yields the semantics of the context and the questions. Experiments demonstrate that the performance of the proposed method on the Stanford Question Answering Dataset is better when compared with the traditional BERT baseline model. The experimental results illustrate that our proposed “MgSG” module effectively utilizes the graph structure to learn the internal features of sentences, solve the problem of long-distance semantics, while effectively improving the performance of PrLM in machine reading comprehension.

Over 97 million inhabitants speak Vietnamese as the native language in the world. However, there are few research studies on machine reading comprehension (MRC) in Vietnamese, the task of understanding a document or text, and answering questions related to it. Due to the lack of benchmark datasets for Vietnamese, we present the Vietnamese Question Answering Dataset (UIT-ViQuAD), a new dataset for the low-resource language as Vietnamese to evaluate MRC models. This dataset comprises over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Vietnamese articles from Wikipedia. In particular, we propose a new process of dataset creation for Vietnamese MRC. Our in-depth analyses illustrate that our dataset requires abilities beyond simple reasoning like word matching and demands complicate reasoning such as single-sentence and multiple-sentence inferences. Besides, we conduct experiments on state-of-the-art MRC methods in English and Chinese as the first experimental models on UIT-ViQuAD, which will be compared to further models. We also estimate human performances on the dataset and compare it to the experimental results of several powerful machine models. As a result, the substantial differences between humans and the best model performances on the dataset indicate that improvements can be explored on UIT-ViQuAD through future research. Our dataset is freely available to encourage the research community to overcome challenges in Vietnamese MRC.

Machine Reading Comprehension (MRC) is the process where computers or, machines are taught to understand a paragraph or more technically called a context. Like humans, machines also need to be evaluated for their understanding on question answering. MRC is one of the formidable sub-domains in the Natural Language Processing (NLP) domain, which has seen considerable progress over the years. In recent years, many novel datasets have tried to challenge the Machine Reading Comprehension (MRC) models with inference based question answering. With the advancement in NLP, many models have surpassed human-level performance on these datasets, albeit ignoring the obvious disparity between genuine human-level performance and state-of-the-art performance. This highlights the need for attention on the collective improvement of existing datasets, metrics, and models towards “real” prehension. Addressing the lack of sanity in the domain, this paper performs a comparative study on various transformer based models and tries to highlight the success factors of each model. Subsequently, we discuss an MRC model that performs comparatively better, if not the best, on question answering and give directions for future research.KeywordsTransformerNatural language processingReading comprehensionQuestion answeringDeep learning

The essence of "Internet plus" is the relationship and its intelligent connection mode, which promotes the cross-border integration of information communication technology with all walks of life and reduces the information asymmetry. At a time when so much new knowledge is generated every day in the form of text that no one can digest on their own, machine reading comprehension can process vast amounts of information for people to use. With the development of psychometric theory and network technology, computerized adaptive testing has more and more advantages than traditional testing. Most of the current selection strategies are designed based on project response theory and cognitive diagnosis theory, which cannot point out students' learning ability, while cognitive diagnosis can not only evaluate students' ability, but also point out the knowledge of the subjects. This article tries to explore the application of adaptive machine reading comprehension in the measurement system, and build a machine reading comprehension model, introduces the application value of the machine in reading comprehension, secondly to build an adaptive measurement system can be used to machine reading model, finally attempts to solve through the experiment measurement system of text slices and accurately identify the problem.

Numerical reasoning in machine reading comprehension (MRC) has demonstrated significant performance improvements in the past few years. However, due to the process being restricted to specific languages, low-resource languages are not considered, and MRC studies on such languages are limited. In addition, the methods that rely on existing information extracted within the span of a paragraph have limitations in responding to questions requiring actual reasoning. To overcome these shortcomings, this study establishes a dataset for learning Korean Question and Answering (QA) models that not only answer within the span of passages but also perform numerical reasoning on passages and questions. Its efficacy was verified by training the model. We recruited eight annotators to tag the ground truth label, and they annotated datasets with 920, 115, and 115 passages in the train, dev, and test, respectively. A simple yet sophisticated automatic inter-annotation tool was created by effectively reducing the possibility of inaccuracy and error entailed by humans in the data construction process. This tool used common KoBERT and KoELECTRA. We defined four general conditions, and six conditions humans must inspect and fine-tune the pre-trained language models with numerically aware architecture. The KoELECTRA and NumNet+ with KoELECTRA were fine-tuned, and experiments in identical hyperparameter settings showed that compared with other models, the performance of NumNet+ with KoELECTRA was higher by more than 1.3 points. Our research contributes to the Korean MRC research and suggests potential and insight into MRC models capable of numerical reasoning.

We propose a simple and effective strategy for data augmentation for low-resource machine reading comprehension (MRC). Our approach first pretrains the answer extraction components of a MRC system on the augmented data that contains approximate context of the correct answers, before training it on the exact answer spans. The approximate context helps the QA method components in narrowing the location of the answers. We demonstrate that our simple strategy substantially improves both document retrieval and answer extraction performance by providing larger context of the answers and additional training data. In particular, our method significantly improves the performance of BERT based retriever (15.12%), and answer extractor (4.33% F1) on TechQA, a complex, low-resource MRC task. Further, our data augmentation strategy yields significant improvements of up to 3.9% exact match (EM) and 2.7% F1 for answer extraction on PolicyQA, another practical but moderate sized QA dataset that also contains long answer spans.

Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures. In contrast, pretrained language models with Transformer layers, such as GPT (Radford et al., 2018) and BERT (Devlin et al., 2018), have achieved competitive performance on MRC. A research question that naturally arises is: apart from the benefits of pre-training, how many performance gain comes from the unified network architecture. In this work, we evaluate and analyze unifying encoding and matching components with Transformer for the MRC task. Experimental results on SQuAD show that the unified model outperforms previous networks that separately treat encoding and matching. We also introduce a metric to inspect whether a Transformer layer tends to perform encoding or matching. The analysis results show that the unified model learns different modeling strategies compared with previous manually-designed models.

Reading comprehension is one of the most critical abilities in human intelligence, and machine reading comprehension is one of the most popular research directions in the field of natural language processing. Machine reading is one of the key aspects to achieve artificial intelligence. This paper first analyzes the current state of development of machine reading comprehension, and based on the current problems in machine reading comprehension, this paper applies various attention mechanisms in the reading comprehension model, so that the information of articles and questions can be fused with each other to extract the key information and predict the answers more accurately. This paper also designs experiments and verifies that this method has a higher accuracy rate compared with other methods.

Pretrained language models (PrLMs) have been widely used in machine reading comprehension (MRC) tasks. Although PrLMs can learn powerful representations from large-scale corpora, it introduces the problem of model redundancy, which severely restricts the application of mobile devices. Recent studies have suggested the use of early exit (EE) methods for the fast inference of PrLMs. However, there are two limitations to the application of existing EE methods. First, different Transformer layers have different capabilities for learning semantic features. Thus, the lower layers may behave differently from the higher layers. As a result, the earlier the exit from the model, the lower its performance. Second, a greedy search is applied to the classifiers exiting either the start or the end position. However, the start and end positions may also be related. Choosing only one best candidate might be suitable for the current step, but it may be a suboptimal choice in global inference. To address these issues, this study proposes an on-device MRC model with fast inference based on an EE strategy. To ensure that the lower layers have similar abilities as the higher layers, we propose the use of self-distillation to transfer knowledge from the higher layers to the lower ones. Furthermore, a dynamic programming strategy was applied to choose the optimal confidence combination of both the start and end classifiers to better select the exit layers in the inference stage. Experimental results on both SQuAD 1.1 and 2.0 show that the inference of the proposed model is accelerated at the minimum cost of performance loss, thereby outperforming previous EE models.KeywordsMachine reading comprehensionDynamic early exitSelf-distillationAdaptive inference