Various Legal Factors Extraction Based on Machine Reading Comprehension

Machine reading comprehension (MRC) is a challenging natural language processing (NLP) task. It has a wide application potential in the fields of question answering robots, human‐computer interactions in mobile virtual reality systems, etc. Recently, the emergence of pretrained models (PTMs) has brought this research field into a new era, in which the training objective plays a key role. The masked language model (MLM) is a self‐supervised training objective widely used in various PTMs. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking, entity masking, phrase masking, and span masking. In different MLMs, the length of the masked tokens is different. Similarly, in different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence. Thus, in MRC tasks with different answer lengths, whether the length of MLM is related to performance is a question worth studying. If this hypothesis is true, it can guide us on how to pretrain the MLM with a relatively suitable mask length distribution for MRC tasks. In this paper, we try to uncover how much of MLM’s success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in the MRC dataset. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple‐choice cloze task, and long multiple‐choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation.

Machine reading comprehension(MRC), which employs computers to answer questions from given passages, is a popular research field. In natural language, a natural hierarchical representation can be seen: characters, words, phrases, sentences, paragraphs, and documents. Current studies have demonstrated that hierarchical information can help machines understand natural language. However, prior works focused on the overall performance of MRC tasks without considering hierarchical information. In addition, the noise problem still has not been adequately addressed, even though many researchers have adopted the technique of passage reranking. Thus, in this paper, focusing on noise information processing and the extraction of hierarchical information, we propose a model (PH-Model) with a passage reranking framework (P) and hierarchical neural network (H) for a Chinese multi-passage MRC task. PH-Model produces more precise answers by reducing noise information and extracting hierarchical information. Experimental results on the DuReader 2.0 dataset (a large scale real-world Chinese MRC dataset) show that PH-Model outperforms the ROUGE-L and BLEU-4 baseline by 18.24% and 24.17%, respectively.

BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance.ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.

Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.

Event extraction as machine reading comprehension with question-context bridging

Machine Reading Comprehension(MRC), including a series of tasks that test the ability of models to understand natural language, has received quite a few attention in Natural Language Processing(NLP). Most existing works deal with MRC tasks by exploiting the expression capability of neural networks. Some of them have achieved impressive performance. Despite the rapid iteration of the models used, few work have focused on output layer and prediction method of answer span - also known as span extraction. In this paper, we focus on span extraction in the Question Answering(QA) task. A cross-sectional comparison of widely used span extraction methods is presented, with their strengths and weaknesses noted in detail. Furthermore, inspired by Faster R-CNN, we propose a brand new span extraction method. Experiment results show that our proposed method outperforms existing span extraction methods on both English and Chinese MRC tasks.

Machine reading comprehension is a classic issue artificial intelligence. It is a key technology in the next generation search engine and intelligent interactive service. The traditional methods usually work in a small scale of data sets. The traditional system cannot meet the emerging demand. Deep learning and cloud computation have ability to deal with the large scale data sets. In real scene, the parameters affect the performance of machine reading comprehension task. In this paper, we analyze how the parameters of deep neural network affect the machine reading comprehension. The experiment results show that the performance is only sensitive to a few parameters which should be key point for engineers.

As a broad and major category in machine reading comprehension (MRC), the generalized goal of discriminative MRC is answer prediction from the given materials. However, the focuses of various discriminative MRC tasks may be diverse enough: multi-choice MRC requires model to highlight and integrate all potential critical evidence globally; while extractive MRC focuses on higher local boundary preciseness for answer extraction. Among previous works, there lacks a unified design with pertinence for the overall discriminative MRC tasks. To fill in above gap, we propose a lightweight POS-Enhanced Iterative Co-Attention Network (POI-Net) as the first attempt of unified modeling with pertinence, to handle diverse discriminative MRC tasks synchronously. Nearly without introducing more parameters, our lite unified design brings model significant improvement with both encoder and decoder components. The evaluation results on four discriminative MRC benchmarks consistently indicate the general effectiveness and applicability of our model, and the code is available at https://github.com/Yilin1111/poi-net.

One of the emerging research trends in natural language understanding is machine reading comprehension (MRC) which is the task to find answers to human questions based on textual data. While many datasets have been developed for MRC research for other languages, there is a lack of such resources for the Vietnamese language. Although many datasets and methodologies have been developed for English and Chinese, many Vietnamese machine reading comprehension limitations need to be solved further. Existing Vietnamese datasets for MRC research concentrate solely on answerable questions. However, in reality, questions can be unanswerable for which the correct answer is not stated in the given textual data. To address the weakness, we provide the research community with a benchmark dataset named UIT-ViQuAD 2.0 for evaluating the MRC task and question answering systems for the Vietnamese language. We use UIT-ViQuAD 2.0 as a benchmark dataset for the shared task on Vietnamese machine reading comprehension (VLSP2021-MRC) at the Eighth Workshop on Vietnamese Language and Speech Processing (VLSP 2021). This task attracted 77 participant teams from 34 universities and other organizations. Each participant was provided with the training data, including 28,457 annotated question-answer pairs, and returned the result on a public test set of more than 3,821 questions and a private test set of 3,712 questions. In this article, we present details of the organization of the shared task, an overview of the methods employed by shared-task participants, and the results. The highest performances in this competition are 77.24% (in EM) and 67.43% (in F1-score) on the private test set. The Vietnamese MRC systems proposed by the top 3 teams use XLM-RoBERTa, a powerful pre-trained language model based on the transformer architecture that has achieved state-of-the-art results on many natural language processing tasks. We believe that releasing the UIT-ViQuAD 2.0 dataset motivates more researchers to improve Vietnamese machine reading comprehension.

This paper presents the development process of a Vietnamese spoken language corpus for machine reading comprehension (MRC) tasks and provides insights into the challenges and opportunities associated with using realworld data for machine reading comprehension tasks.The existing MRC corpora in Vietnamese mainly focus on formal written documents such as Wikipedia articles, online newspapers, or textbooks.In contrast, the VlogQA consists of 10,076 question-answer pairs based on 1,230 transcript documents sourced from YouTube -an extensive source of user-uploaded content, covering the topics of food and travel.By capturing the spoken language of native Vietnamese speakers in natural settings, an obscure corner overlooked in Vietnamese research, the corpus provides a valuable resource for future research in reading comprehension tasks for the Vietnamese language.Regarding performance evaluation, our deep-learning models achieved the highest F1 score of 75.34% on the test set, indicating significant progress in machine reading comprehension for Vietnamese spoken language data.In terms of EM, the highest score we accomplished is 53.97%, which reflects the challenge in processing spoken-based content and highlights the need for further improvement.

The task of machine reading comprehension (MRC) has evolved from answering simple questions from well-edited text to answering real questions from users out of web data. In the real-world setting, full-body text from multiple relevant documents in the top search results are provided as context for questions from user queries, including not only questions with a single, short, and factual answer, but also questions about reasons, procedures, and opinions. In this case, multiple answers could be equally valid for a single question and each answer may occur multiple times in the context, which should be taken into consideration when we build MRC system. We propose a multi-answer multi-task framework, in which different loss functions are used for multiple reference answers. Minimum Risk Training is applied to solve the multi-occurrence problem of a single answer. Combined with a simple heuristic passage extraction strategy for overlong documents, our model increases the ROUGE-L score on the DuReader dataset from 44.18, the previous state-of-the-art, to 51.09.

Task requirements (TRs) writing is an important question type in Key English Test and Preliminary English Test. A TR writing question may include multiple requirements and a high-quality essay must respond to each requirement thoroughly and accurately. However, the limited teacher resources prevent students from getting detailed grading instantly. The majority of existing automatic essay scoring systems focus on giving a holistic score but rarely provide reasons to support it. In this paper, we proposed an end-to-end framework based on machine reading comprehension (MRC) to address this problem to some extent. The framework not only detects whether an essay responds to a requirement question, but clearly marks where the essay answers the question. Our framework consists of three modules: question normalization module, ELECTRA based MRC module and response locating module. We extensively explore state-of-the-art MRC methods. Our approach achieves 0.93 accuracy score and 0.85 F1 score on a real-world educational dataset. To encourage reproducible results, we make our code publicly available at https://github.com/aied2021TRMRC/AIED_2021_TRMRC_code.KeywordsTask requirements writingMachine reading comprehensionPre-training language modelNeural networks

Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. Here, we demonstrate a simple yet effective method to attack MRC models and reveal the statistical biases in these models. We apply the method to the RACE dataset, for which the answer to each MRC question is selected from 4 options. It is found that several pre-trained language models, including BERT, ALBERT, and RoBERTa, show consistent preference to some options, even when these options are irrelevant to the question. When interfered by these irrelevant options, the performance of MRC models can be reduced from human-level performance to the chance-level performance. Human readers, however, are not clearly affected by these irrelevant options. Finally, we propose an augmented training method that can greatly reduce models' statistical biases.

Medical Named Entity Recognition (NER) is a fundamental but challenging task due to the lack of specialized entity datasets like tumor entities, which are often overlapped and discontinuous. In this paper, we propose a novel Prompt Enhanced Generative Machine Reading Comprehension Framework (PGMRC) to improve the overlapped and discontinuous NER performance. Specifically, we formulate NER as a Machine Reading Comprehension (MRC) task and employ a pre-trained encoder-decoder module to generate entity span sequences according to their entity query. In this way, we adopt query to guide the model to focus on answer entities in context, which can naturally solve entity overlap and alleviate the exposure bias of the generative model. Then, we introduce continuous prompts to the self-attention mechanism in Transformer to reduce the dependence on manually constructed queries. In addition, we annotate 875 pathological documents of pancreatic cancer and construct a Chinese pathological NER dataset (PAN) containing overlapped and discontinuous entities. Finally, we conduct our experiments on three widely used benchmarks (GENIA, ACE04, ACE05) and our dataset PAN. Experiments have demonstrated its effectiveness and better performance than state-of-the-art methods.

Pretrained language models (PrLMs) have been widely used in machine reading comprehension (MRC) tasks. Although PrLMs can learn powerful representations from large-scale corpora, it introduces the problem of model redundancy, which severely restricts the application of mobile devices. Recent studies have suggested the use of early exit (EE) methods for the fast inference of PrLMs. However, there are two limitations to the application of existing EE methods. First, different Transformer layers have different capabilities for learning semantic features. Thus, the lower layers may behave differently from the higher layers. As a result, the earlier the exit from the model, the lower its performance. Second, a greedy search is applied to the classifiers exiting either the start or the end position. However, the start and end positions may also be related. Choosing only one best candidate might be suitable for the current step, but it may be a suboptimal choice in global inference. To address these issues, this study proposes an on-device MRC model with fast inference based on an EE strategy. To ensure that the lower layers have similar abilities as the higher layers, we propose the use of self-distillation to transfer knowledge from the higher layers to the lower ones. Furthermore, a dynamic programming strategy was applied to choose the optimal confidence combination of both the start and end classifiers to better select the exit layers in the inference stage. Experimental results on both SQuAD 1.1 and 2.0 show that the inference of the proposed model is accelerated at the minimum cost of performance loss, thereby outperforming previous EE models.KeywordsMachine reading comprehensionDynamic early exitSelf-distillationAdaptive inference