Neural Domain Adaptation for Biomedical Question Answering

Biomedical factoid question answering is an important task in biomedical question answering applications. It has attracted much attention because of its reliability. In question answering systems, better representation of words is of great importance, and proper word embedding can significantly improve the performance of the system. With the success of pretrained models in general natural language processing tasks, pretrained models have been widely used in biomedical areas, and many pretrained model-based approaches have been proven effective in biomedical question-answering tasks. In addition to proper word embedding, name entities also provide important information for biomedical question answering. Inspired by the concept of transfer learning, in this study, we developed a mechanism to fine-tune BioBERT with a named entity dataset to improve the question answering performance. Furthermore, we applied BiLSTM to encode the question text to obtain sentence-level information. To better combine the question level and token level information, we use bagging to further improve the overall performance. The proposed framework was evaluated on BioASQ 6b and 7b datasets, and the results have shown that our proposed framework can outperform all baselines.

This paper compares some existing evaluation metrics for factoid question answering (QA) from the viewpoint of stability and sensitivity, using the NTCIR-4 QAC2 Japanese factoid QA tasks and the Buckley/Voorhees stability method and Voorhees/Buckley swap method. Our main findings are: (1) For QA evaluation with ranked lists containing up to five answers, the fraction of questions with a correct answer within top 5 (NQcorrect5) and that with a correct answer at rank 1 (NQcorrect1) are not as stable and sensitive as reciprocal rank. (2) Q-measure, which can handle multiple correct answers and answer correctness levels, is at least as stable and sensitive as reciprocal rank, provided that a mild gain value assignment is used. Emphasizing answer correctness levels tends to hurt stability and sensitivity, while handling multiple correct answers improves them. As our experimental methods are language-independent, we believe that these findings apply to QA in languages other than Japanese as well.

Question Answering System (QAS) aims at providing the most appropriate answer to the user’s question asked in any natural language. It emerged as a future of web search that crawls structured and unstructured collections of documents to find answers. A lot of research has already been done on answering factoid questions and achieved promising accuracy, whereas non-factoid question answering is a challenging task and explores newer avenues for fact-finding and analysis. The research on handling different modules of non-factoid question answering has drastically increased from 2012 which evokes the need for summarizing and providing a comprehensive analysis of the work that has already been done in this field. The survey identifies different challenges faced in each module, how they have been addressed by the researchers with their advantages or disadvantages and further improvements. The survey is written with an objective to help the researchers understand the field with how it has been undertaken by different approaches with their impact on performance. The paper finally concludes by investigating research gaps and research objectives that are promising to get an insight into future research directions.

The question of classical Factoid Question Answering (FQA) task is always in the form of a single sentence. There also exists another kind of FQA task, whose question is a descriptive paragraph, such as quiz bowl question answering. Recently, some works try to automatically answer paragraph questions by applying machine learning methods. However, these methods neglect the correlation information between sentences in a paragraph and do not take full advantage of answer embedding information. In this paper, we propose a novel Hierarchical Convolutional Neural Network, called HCNN-E, to settle the task by considering ordinal information of sentences in paragraph and the information of answer embeddings. The experimental results on two public datasets demonstrate the effectiveness of proposed method, and the proposed method can achieve approximately 10% - 20% improvements, when comparing with the baselines.

Open-domain question answering is an emerging field of natural language processing. Application areas of question answering (QA) systems are increasing with the availability of new datasets, more powerful machines and easy access to voice-enabled devices that interact with humans in their daily lives. Users can get into interactions with QA systems without the need to be limited by any domain. However, there is still need for QA systems that can answer questions in a specific field such as international criminal law, violence of human rights or bill of rights. The recent approaches on open-domain question answering made it possible to extend open-domain QA systems towards specific fields. This paper presents Quill-QA, the new feature of the Quill Platform. Quill Platform is a web platform providing set of tools that digitize and visualize negotiated texts, any text in the form of formal negotiations. Quill-QA is based on Facebook's DrQA system and as a result it adopts a flexible Ranker-Reader architecture. Quill-QA answers questions addressed on the samples of negotiated texts such as U.S. Constitutional Convention of 1787, Utah State Constitutional Convention of 1895 and European Union Act 2017. Without limiting the scope of the input questions to a finite set of templates, Quill-QA is the first QA system that focuses on negotiated texts.

BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance.ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.

Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system for the management of multilingual collections. BRUJ rkstions (English, Spanish and French). The BRUJA architecture is not formed with three monolingual QA systems but instead uses English as Interlingua to make usual QA tasks such as question classifications and answer extractions. In addition, BRUJA uses Cross Language Information Retrieval (CLIR) techniques to retrieve relevant documents from a multilingual collection. On the one hand, we have more documents to find answers from but on the other hand, we are introducing noise into the system because of translations to the Interlingua (English) and the CLIR module. The question is whether the difficulty of managing three languages is worth it or whether a monolingual QA system delivers better results. We report on in-depth experimentation and demonstrate that our multilingual QA system gets better results than its monolingual counterpart whenever it uses good translation resources and, especially, CLIR techniques that are state-of-the-art.

This paper presents a study of the negative effect of Machine Translation (MT) on the precision of Cross---Lingual Question Answering (CL---QA). For this research, a English---Spanish Question Answering (QA) system is used. Also, the sets of 200 official questions from CLEF 2004 and 2006 are used. The CL experimental evaluation using MT reveals that the precision of the system drops around 30% with regard to the monolingual Spanish task. Our main contribution consists on a taxonomy of the identified errors caused by using MT and how the errors can be overcome by using our proposals. An experimental evaluation proves that our approach performs better than MT tools, at the same time contributing to this CL---QA system being ranked first at English---Spanish QA CLEF 2006.

Natural language processing systems are often downstream of unreliable inputs: machine translation, optical character recognition, or speech recognition. For instance, virtual assistants can only answer your questions after understanding your speech. We investigate and mitigate the effects of noise from Automatic Speech Recognition systems on two factoid Question Answering (QA) tasks. Integrating confidences into the model and forced decoding of unknown words are empirically shown to improve the accuracy of downstream neural QA systems. We create and train models on a synthetic corpus of over 500,000 noisy sentences and evaluate on two human corpora from Quizbowl and Jeopardy! competitions.

Number of people having expertise in a certain domain is less than people who need information in that domain. In this situation, an automatic question answering (QA) system is necessary. Observing available manual QA sites on internet, the real world question that people usually ask have different expected answer type (EAT) compared to a common automatic QA. Addressing a case study of a religion domain which makes it a closed domain QA, we proposed the EAT into 6 types: LAW, DEFINITION, COMPARISON, METHOD, TIME and PERSON. Different with common QA approach, we built the QA system using case based approach which consists of two main components: Question Analyzer and Case Retriever. Related with the case based reasoning (CBR) framework, these two main components act as the Retrieve and Reuse process while the Revise and Retain process is handle by Case Retainer component. The QA system was built using available Indonesian Natural Language Processing (NLP) tools and FreeCBR as the CBR library. The experiments were done to calculate the accuracy and testing the system with unknown case. By using 77 cases collected from internet with assumption that all answers are available, the experiments achieved 97% accuracy. And by using 10 test cases for the unknown case, the similarity score calculated by the system showed that the test questions have no answer in the available case base.

While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects. First, we lack QA datasets covering complex questions that involve answers as well as the reasoning processes to get them. As a result, the state-of-the-art QA research on numerical reasoning still focuses on simple calculations and does not provide the mathematical expressions or evidence justifying the answers. Second, the QA community has contributed a lot of effort to improve the interpretability of QA models. However, they fail to explicitly show the reasoning process, such as the evidence order for reasoning and the interactions between different pieces of evidence. To address the above shortcoming, we introduce NOAHQA, a conversational and bilingual QA dataset with questions requiring numerical reasoning with compound mathematical expressions. With NOAHQA, we develop an interpretable reasoning graph as well as the appropriate evaluation metric to measure the answer quality. We evaluate the state-of-the-art QA models trained using existing QA datasets on NOAHQA and show that the best among them can only achieve 55.5 exact match scores, while the human performance is 89.7. We also present a new QA model for generating a reasoning graph where the reasoning graph metric still has a large gap compared with that of humans, eg, 28 scores.

Previous works on evaluating the performance of Question Answering (QA) systems are focused on the evaluation of the precision. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, using the proposed measures. The main conclusion is that the evaluation of QA systems in realtime can be a new scenario for the evaluation of QA systems.

Conversational search is the dominant intent of Question Answering, which is achieved through different NLP techniques, Deep Learning models. The advent of Transformer models has been a breakthrough in Natural Language Processing applications, which has attained a benchmark on state of NLP tasks such as question answering. Here we propose a semantic Malayalam Question Answering system that automatically answers the queries related to health issues. The Biomedical Question-Answering, especially in the Malayalam language, is a tedious and challenging task. The proposed model uses a neural network-based Bidirectional Encoder Representation from Transformers (BERT), to implement the question answering system. In this study, we investigate how to train and fine-tune a BERT model for Question-Answering. The system has been tested with our own annotated Malayalam SQUAD form health dataset. In comparing the result with our previous works - Word embedding and RNN based model, identified we find that our BERT model is more accurate than the previous models and achieves an F1 score of 86%.

We present a semi-automated framework for constructing factoid question answering (QA) datasets, where an array of question characteristics are formalized, including structure complexity, function, commonness, answer cardinality, and paraphrasing. Instead of collecting questions and manually characterizing them, we employ a reverse procedure, first generating a kind of graph-structured logical forms from a knowledge base, and then converting them into questions. Our work is the first to generate questions with explicitly specified characteristics for QA evaluation. We construct a new QA dataset with over 5,000 logical form-question pairs, associated with answers from the knowledge base, and show that datasets constructed in this way enable finegrained analyses of QA systems. The dataset can be found in https://github.com/ ysu1989/GraphQuestions.

Datasets that boosted state-of-the-art solutions for Question Answering (QA) systems prove that it is possible to ask questions in natural language manner. However, users are still used to query-like systems where they type in keywords to search for answer. In this study we validate which parts of questions are essential for obtaining valid answer. In order to conclude that, we take advantage of LIME - a framework that explains prediction by local approximation. We find that grammar and natural language is disregarded by QA. State-of-the-art model can answer properly even if 'asked' only with a few words with high coefficients calculated with LIME. According to our knowledge, it is the first time that QA model is being explained by LIME.