Metaknowledge Enhanced Open Domain Question Answering with Wiki Documents

Open-domain question answering is an emerging field of natural language processing. Application areas of question answering (QA) systems are increasing with the availability of new datasets, more powerful machines and easy access to voice-enabled devices that interact with humans in their daily lives. Users can get into interactions with QA systems without the need to be limited by any domain. However, there is still need for QA systems that can answer questions in a specific field such as international criminal law, violence of human rights or bill of rights. The recent approaches on open-domain question answering made it possible to extend open-domain QA systems towards specific fields. This paper presents Quill-QA, the new feature of the Quill Platform. Quill Platform is a web platform providing set of tools that digitize and visualize negotiated texts, any text in the form of formal negotiations. Quill-QA is based on Facebook's DrQA system and as a result it adopts a flexible Ranker-Reader architecture. Quill-QA answers questions addressed on the samples of negotiated texts such as U.S. Constitutional Convention of 1787, Utah State Constitutional Convention of 1895 and European Union Act 2017. Without limiting the scope of the input questions to a finite set of templates, Quill-QA is the first QA system that focuses on negotiated texts.

Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system for the management of multilingual collections. BRUJ rkstions (English, Spanish and French). The BRUJA architecture is not formed with three monolingual QA systems but instead uses English as Interlingua to make usual QA tasks such as question classifications and answer extractions. In addition, BRUJA uses Cross Language Information Retrieval (CLIR) techniques to retrieve relevant documents from a multilingual collection. On the one hand, we have more documents to find answers from but on the other hand, we are introducing noise into the system because of translations to the Interlingua (English) and the CLIR module. The question is whether the difficulty of managing three languages is worth it or whether a monolingual QA system delivers better results. We report on in-depth experimentation and demonstrate that our multilingual QA system gets better results than its monolingual counterpart whenever it uses good translation resources and, especially, CLIR techniques that are state-of-the-art.

This paper presents a study of the negative effect of Machine Translation (MT) on the precision of Cross---Lingual Question Answering (CL---QA). For this research, a English---Spanish Question Answering (QA) system is used. Also, the sets of 200 official questions from CLEF 2004 and 2006 are used. The CL experimental evaluation using MT reveals that the precision of the system drops around 30% with regard to the monolingual Spanish task. Our main contribution consists on a taxonomy of the identified errors caused by using MT and how the errors can be overcome by using our proposals. An experimental evaluation proves that our approach performs better than MT tools, at the same time contributing to this CL---QA system being ranked first at English---Spanish QA CLEF 2006.

Number of people having expertise in a certain domain is less than people who need information in that domain. In this situation, an automatic question answering (QA) system is necessary. Observing available manual QA sites on internet, the real world question that people usually ask have different expected answer type (EAT) compared to a common automatic QA. Addressing a case study of a religion domain which makes it a closed domain QA, we proposed the EAT into 6 types: LAW, DEFINITION, COMPARISON, METHOD, TIME and PERSON. Different with common QA approach, we built the QA system using case based approach which consists of two main components: Question Analyzer and Case Retriever. Related with the case based reasoning (CBR) framework, these two main components act as the Retrieve and Reuse process while the Revise and Retain process is handle by Case Retainer component. The QA system was built using available Indonesian Natural Language Processing (NLP) tools and FreeCBR as the CBR library. The experiments were done to calculate the accuracy and testing the system with unknown case. By using 77 cases collected from internet with assumption that all answers are available, the experiments achieved 97% accuracy. And by using 10 test cases for the unknown case, the similarity score calculated by the system showed that the test questions have no answer in the available case base.

Question answering (QA) models use retriever and reader systems to answer questions. Reliance on training data by QA systems can amplify or reflect inequity through their responses. Many QA models, such as those for the SQuAD dataset, are trained and tested on a subset of Wikipedia articles which encode their own biases and also reproduce real-world inequality. Understanding how training data affects bias in QA systems can inform methods to mitigate inequity. We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias. We feed three deep-learning-based QA systems with our question sets and evaluate responses for bias via the metrics. Using our metrics, we find that open-domain QA models amplify biases more than their closed-domain counterparts and propose that biases in the retriever surface more readily due to greater freedom of choice.

Most Knowledge Graph-based Question Answering (KGQA) systems rely on training data to reach their optimal performance. However, acquiring training data for supervised systems is both time-consuming and resource-intensive. To address this, in this paper, we propose <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Tree-KGQA</b> , an unsupervised KGQA system leveraging pre-trained language models and tree-based algorithms. Entity and relation linking are essential components of any KGQA system. We employ several pre-trained language models in the entity linking task to recognize the entities mentioned in the question and obtain the contextual representation for indexing. Furthermore, for relation linking we incorporate a pre-trained language model previously trained for language inference task. Finally, we introduce a novel algorithm for extracting the answer entities from a KG, where we construct a forest of interpretations and introduce tree-walking and tree disambiguation techniques. Our algorithm uses the linked relation and predicts the tree branches that eventually lead to the potential answer entities. The proposed method achieves 4.5% and 7.1% gains in F1 score in entity linking tasks on LC-QuAD 2.0 and LC-QuAD 2.0 (KBpearl) datasets, respectively, and a 5.4% increase in the relation linking task on LC-QuAD 2.0 (KBpearl). The comprehensive evaluations demonstrate that our unsupervised KGQA approach outperforms other supervised state-of-the-art methods on the WebQSP-WD test set (1.4% increase in F1 score) - without training on the target dataset.

Question Answering (QA) systems are becoming the inspiring model for the future of search engines. While, recently, datasets underlying QA systems have been promoted from unstructured datasets to structured datasets with semantically highly enriched metadata, question answering systems are still facing serious challenges and are therefore not meeting users' expectations. This paper provides an exhaustive insight of challenges known so far for building QA systems, with a special focus on employing structured data (i.e. knowledge graphs).It thus helps researchers to easily spot gaps to fill with their future research agendas.

Previous works on evaluating the performance of Question Answering (QA) systems are focused on the evaluation of the precision. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, using the proposed measures. The main conclusion is that the evaluation of QA systems in realtime can be a new scenario for the evaluation of QA systems.

Aiming at open domain question answering system evaluation task in the fourth CCF Natural Language Processing and Chinese Computing Conference NLPCC2015, a solution of automatic question answering which can answer natural language questions is proposed. Firstly, SPE Subject Predicate Extraction algorithm is presented to find answers from the knowledge base, and then WKE Web Knowledge Extraction algorithm is used to extract answers from search engine query result. Experimental data provided in the evaluation task includes the knowledge base and questions in natural language. The evaluation result shows that MRR is 0.5670, accuracy is 0.5700, and average F1 is 0.5240, and indicates the proposed method is feasible in open domain question answering system.

PurposeThe purpose of this paper is to build a better question answering (QA) system that can furnish more improved retrieval of answers related to COVID-19 queries from the COVID-19 open research data set (CORD-19). As CORD-19 has an up-to-date collection of coronavirus literature, text mining approaches can be successfully used to retrieve answers pertaining to all coronavirus-related questions. The existing a lite BERT for self-supervised learning of language representations (ALBERT) model is finetuned for retrieving all COVID relevant information to scientific questions posed by the medical community and to highlight the context related to the COVID-19 query.Design/methodology/approachThis study presents a finetuned ALBERT-based QA system in association with Best Match25 (Okapi BM25) ranking function and its variant BM25L for context retrieval and provided high scores in benchmark data sets such as SQuAD for answers related to COVID-19 questions. In this context, this paper has built a QA system, pre-trained on SQuAD and finetuned it on CORD-19 data to retrieve answers related to COVID-19 questions by extracting semantically relevant information related to the question.FindingsBM25L is found to be more effective in retrieval compared to Okapi BM25. Hence, finetuned ALBERT when extended to the CORD-19 data set provided accurate results.Originality/valueThe finetuned ALBERT QA system was developed and tested for the first time on the CORD-19 data set to extract context and highlight the span of the answer for more clarity to the user.

Information can also be a means of learning for humans. Including information about history because history can be a means of learning for the younger generation to appreciate the nation's culture and build national identity. In the past, the Sumedang Larang kingdom was one of the many kingdoms in West Java, Indonesia, that could be used as much information as a lesson. Technological developments make more and more information available for study. We need the proper means to find the information we need. This study aims to build a Question Answering (QA) system to create a means for the younger generation to be more familiar with the history of the kingdom in the past. The QA system offers an information retrieval system that is easy to access and can immediately provide the answers we need. This QA system was built using ontology as a knowledge base and cosine similarity to determine the similarity between user questions and the dataset. The QA system that has been built is tested by providing a set of questions so that the system's performance can be measured, and the results of system testing get a precision value of 70% and a recall value of 90%.

In recent years, medical Question Answering (QA) systems have gained a lot of attentions, since they can not only help professionals make quick decisions, but also give ordinary people advice when they seek for helpful information. However, the interpretability and accuracy problems have plagued QA systems for a long time. In this paper, we propose a QA system, DSQA, based on knowledge graph for answering domain-specific medical questions. The data we use are from Electronic Medical Records (EMRs) that are complex and heterogeneous with varied qualities. We introduce doctor-in-the-loop mechanism to design the knowledge graph, which improves the interpretability and the accuracy of our QA system. For every natural-language question, we generate the answer with a hybrid of the traditional deep learning model and the reasoning on the knowledge graph. Finally, the answer is returned to the user in the format of a subgraph with corresponding natural-language sentences. Our system shows high interpretability and gives excellent performance according to the experiment results.

It is challenging for current one-step retrieve-and-read question answering (QA) systems to answer questions like “Which novel by the author of ‘Armada’ will be adapted as a feature film by Steven Spielberg?” because the question seldom contains retrievable clues about the missing entity (here, the author). Answering such a question requires multi-hop reasoning where one must gather information about the missing entity (or facts) to proceed with further reasoning. We present GoldEn (Gold Entity) Retriever, which iterates between reading context and retrieving more supporting documents to answer open-domain multi-hop questions. Instead of using opaque and computationally expensive neural retrieval models, GoldEn Retriever generates natural language search queries given the question and available context, and leverages off-the-shelf information retrieval systems to query for missing entities. This allows GoldEn Retriever to scale up efficiently for open-domain multi-hop reasoning while maintaining interpretability. We evaluate GoldEn Retriever on the recently proposed open-domain multi-hop QA dataset, HotpotQA, and demonstrate that it outperforms the best previously published model despite not using pretrained language models such as BERT.

An open domain question answering system is one of the emerging information retrieval systems available on the World Wide Web that is becoming popular day by day to get succinct and relevant answers in response of users' questions. The validation of the correctness of the answer is an important issue in the field of question answering. In this paper, we are proposing a World Wide Web based solution for answer validation where answers returned by open domain Question Answering Systems can be validated using online resources such as Wikipedia and Google. We have applied several heuristics for answer validation task and tested them against some popular World Wide Web based open domain Question Answering Systems over a collection of 500 questions collected from standard sources such as TREC, the Worldbook, and the Worldfactbook. We found that the proposed method is yielding promising results for automatic answer validation task.

Datasets that boosted state-of-the-art solutions for Question Answering (QA) systems prove that it is possible to ask questions in natural language manner. However, users are still used to query-like systems where they type in keywords to search for answer. In this study we validate which parts of questions are essential for obtaining valid answer. In order to conclude that, we take advantage of LIME - a framework that explains prediction by local approximation. We find that grammar and natural language is disregarded by QA. State-of-the-art model can answer properly even if 'asked' only with a few words with high coefficients calculated with LIME. According to our knowledge, it is the first time that QA model is being explained by LIME.