A survey on question answering systems over linked data and documents

Open-domain question answering is an emerging field of natural language processing. Application areas of question answering (QA) systems are increasing with the availability of new datasets, more powerful machines and easy access to voice-enabled devices that interact with humans in their daily lives. Users can get into interactions with QA systems without the need to be limited by any domain. However, there is still need for QA systems that can answer questions in a specific field such as international criminal law, violence of human rights or bill of rights. The recent approaches on open-domain question answering made it possible to extend open-domain QA systems towards specific fields. This paper presents Quill-QA, the new feature of the Quill Platform. Quill Platform is a web platform providing set of tools that digitize and visualize negotiated texts, any text in the form of formal negotiations. Quill-QA is based on Facebook's DrQA system and as a result it adopts a flexible Ranker-Reader architecture. Quill-QA answers questions addressed on the samples of negotiated texts such as U.S. Constitutional Convention of 1787, Utah State Constitutional Convention of 1895 and European Union Act 2017. Without limiting the scope of the input questions to a finite set of templates, Quill-QA is the first QA system that focuses on negotiated texts.

Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system for the management of multilingual collections. BRUJ rkstions (English, Spanish and French). The BRUJA architecture is not formed with three monolingual QA systems but instead uses English as Interlingua to make usual QA tasks such as question classifications and answer extractions. In addition, BRUJA uses Cross Language Information Retrieval (CLIR) techniques to retrieve relevant documents from a multilingual collection. On the one hand, we have more documents to find answers from but on the other hand, we are introducing noise into the system because of translations to the Interlingua (English) and the CLIR module. The question is whether the difficulty of managing three languages is worth it or whether a monolingual QA system delivers better results. We report on in-depth experimentation and demonstrate that our multilingual QA system gets better results than its monolingual counterpart whenever it uses good translation resources and, especially, CLIR techniques that are state-of-the-art.

The paper addresses the limitations of traditional evaluation metrics for Question Answering (QA) systems that primarily focus on syntax and n-gram similarity. We propose a novel model-based evaluation metric, MQA-metric, and create a human-judgment-based dataset, squad-qametric and marco-qametric, to validate our approach. The research aims to solve several key problems: the objectivity in dataset labeling, the effectiveness of metrics when there is no syntax similarity, the impact of answer length on metric performance, and the influence of real answer quality on metric results. To tackle these challenges, we designed an interface for dataset labeling and conducted extensive experiments with human reviewers. Our analysis shows that the MQA-metric outperforms traditional metrics like BLEU, ROUGE and METEOR. Unlike existing metrics, MQA-metric leverages semantic comprehension through large language models (LLMs), enabling it to capture contextual nuances and synonymous expressions more effectively. This approach sets a standard for evaluating QA systems by prioritizing semantic accuracy over surface-level similarities. The proposed metric correlates better with human judgment, making it a more reliable tool for evaluating QA systems. Our contributions include the development of a robust evaluation workflow, creation of high-quality datasets, and an extensive comparison with existing evaluation methods. The results indicate that our model-based approach provides a significant improvement in assessing the quality of QA systems, which is crucial for their practical application and trustworthiness.

This paper presents a study of the negative effect of Machine Translation (MT) on the precision of Cross---Lingual Question Answering (CL---QA). For this research, a English---Spanish Question Answering (QA) system is used. Also, the sets of 200 official questions from CLEF 2004 and 2006 are used. The CL experimental evaluation using MT reveals that the precision of the system drops around 30% with regard to the monolingual Spanish task. Our main contribution consists on a taxonomy of the identified errors caused by using MT and how the errors can be overcome by using our proposals. An experimental evaluation proves that our approach performs better than MT tools, at the same time contributing to this CL---QA system being ranked first at English---Spanish QA CLEF 2006.

Question answering systems in general use external knowledge sources for extracting answers. Domain specific question answering systems require pre-constructed knowledge sources like domain ontology. A major challenge in knowledge-based question answering system development is building a huge knowledge base with the objective and correct factual knowledge in the preferred domain. The process of collecting useful knowledge from various sources and maintaining in a knowledge repository is a useful process for providing required answer on demand with greater accuracy and efficiency. In this paper an experimental framework has been proposed for concept-relational ontology-based question answering process. The question answering framework proposed in this paper includes two subsystems: 1) a dynamic concept relational ontology construction module, is capable to extract new concepts from the web and incorporate the extracted knowledge into the concept relational ontology knowledge base; 2) the answer extraction module formulates the query string from the natural language question according to the expected answer and retrieves the information from the ontology for answer formation. Furthermore, a novel statistical validation framework for evaluating the prototype implementation of the proposed question answering system is introduced.

PurposeThe purpose of this paper is to build a better question answering (QA) system that can furnish more improved retrieval of answers related to COVID-19 queries from the COVID-19 open research data set (CORD-19). As CORD-19 has an up-to-date collection of coronavirus literature, text mining approaches can be successfully used to retrieve answers pertaining to all coronavirus-related questions. The existing a lite BERT for self-supervised learning of language representations (ALBERT) model is finetuned for retrieving all COVID relevant information to scientific questions posed by the medical community and to highlight the context related to the COVID-19 query.Design/methodology/approachThis study presents a finetuned ALBERT-based QA system in association with Best Match25 (Okapi BM25) ranking function and its variant BM25L for context retrieval and provided high scores in benchmark data sets such as SQuAD for answers related to COVID-19 questions. In this context, this paper has built a QA system, pre-trained on SQuAD and finetuned it on CORD-19 data to retrieve answers related to COVID-19 questions by extracting semantically relevant information related to the question.FindingsBM25L is found to be more effective in retrieval compared to Okapi BM25. Hence, finetuned ALBERT when extended to the CORD-19 data set provided accurate results.Originality/valueThe finetuned ALBERT QA system was developed and tested for the first time on the CORD-19 data set to extract context and highlight the span of the answer for more clarity to the user.

Information can also be a means of learning for humans. Including information about history because history can be a means of learning for the younger generation to appreciate the nation's culture and build national identity. In the past, the Sumedang Larang kingdom was one of the many kingdoms in West Java, Indonesia, that could be used as much information as a lesson. Technological developments make more and more information available for study. We need the proper means to find the information we need. This study aims to build a Question Answering (QA) system to create a means for the younger generation to be more familiar with the history of the kingdom in the past. The QA system offers an information retrieval system that is easy to access and can immediately provide the answers we need. This QA system was built using ontology as a knowledge base and cosine similarity to determine the similarity between user questions and the dataset. The QA system that has been built is tested by providing a set of questions so that the system's performance can be measured, and the results of system testing get a precision value of 70% and a recall value of 90%.

In recent years, medical Question Answering (QA) systems have gained a lot of attentions, since they can not only help professionals make quick decisions, but also give ordinary people advice when they seek for helpful information. However, the interpretability and accuracy problems have plagued QA systems for a long time. In this paper, we propose a QA system, DSQA, based on knowledge graph for answering domain-specific medical questions. The data we use are from Electronic Medical Records (EMRs) that are complex and heterogeneous with varied qualities. We introduce doctor-in-the-loop mechanism to design the knowledge graph, which improves the interpretability and the accuracy of our QA system. For every natural-language question, we generate the answer with a hybrid of the traditional deep learning model and the reasoning on the knowledge graph. Finally, the answer is returned to the user in the format of a subgraph with corresponding natural-language sentences. Our system shows high interpretability and gives excellent performance according to the experiment results.

Number of people having expertise in a certain domain is less than people who need information in that domain. In this situation, an automatic question answering (QA) system is necessary. Observing available manual QA sites on internet, the real world question that people usually ask have different expected answer type (EAT) compared to a common automatic QA. Addressing a case study of a religion domain which makes it a closed domain QA, we proposed the EAT into 6 types: LAW, DEFINITION, COMPARISON, METHOD, TIME and PERSON. Different with common QA approach, we built the QA system using case based approach which consists of two main components: Question Analyzer and Case Retriever. Related with the case based reasoning (CBR) framework, these two main components act as the Retrieve and Reuse process while the Revise and Retain process is handle by Case Retainer component. The QA system was built using available Indonesian Natural Language Processing (NLP) tools and FreeCBR as the CBR library. The experiments were done to calculate the accuracy and testing the system with unknown case. By using 77 cases collected from internet with assumption that all answers are available, the experiments achieved 97% accuracy. And by using 10 test cases for the unknown case, the similarity score calculated by the system showed that the test questions have no answer in the available case base.

Previous works on evaluating the performance of Question Answering (QA) systems are focused on the evaluation of the precision. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, using the proposed measures. The main conclusion is that the evaluation of QA systems in realtime can be a new scenario for the evaluation of QA systems.

This paper presents a Retrieval-Augmented Generation (RAG) system for question answering over the set of documents of a public tender. The proof-of-concept chatbot is intended to provide support to officers participating in the evaluation process of a tender whose documents are indexed. The system implements a dual-mode retrieval strategy —supporting both semantic and lexical search— to extract relevant context and uses an instruction-following language model to generate grounded responses based strictly on the retrieved information. The system is designed to assist officers in exploring tenders, evaluation reports, and award justifications through natural language queries. In order to assess the system, the documents of a tender have been generated symtheticly. Qualitative testing demonstrated that the system provided accurate, verifiable answers, avoiding unsupported inferences. The architecture emphasizes document fidelity, interpretability, and usability for non-technical users involved in procurement oversight. Future work may extend the framework to support the identification of fraud-related patterns.

Datasets that boosted state-of-the-art solutions for Question Answering (QA) systems prove that it is possible to ask questions in natural language manner. However, users are still used to query-like systems where they type in keywords to search for answer. In this study we validate which parts of questions are essential for obtaining valid answer. In order to conclude that, we take advantage of LIME - a framework that explains prediction by local approximation. We find that grammar and natural language is disregarded by QA. State-of-the-art model can answer properly even if 'asked' only with a few words with high coefficients calculated with LIME. According to our knowledge, it is the first time that QA model is being explained by LIME.

Recently, there has been an upsurge in the number of knowledge graphs (KG) that can only be accessed by experts. Non-expert users lack an adequate understanding of the queried knowledge graph's vocabulary and structure, as well as the syntax of the structured query language used to express the user's information needs. To increase the user base of these KGs, a set of Question Answering (QA) systems that use natural language to query these knowledge graphs have been introduced. However, finding a benchmark that accurately evaluates the quality of a QA system is a difficult task due to (1) the high degree of variation in the fine-grained properties among the existing benchmarks, (2) the static nature of the existing benchmarks versus the evolving nature of KGs, and (3) the limited number of KGs targeted by existing benchmarks, which hinders the usability of QA systems in real-world deployment over KGs that are different from those that were used in the evaluation of the QA systems. In this paper, we introduce Maestro, a benchmark generation system for question answering over knowledge graphs. Maestro can generate a new benchmark for any KG given the KG and, optionally, a text corpus that covers this KG. The benchmark generated by Maestro is guaranteed to cover all the properties of the natural language questions and queries that were encountered in the literature as long as the targeted KG includes these properties. Maestro also generates high-quality natural language questions with various utterances that are on par with manually-generated ones to better evaluate QA systems.

To address the challenges of extensive domain expertise and deficient semantic comprehension in the digital preservation of ancient ceramics, this paper proposes a knowledge question answering (QA) system integrating Low-Rank Adaptation (LoRA) fine-tuning and Graph Retrieval-Augmented Generation (GraphRAG). First, textual information of ceramic images is generated using the GLM-4V-9B model. These texts are then enriched with domain literature to produce ancient ceramic QA pairs via ERNIE 4.0 Turbo, culminating in a high-quality dataset of 2143 curated question–answer groups after manual refinement. Second, LoRA fine-tuning was employed on the Qwen2.5-7B-Instruct foundation model, significantly enhancing its question-answering proficiency specifically for the ancient ceramics domain. Finally, the GraphRAG framework is integrated, combining the fine-tuned large language model with knowledge graph path analysis to augment multi-hop reasoning capabilities for complex queries. Experimental results demonstrate performance improvements of 24.08% in ROUGE-1, 34.75% in ROUGE-2, 29.78% in ROUGE-L, and 4.52% in BERTScore_F1 over the baseline model. This evidence shows that the synergistic implementation of LoRA fine-tuning and GraphRAG delivers significant performance enhancements for ceramic knowledge systems, establishing a replicable technical framework for intelligent cultural heritage knowledge services.

Question Answering (QA) systems are becoming the inspiring model for the future of search engines. While, recently, datasets underlying QA systems have been promoted from unstructured datasets to structured datasets with semantically highly enriched metadata, question answering systems are still facing serious challenges and are therefore not meeting users' expectations. This paper provides an exhaustive insight of challenges known so far for building QA systems, with a special focus on employing structured data (i.e. knowledge graphs).It thus helps researchers to easily spot gaps to fill with their future research agendas.