• Home
  • Search
  • On the reliability of factoid question answering evaluation
  • Cite Icon8
  • https://doi.org/10.1145/1227850.1227853Copy DOI Icon

On the reliability of factoid question answering evaluation

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

This paper compares some existing evaluation metrics for factoid question answering (QA) from the viewpoint of stability and sensitivity, using the NTCIR-4 QAC2 Japanese factoid QA tasks and the Buckley/Voorhees stability method and Voorhees/Buckley swap method. Our main findings are: (1) For QA evaluation with ranked lists containing up to five answers, the fraction of questions with a correct answer within top 5 (NQcorrect5) and that with a correct answer at rank 1 (NQcorrect1) are not as stable and sensitive as reciprocal rank. (2) Q-measure, which can handle multiple correct answers and answer correctness levels, is at least as stable and sensitive as reciprocal rank, provided that a mild gain value assignment is used. Emphasizing answer correctness levels tends to hurt stability and sensitivity, while handling multiple correct answers improves them. As our experimental methods are language-independent, we believe that these findings apply to QA in languages other than Japanese as well.

Similar Papers
  • PDF
  • Conference Article
  • Citations129

Neural Domain Adaptation for Biomedical Question Answering

  • Jan 01, 2017
  • Georg Wiese +2
  • Research Article

PlainQAFact: Retrieval-augmented factual consistency evaluation metric for biomedical plain language summarization.

  • Jun 01, 2026
  • Journal of biomedical informatics
  • Zhiwen You +1
  • Conference Article
  • Citations6

A Novel Hierarchical Convolutional Neural Network for Question Answering over Paragraphs

  • Dec 01, 2015
  • Suncong Zheng +5
  • Research Article
  • Citations10

A Novel Evaluation Metric Based on Dispersion of Wear Distance for In Situ Tool Condition Monitoring

  • Jan 01, 2023
  • IEEE Transactions on Instrumentation and Measurement
  • Zhichao You +5
  • Research Article

Automated Taxonomy Induction and its Applications

  • Jan 01, 2017
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne)
  • Amit Gupta
  • Research Article
  • Citations7

Evaluation metrics for video captioning: A survey

  • Aug 11, 2023
  • Machine Learning with Applications
  • Andrei De Souza Inácio +1
  • Conference Article
  • Citations2

Small Segment Emphasized Performance Evaluation Metric for Medical Images

  • Jul 01, 2020
  • R Ammu +1
  • Research Article

A Novel Scholar's Evaluation from the Perspective of Academic Equilibrium Value: A Case Study of Computer Science

  • Dec 01, 2025
  • Wuhan University Journal of Natural Sciences
  • Zonghuang Xu +3
  • Research Article

A Large-Scale Empirical Evaluation of LLMs for Automated Self-Admitted Technical Debt Repayment

  • Feb 10, 2026
  • ACM Transactions on Software Engineering and Methodology
  • Mohammad Sadegh Sheikhaei +3
  • Research Article
  • Citations119

A Literature Survey and Experimental Evaluation of the State-of-the-Art in Uplift Modeling: A Stepping Stone Toward the Development of Prescriptive Analytics.

  • Mar 01, 2018
  • Big Data
  • Floris Devriendt +2
  • Research Article
  • Citations7

A survey on non-factoid question answering systems

  • Jul 10, 2021
  • International Journal of Computers and Applications
  • Manvi Breja +1
  • Conference Article
  • Citations13

Mitigating Noisy Inputs for Question Answering

  • Sep 15, 2019
  • Denis Peskov +4
  • Research Article
  • Citations20

A study of the evaluation metrics for generative images containing combinational creativity

  • Jan 01, 2023
  • Artificial Intelligence for Engineering Design, Analysis and Manufacturing
  • Boheng Wang +5
  • Conference Article
  • Citations27

From text question-answering to multimedia QA on web-scale media resources

  • Oct 23, 2009
  • Tat-Seng Chua +3
  • Conference Article
  • Citations3

Evaluating the Performance of ChatGPT in Accounting and Auditing Exams: An Experimental Study in North Macedonia

  • Dec 15, 2023
  • Economic and Business Trends Shaping the Future
  • Atanasko Atanasovski +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.