• Home
  • Search
  • Towards a Better Metric for Evaluating Question Generation Systems
  • Cite Icon113
  • https://doi.org/10.18653/v1/d18-1429Copy DOI Icon

Towards a Better Metric for Evaluating Question Generation Systems

  • Jan 1, 2018
  • Preksha Nema +1 more
Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

There has always been criticism for using n-gram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems. However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc. Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task. In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc. In this work, we show that current automatic evaluation metrics based on n-gram similarity do not always correlate well with human judgments about answerability of a question. To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments. The scripts and data developed as a part of this work are made publicly available.

Loading PDF

Similar Papers
  • Research Article
  • Citations4

Comparison of template-based and multilayer perceptron-based approach for automatic question generation system

  • Dec 01, 2022
  • Indonesian Journal of Electrical Engineering and Computer Science
  • Walelign Tewabe Sewunetie +1
  • Research Article
  • Citations80

G-Asks: An Intelligent Automatic Question Generation System for Academic Writing Support

  • Mar 16, 2012
  • Dialogue & Discourse
  • Ming Liu +2
  • Dissertation

Infusing Automatic Question Generation with Natural Language Understanding

  • Dec 01, 2016
  • Karen Mazidi
  • Conference Article
  • Citations6

Thematic Question Generation over Knowledge Bases

  • Dec 01, 2018
  • Tanguy Raynaud +2
  • Supplementary Content
  • Citations114

Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

  • Jan 01, 2023
  • Progress in Artificial Intelligence
  • Nikahat Mulla +1
  • PDF
  • Research Article
  • Citations8

Answer-Aware Question Generation from Tabular and Textual Data using T5

  • Sep 20, 2021
  • International Journal of Emerging Technologies in Learning (iJET)
  • Saichandra Pandraju +1
  • Dissertation

Automatic question generation from natural language texts

  • Jan 01, 2022
  • Zhen Cao
  • Book Chapter
  • Citations9

Automatic Question Generation System for English Reading Comprehension

  • Jan 01, 2020
  • Yin-Chun Fung +4
  • Research Article
  • Citations15

A novel framework for Automatic Chinese Question Generation based on multi-feature neural network model

  • Jan 01, 2018
  • Computer Science and Information Systems
  • Hai-Tao Zheng +3
  • Conference Article
  • Citations21

Automatic Question Generation for Repeated Testing to Improve Student Learning Outcome

  • Jul 01, 2021
  • Danny C.L Tsai +3
  • PDF
  • Conference Article
  • Citations58

Cross-Lingual Training for Automatic Question Generation

  • Jan 01, 2019
  • Vishwajeet Kumar +4
  • PDF
  • Research Article
  • Citations7

AUTOMATIC GENERATION OF ENGLISH LANGUAGE TEST QUESTIONS USING MATHEMATICA

  • Sep 17, 2017
  • CBU International Conference Proceedings
  • Anna Malinova +1
  • Research Article
  • Citations10

An Automatic Question Generation System Using Rule-Based Approach in Bloom’s Taxonomy

  • Aug 30, 2021
  • Recent Advances in Computer Science and Communications
  • Deena Gnanasekaran +2
  • Book Chapter
  • Citations31

Leveraging Multiple Views of Text for Automatic Question Generation

  • Jan 01, 2015
  • Karen Mazidi +1
  • Research Article

Comparative Analysis of Large Language Models for Automated Question Generation from Video-Based Learning Content

  • Jan 01, 2025
  • IEEE Transactions on Artificial Intelligence
  • Yong-Bin Kang +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.