• Home
  • Search
  • Aggregation of Multiple Judgments for Evaluating Ordered Lists
  • Open Access IconOpen Access
  • Cite Icon9
  • https://doi.org/10.1007/978-3-642-12275-0_17Copy DOI Icon

Aggregation of Multiple Judgments for Evaluating Ordered Lists

  • Jan 1, 2010
  • Hyun Duk Kim +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Many tasks (e.g., search and summarization) result in an ordered list of items. In order to evaluate such an ordered list of items, we need to compare it with an ideal ordered list created by a human expert for the same set of items. To reduce any bias, multiple human experts are often used to create multiple ideal ordered lists. An interesting challenge in such an evaluation method is thus how to aggregate these different ideal lists to compute a single score for an ordered list to be evaluated. In this paper, we propose three new methods for aggregating multiple order judgments to evaluate ordered lists: weighted correlation aggregation, rank-based aggregation, and frequent sequential pattern-based aggregation. Experiment results on ordering sentences for text summarization show that all the three new methods outperform the state of the art average correlation methods in terms of discriminativeness and robustness against noise. Among the three proposed methods, the frequent sequential pattern-based method performs the best due to the flexible modeling of agreements and disagreements among human experts at various levels of granularity.

Similar Papers
  • Research Article
  • Citations45

Retrieval, alignment, and clustering of computational models based on semantic annotations

  • Jan 01, 2011
  • Molecular Systems Biology
  • Marvin Schulz +4
  • Single Report
  • Citations5

Networking Foundations for Collaborative Computing at Internet Scope

  • Jan 01, 2006
  • H-Peter Dommel +1
  • PDF
  • Research Article
  • Citations25

Gender-based homophily in collaborations across a heterogeneous scholarly landscape.

  • Apr 05, 2023
  • PloS one
  • Y Samuel Wang +4
  • Research Article
  • Citations3

Provenance for business events

  • Jan 01, 2014
  • International Journal of Big Data Intelligence
  • Rafat Hammad +1
  • Conference Article
  • Citations113

Three-dimensional cache design exploration using 3DCacti

  • Jan 10, 2017
  • Yuh-Fang Tsai +3
  • PDF
  • Research Article
  • Citations12

Multi-dimensional discovery of biomarker and phenotype complexes

  • Oct 01, 2010
  • BMC Bioinformatics
  • Philip Ro Payne +5
  • PDF
  • Research Article
  • Citations1

An efficient and scalable graph modeling approach for capturing information at different levels in next generation sequencing reads.

  • Sep 01, 2013
  • BMC Bioinformatics
  • Julia D Warnke +1
  • Conference Article
  • Citations1

Scaffolding RAMI4.0-Exploration as Design Support

  • Sep 05, 2018
  • Claudia Kaar +2
  • Research Article
  • Citations14

Response selection from unstructured documents for human-computer conversation systems

  • Dec 02, 2017
  • Knowledge-Based Systems
  • Zhao Yan +5
  • PDF
  • Abstract
  • Citations2

Boundary representation of neural architecture and connectivity

  • Jul 18, 2011
  • BMC Neuroscience
  • Mario Negrello +2
  • Research Article
  • Citations24

An integrated use of IDEFO, IDEF3 and Petri net methods in support of business process modelling

  • Nov 01, 2001
  • Proceedings of the Institution of Mechanical Engineers, Part E: Journal of Process Mechanical Engineering
  • C-H Kim +2
  • Research Article

SEA: Hierarchically searching efficient adapters for pre-trained models.

  • Sep 01, 2026
  • Neural networks : the official journal of the International Neural Network Society
  • Shun Lu +6
  • Book Chapter
  • Citations9

A Quantitative Categorization of Phonemic Dialect Features in Context

  • Jan 01, 2005
  • Naomi Nagy +3
  • Research Article
  • Citations19

Representing network knowledge using provenance-aware formalisms for cyber-situational awareness

  • Jan 01, 2018
  • Procedia Computer Science
  • Leslie F Sikos +5
  • Conference Article
  • Citations20

Kestrel

  • Nov 16, 2009
  • Lance Stout +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.