• Home
  • Search
  • Axiomatic analysis of aggregation methods for collective annotation
  • Cite Icon17
  • https://doi.org/10.5555/2615731.2617437Copy DOI Icon

Axiomatic analysis of aggregation methods for collective annotation

  • May 5, 2014
  • Justin Kruger +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Crowdsourcing is an important tool, e.g., in computational linguistics and computer vision, to efficiently label large amounts of data using nonexpert annotators. The individual annotations collected need to be aggregated into a single collective annotation. The hope is that the quality of this collective annotation will be comparable to that of a traditionally sourced expert annotation. In practice, most scientists working with crowdsourcing methods use simple majority voting to aggregate their data, although some have also used probabilistic models and treated aggregation as a problem of maximum likelihood estimation. The observation that the aggregation step in a collective annotation exercise may be considered a problem of social choice has only been made very recently. Following up on this observation, we show that the axiomatic method, as practiced in social choice theory, can make a contribution to this important domain and we develop an axiomatic framework for collective annotation, focusing amongst other things on the notion of an annotator's bias. We complement our theoretical study with a discussion of a crowdsourcing experiment using data from dialogue modelling in computational linguistics.

Similar Papers
  • Book Chapter
  • Citations13

Theory of Social Choice

  • Jan 01, 1998
  • Anatol Rapoport
  • Research Article
  • Citations9

Reference functions and possibility theorems for cardinal social choice problems

  • Dec 13, 1996
  • Social Choice and Welfare
  • John P Conley +2
  • Book Chapter

Recent Developments in Collective Decision Making in Combinatorial Domains

  • Jan 01, 2013
  • Ulle Endriss
  • Book Chapter
  • Citations1

Possibility Theorems, Incentives, and Progress in Economic Theory

  • Jan 01, 1986
  • Barry W Ickes
  • Research Article
  • Citations43

Fuzzy preferences and Arrow-type problems in social choice

  • Apr 01, 1994
  • Social Choice and Welfare
  • Asis Banerjee
  • Research Article
  • Citations2

Social choice problems with fixed sets of alternatives

  • Apr 01, 1991
  • Mathematical Social Sciences
  • Ye Yanovskaya
  • Book Chapter

Individually Rational Strategy-Proof Social Choice with Exogenous Indifference Sets

  • Jan 01, 2016
  • Mingyu Guo +3
  • Research Article
  • Citations90

A Multimedia Recommender System

  • Nov 01, 2013
  • ACM Transactions on Internet Technology
  • Massimiliano Albanese +4
  • Research Article

Book Review: An Anatomy of Values: Problems of Personal and Social Choice

  • May 01, 1971
  • Theological Studies
  • Thomas V Curley
  • Research Article

Abundance and American Democracy: A Comment

  • May 01, 1982
  • The Journal of Politics
  • David W Orr
  • Research Article
  • Citations11

Group choice with lexicographic individual orderings

  • Mar 01, 1973
  • Behavioral Science
  • Prasanta K Pattanaik
  • Research Article
  • Citations59

Assignment Problem Based on Ordinal Preferences

  • Nov 01, 1973
  • Management Science
  • Peter Gardenfors
  • Research Article
  • Citations3

The core of social choice problems with monotone preferences and a feasibility constraint

  • Jun 01, 2004
  • Social Choice and Welfare
  • Adam Meirowitz
  • Research Article
  • Citations64

Arrow's Theorem, Indeterminacy, and Multiplicity Reconsidered

  • Jul 01, 2001
  • Ethics
  • Mathias Risse
  • Research Article
  • Citations4

Guest editorial: special issue on computational social choice

  • Oct 15, 2010
  • Autonomous Agents and Multi-Agent Systems
  • Edith Elkind +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.