• Home
  • Search
  • Crowdsourcing Human Annotation on Web Page Structure
  • Cite Icon21
  • https://doi.org/10.1145/2870649Copy DOI Icon

Crowdsourcing Human Annotation on Web Page Structure

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Parsing the semantic structure of a web page is a key component of web information extraction. Successful extraction algorithms usually require large-scale training and evaluation datasets, which are difficult to acquire. Recently, crowdsourcing has proven to be an effective method of collecting large-scale training data in domains that do not require much domain knowledge. For more complex domains, researchers have proposed sophisticated quality control mechanisms to replicate tasks in parallel or sequential ways and then aggregate responses from multiple workers. Conventional annotation integration methods often put more trust in the workers with high historical performance; thus, they are called performance-based methods. Recently, Rzeszotarski and Kittur have demonstrated that behavioral features are also highly correlated with annotation quality in several crowdsourcing applications. In this article, we present a new crowdsourcing system, called Wernicke, to provide annotations for web information extraction. Wernicke collects a wide set of behavioral features and, based on these features, predicts annotation quality for a challenging task domain: annotating web page structure. We evaluate the effectiveness of quality control using behavioral features through a case study where 32 workers annotate 200 Q&A web pages from five popular websites. In doing so, we discover several things: (1) Many behavioral features are significant predictors for crowdsourcing quality. (2) The behavioral-feature-based method outperforms performance-based methods in recall prediction, while performing equally with precision prediction. In addition, using behavioral features is less vulnerable to the cold-start problem, and the corresponding prediction model is more generalizable for predicting recall than precision for cross-website quality analysis. (3) One can effectively combine workers’ behavioral information and historical performance information to further reduce prediction errors.

Similar Papers
  • Research Article
  • Citations39

Local Feature Weighting in Nearest Prototype Classification

  • Jan 01, 2008
  • IEEE Transactions on Neural Networks
  • F Fernandez +1
  • Conference Article
  • Citations2

Max margin learning on domain-independent web information extraction

  • Oct 24, 2011
  • Bin Zhao +2
  • PDF
  • Research Article
  • Citations4

Online Task Scheduling With Workers Variabilities in Crowdsourcing

  • Jan 01, 2021
  • IEEE Access
  • Qi Li +1
  • Conference Article
  • Citations25

Autism detection based on eye movement sequences on the web

  • Apr 20, 2020
  • Sukru Eraslan +3
  • Research Article
  • Citations54

C 3H 8 separation from CH 4 and H 2 using a synthesized PDMS membrane: Experimental and neural network modeling

  • Sep 11, 2009
  • Journal of Membrane Science
  • Mazdak Shokrian +2
  • Research Article

Learning Local Reconstruction Errors for Face Forgery Detection

  • Jan 01, 2024
  • International Journal of Advanced Computer Science and Applications
  • Haoyu Wu +2
  • Conference Article
  • Citations28

Web Information Extraction by HTML Tree Edit Distance Matching

  • Nov 01, 2007
  • Yeonjung Kim +3
  • Research Article
  • Citations10

Addressing Challenges in Web Accessibility for the Blind and Visually Impaired

  • Oct 01, 2011
  • International Journal of Distance Education Technologies
  • Angela Guercio +3
  • Research Article
  • Citations4

Addressing Challenges in Web Accessibility for the Blind and Visually Impaired

  • Jan 01, 2011
  • International Journal of Distance Education Technologies
  • Angela Guercio +3
  • Book Chapter
  • Citations6

A Dynamic Rearrangement Mechanism of Web Page Layouts Using Web Agents

  • Jan 01, 2009
  • Masato Nakamura +3
  • Research Article

A Survey on Improving the Web Search Ranking by User Behavior Information

  • Jan 01, 2010
  • SSRN Electronic Journal
  • Mohamed Husain +3
  • Conference Article
  • Citations16

Web Page Segmentation Revisited

  • Oct 19, 2020
  • Johannes Kiesel +5
  • PDF
  • Addendum
  • Citations2

Correction: IHWC: intelligent hidden web crawler for harvesting data in urban domains

  • Aug 10, 2022
  • Complex & Intelligent Systems
  • Sawroop Kaur +3
  • Conference Article

Transforming Mental Health Assessment: Machine Learning for Early Detection and Personalized Care Among College Students

  • Jan 01, 2025
  • AHFE international
  • Madhav Theeng Tamang +2
  • Research Article
  • Citations68

Analysis and evaluation of query reformulations in different task types

  • Nov 01, 2010
  • Proceedings of the American Society for Information Science and Technology
  • Chang Liu +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.