- Research Article
52
- 10.1093/pan/mpt050
Extracting Wisdom from Experts and Small Crowds: Strategies for Improving Informant-based Measures of Political Concepts
- Jan 01, 2014
- Political Analysis
- Cherie D Maestas + 2 more +2
Social scientists have increasingly turned to expert judgments to generate data for difficult-to-measure concepts, but getting access to and response from highly expert informants can be costly and challenging. We examine how informant selection and post-survey response aggregation influence the validity and reliability of measures built from informant observations. We draw upon three surveys with parallel survey questions of candidate characteristics to examine the trade-off between expanding the size of the local informant pool and the pool's level of expertise. We find that a “wisdom-of-crowds” effect trumps the benefits associated with the expertise of individual informants when the size of the rater pool is modestly increased. We demonstrate that the benefits of expertise are best realized by prescreening potential informants for expertise rather than post-survey weighting by expertise.
Read more