When Experts Disagree: Response Aggregation and its Consequences in Expert Surveys

Political scientists use expert surveys to assess the latent features of political actors. Experts, though, are unlikely to be equally informed and assess all actors equally well. The literature acknowledges variance in measurement quality but pays little attention to the implications of uncertainty...

Full description

Saved in:
Bibliographic Details
Published in:Political science research and methods 2020-07, Vol.8 (3), p.580-588
Main Authors: Lindstädt, René, Proksch, Sven-Oliver, Slapin, Jonathan B.
Format: Article
Language:English
Subjects:
Accession
Bias
Economic policy
Experts
Noise
Political parties
Political scientists
Polls & surveys
Public opinion surveys
Research Notes
Researchers
Simulation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Political scientists use expert surveys to assess the latent features of political actors. Experts, though, are unlikely to be equally informed and assess all actors equally well. The literature acknowledges variance in measurement quality but pays little attention to the implications of uncertainty for aggregating responses. We discuss the nature of the measurement problem in expert surveys. We then propose methods to assess the ability of experts to judge where actors stand and to aggregate expert responses. We examine the effects of aggregation for a prominent survey in the literature on party politics and EU integration. Using a Monte Carlo simulation, we demonstrate that it is better to aggregate expert responses using the median or modal response, rather than the mean.
ISSN:2049-8470
2049-8489
DOI:10.1017/psrm.2018.52