Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular Datasets

• Published in: Proceedings of the ACM on human-computer interaction 2021-10, Vol.5 (CSCW2), p.1-25, Article 448
• Main Authors: Wallace, Shaun, Papoutsaki, Alexandra, Tan, Neilly H., Guo, Hua, Huang, Jeff
• Format: Article
• Language: English
• Subjects: Asynchronous editors; Collaborative and social computing; Collaborative and social computing systems and tools; Collaborative and social computing theory, concepts and paradigms; Computer supported cooperative work; Crowdsourcing; Data cleaning; Data management systems; Data model extensions; Database design and models; Human computer interaction (HCI); Human-centered computing; Incomplete data; Information integration; Information systems; Information systems applications; Web applications; World Wide Web
• ISSN: 2573-0142; 2573-0142
• DOI: 10.1145/3479592

The life cycle of a peer-produced dataset follows the phases of growth, maturity, and decline. Paying crowdworkers is a proven method to collect and organize information into structured tables. However, these tabular representations may contain inaccuracies due to errors or data changing over time. Thus, the maturation phase of a dataset can benefit from the additional human examination. One method to improve accuracy is to recruit additional paid crowdworkers to verify and correct errors. An alternative method relies on unpaid contributors, collectively editing the dataset during regular use. We describe two case studies to examine different strategies for human verification and maintenance of in-flux tabular datasets. The first case study examines traditional micro-task verification strategies with paid crowdworkers, while the second examines long-term maintenance strategies with unpaid contributions from non-crowdworkers. Two paid verification strategies that produced more accurate corrections at a lower cost per accurate correction were redundant data collection followed by final verification from a trusted crowdworker and allowing crowdworkers to review any data freely. In the unpaid maintenance strategies, contributors provided more accurate corrections when asked to review data matching their interests. This research identifies considerations and future approaches to collectively improving information accuracy and longevity of tabular information.