Measuring Misinformation in Video Search Platforms: An Audit Study on YouTube

• Published in: Proceedings of the ACM on human-computer interaction 2020-05, Vol.4 (CSCW1), p.1-27, Article 48
• Main Authors: Hussein, Eslam, Juneja, Prerna, Mitra, Tanushree
• Format: Article
• Language: English
• Subjects: Collaborative and social computing; Content ranking; Empirical studies in collaborative and social computing; Empirical studies in HCI; Human computer interaction (HCI); Human-centered computing; Information systems; Personalization; Web crawling; Web search engines; Web searching and information discovery; World Wide Web
• ISSN: 2573-0142; 2573-0142
• DOI: 10.1145/3392854

Search engines are the primary gateways of information. Yet, they do not take into account the credibility of search results. There is a growing concern that YouTube, the second largest search engine and the most popular video-sharing platform, has been promoting and recommending misinformative content for certain search topics. In this study, we audit YouTube to verify those claims. Our audit experiments investigate whether personalization (based on age, gender, geolocation, or watch history) contributes to amplifying misinformation. After shortlisting five popular topics known to contain misinformative content and compiling associated search queries representing them, we conduct two sets of audits-Search-and Watch-misinformative audits. Our audits resulted in a dataset of more than 56K videos compiled to link stance (whether promoting misinformation or not) with the personalization attribute audited. Our videos correspond to three major YouTube components: search results, Up-Next, and Top 5 recommendations. We find that demographics, such as, gender, age, and geolocation do not have a significant effect on amplifying misinformation in returned search results for users with brand new accounts. On the other hand, once a user develops a watch history, these attributes do affect the extent of misinformation recommended to them. Further analyses reveal a filter bubble effect, both in the Top 5 and Up-Next recommendations for all topics, except vaccine controversies; for these topics, watching videos that promote misinformation leads to more misinformative video recommendations. In conclusion, YouTube still has a long way to go to mitigate misinformation on its platform.