I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items

• Published in: Journal of research in personality 2019-10, Vol.82, p.103849, Article 103849
• Main Authors: Curran, Paul G., Hauser, Kelsey A.
• Format: Article
• Language: English
• Subjects: Carelessness; Data cleaning; Insufficient effort responding; Self-report data; Verbal protocol
• ISSN: 0092-6566; 1095-7251
• DOI: 10.1016/j.jrp.2019.103849

•Carelessness in self-report data can be detected with many methods.•Embedding items in a scale with presumed ‘correct’ responses is one of these.•Properties of these items can impact their usefulness.•Individuals can provide valid justification for ‘incorrect’ responses.•Researchers should know their items, and know the risk of not knowing those items. Participant carelessness is a source of invalidity in psychological data (Huang, Liu, & Bowling, 2015), and many methods have been created to screen for this carelessness (Curran, 2016; Johnson, 2005). These include items that researchers presume thoughtful individuals will answer in a given way (e.g., disagreement with “I am paid biweekly by leprechauns”, Meade & Craig, 2012). This paper reports on two samples in which individuals spoke aloud a series of these questions, and found that (a) individuals do occasionally report valid justifications for presumed invalid responses, (b) there is relatively high variance in this behavior over different items, and (c) items developed for this specific purpose tend to work better than those drawn from other sources or created ad-hoc.