Measuring Divergent Thinking Originality With Human Raters and Text-Mining Models: A Psychometric Comparison of Methods

• Published in: Psychology of aesthetics, creativity, and the arts creativity, and the arts, 2021-11, Vol.15 (4), p.645-663
• Main Authors: Dumas, Denis, Organisciak, Peter, Doherty, Michael
• Format: Article
• Language: English
• Subjects: Cognition & reasoning; Creativity; Data mining; Divergent Thinking; Experimentation; Female; Human; Male; Psychometrics; Quantitative psychology; Subjectivity; Test Reliability; Verbal Fluency; Websites
• ISSN: 1931-3896; 1931-390X
• DOI: 10.1037/aca0000319

Within creativity research, interest and capability in utilizing text-mining models to quantify the Originality of participant responses to Divergent Thinking tasks has risen sharply over the last decade, with many extant studies fruitfully using such methods to uncover substantive patterns among creativity-relevant constructs. However, no systematic psychometric investigation of the reliability and validity of human-rated Originality scores, and scores from various freely available text-mining systems, exists in the literature. Here we conduct such an investigation with the Alternate Uses Task. We demonstrate that, despite their inherent subjectivity, human-rated Originality scores displayed the highest reliability at both the composite and latent factor levels. However, the text-mining system GloVe 840B was highly capable of approximating human-rated scores both in its measurement properties and its correlations to various creativity-related criteria including ideational Fluency, Elaboration, Openness, Intellect, and self-reported Creative Activities. We conclude that, in conjunction with other salient indicators of creative potential, text-mining models (and especially the GloVe 840B system) are capable of supporting reliable and valid inferences about Divergent Thinking. We offer an open-access module for researchers to apply these methods to their own data via our laboratory website (https://openscoring.du.edu/).