Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks

Standard, well-established cognitive tasks that produce reliable effects in group comparisons also lead to unreliable measurement when assessing individual differences. This reliability paradox has been demonstrated in decision-conflict tasks such as the Simon, Flanker, and Stroop tasks, which measu...

Full description

Saved in:
Bibliographic Details
Published in:Nature communications 2023-04, Vol.14 (1), p.2234-2234, Article 2234
Main Authors: Kucina, Talira, Wells, Lindsay, Lewis, Ian, de Salas, Kristy, Kohl, Amelia, Palmer, Matthew A., Sauer, James D., Matzke, Dora, Aidman, Eugene, Heathcote, Andrew
Format: Article
Language:English
Subjects:
706/689/477
706/689/477/2811
Attention
Bayesian analysis
Calibration
Cognitive ability
Cognitive tasks
Experimental psychology
Humanities and Social Sciences
multidisciplinary
Neuropsychological Tests
Paradoxes
Reaction Time
Reliability
Reproducibility of Results
Science
Science (multidisciplinary)
Stroop Test
Validity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Standard, well-established cognitive tasks that produce reliable effects in group comparisons also lead to unreliable measurement when assessing individual differences. This reliability paradox has been demonstrated in decision-conflict tasks such as the Simon, Flanker, and Stroop tasks, which measure various aspects of cognitive control. We aim to address this paradox by implementing carefully calibrated versions of the standard tests with an additional manipulation to encourage processing of conflicting information, as well as combinations of standard tasks. Over five experiments, we show that a Flanker task and a combined Simon and Stroop task with the additional manipulation produced reliable estimates of individual differences in under 100 trials per task, which improves on the reliability seen in benchmark Flanker, Simon, and Stroop data. We make these tasks freely available and discuss both theoretical and applied implications regarding how the cognitive testing of individual differences is carried out. Typical cognitive tasks can produce robust experimental effects yet not measure individual differences reliably. Here the authors use hierarchical Bayesian analysis to develop and calibrate tasks that can efficiently achieve good reliability.
ISSN:2041-1723
2041-1723
DOI:10.1038/s41467-023-37777-2