Type I Error Inflation in the Traditional By-Participant Analysis to Metamemory Accuracy: A Generalized Mixed-Effects Model Perspective

• Published in: Journal of experimental psychology. Learning, memory, and cognition memory, and cognition, 2014-09, Vol.40 (5), p.1287-1306
• Main Authors: Murayama, Kou, Sakaki, Michiko, Yan, Veronica X., Smith, Garry M.
• Format: Article
• Language: English
• Subjects: Accuracy; African Languages; Association (Psychology); Bias; Biological and medical sciences; Cognition & reasoning; Cognition - physiology; Comparative Analysis; Computer Simulation; Correlation; Cues; English; Equations (Mathematics); Error Patterns; Evaluation Methods; Experimental Psychology; Fundamental and applied biological sciences. Psychology; Hierarchical Linear Modeling; Human; Humans; Inferences; Inflation (Economic); Judgment; Judgment - physiology; Learning. Memory; Memory; Memory - physiology; Metacognition; Models, Psychological; Monte Carlo Methods; Perception; Prediction; Psychology; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Recall (Psychology); Regression (Statistics); Signal Detection (Perception); Signal Detection, Psychological - physiology; Simulation; Statistical Analysis; Theories; Visual Stimuli; Word Lists
• ISSN: 0278-7393; 1939-1285
• DOI: 10.1037/a0036914

In order to examine metacognitive accuracy (i.e., the relationship between metacognitive judgment and memory performance), researchers often rely on by-participant analysis, where metacognitive accuracy (e.g., resolution, as measured by the gamma coefficient or signal detection measures) is computed for each participant and the computed values are entered into group-level statistical tests such as the t test. In the current work, we argue that the by-participant analysis, regardless of the accuracy measurements used, would produce a substantial inflation of Type I error rates when a random item effect is present. A mixed-effects model is proposed as a way to effectively address the issue, and our simulation studies examining Type I error rates indeed showed superior performance of mixed-effects model analysis as compared to the conventional by-participant analysis. We also present real data applications to illustrate further strengths of mixed-effects model analysis. Our findings imply that caution is needed when using the by-participant analysis, and recommend the mixed-effects model analysis.