Experiments with More Than One Random Factor: Designs, Analytic Models, and Statistical Power

• Published in: Annual review of psychology 2017-01, Vol.68 (1), p.601-625
• Main Authors: Judd, Charles M, Westfall, Jacob, Kenny, David A
• Format: Article
• Language: English
• Subjects: Data analysis; Design of experiments; effect size; experimental design; Experimental psychology; Humans; Mathematical models; mixed models; Models, Statistical; Psychology; random factors; Research Design; Research methodology; sample size; statistical power
• ISSN: 0066-4308; 1545-2085
• DOI: 10.1146/annurev-psych-122414-033702

Traditional methods of analyzing data from psychological experiments are based on the assumption that there is a single random factor (normally participants) to which generalization is sought. However, many studies involve at least two random factors (e.g., participants and the targets to which they respond, such as words, pictures, or individuals). The application of traditional analytic methods to the data from such studies can result in serious bias in testing experimental effects. In this review, we develop a comprehensive typology of designs involving two random factors, which may be either crossed or nested, and one fixed factor, condition. We present appropriate linear mixed models for all designs and develop effect size measures. We provide the tools for power estimation for all designs. We then discuss issues of design choice, highlighting power and feasibility considerations. Our goal is to encourage appropriate analytic methods that produce replicable results for studies involving new samples of both participants and targets.