Addressing the "Replication Crisis": Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertai...

Full description

Saved in:
Bibliographic Details
Published in:Multivariate behavioral research 2017-05, Vol.52 (3), p.305-324
Main Authors: Anderson, Samantha F., Maxwell, Scott E.
Format: Article
Language:English
Subjects:
Bias
Computer Simulation
Effect size
Humans
Mathematical analysis
power
Psychology
Psychology - methods
Replication
Reproducibility of Results
Research Design
sample size planning
Software
Statistical power
Statistics as Topic - methods
Uncertainty
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertainty and publication bias in estimated effect sizes, resulting in overly optimistic calculations. A psychologist who intends to obtain power of .80 in the replication study, and performs calculations accordingly, may have an actual power lower than .80. We performed simulations to reveal the magnitude of the difference between actual and intended power based on common sample size planning strategies and assessed the performance of methods that aim to correct for effect size uncertainty and/or bias. Our results imply that even if original studies reflect actual phenomena and were conducted in the absence of questionable research practices, popular approaches to designing replication studies may result in a low success rate, especially if the original study is underpowered. Methods correcting for bias and/or uncertainty generally had higher actual power, but were not a panacea for an underpowered original study. Thus, it becomes imperative that 1) original studies are adequately powered and 2) replication studies are designed with methods that are more likely to yield the intended level of power.
ISSN:0027-3171
1532-7906
DOI:10.1080/00273171.2017.1289361