Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well...

Full description

Saved in:
Bibliographic Details
Published in:Journal of research on educational effectiveness 2023-10, Vol.16 (4), p.663-680
Main Authors: Sims, Sam, Anders, Jake, Inglis, Matthew, Lortie-Forgues, Hugues
Format: Article
Language:English
Subjects:
Clinical trials
Educational Research
Effect Size
Intervention
Randomized Controlled Trials
Research Problems
Statistical Analysis
Statistical Bias
Statistical Significance
Type M error
Type S error
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.
ISSN:1934-5747
1934-5739
DOI:10.1080/19345747.2022.2090470