Detecting p‐Hacking

We theoretically analyze the problem of testing for p‐hacking based on distributions of p‐values across multiple studies. We provide general results for when such distributions have testable restrictions (are non‐increasing) under the null of no p‐hacking. We find novel additional testable restricti...

Full description

Saved in:
Bibliographic Details
Published in:Econometrica 2022-03, Vol.90 (2), p.887-906
Main Authors: Elliott, Graham, Kudrin, Nikolay, Wüthrich, Kaspar
Format: Article
Language:English
Subjects:
Bias
complete monotonicity
Hacking
Null hypothesis
publication bias
p‐curve
p‐values
Usefulness
Values
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We theoretically analyze the problem of testing for p‐hacking based on distributions of p‐values across multiple studies. We provide general results for when such distributions have testable restrictions (are non‐increasing) under the null of no p‐hacking. We find novel additional testable restrictions for p‐values based on t‐tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of p‐values. These testable restrictions result in more powerful tests for the null hypothesis of no p‐hacking. When there is also publication bias, our tests are joint tests for p‐hacking and publication bias. A reanalysis of two prominent data sets shows the usefulness of our new tests.
ISSN:0012-9682
1468-0262
DOI:10.3982/ECTA18583