Loading…
Some Properties of p-Curves, With an Application to Gradual Publication Bias
Abstract p-curves provide a useful window for peeking into the file drawer in a way that might reveal p-hacking (Simonsohn, Nelson, & Simmons, 2014a). The properties of p-curves are commonly investigated by computer simulations. On the basis of these simulations, it has been proposed that the sk...
Saved in:
Published in: | Psychological methods 2018-09, Vol.23 (3), p.546-560 |
---|---|
Main Authors: | , |
Format: | Article |
Language: | English |
Subjects: | |
Citations: | Items that cite this one |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Summary: | Abstract
p-curves provide a useful window for peeking into the file drawer in a way that might reveal p-hacking (Simonsohn, Nelson, & Simmons, 2014a). The properties of p-curves are commonly investigated by computer simulations. On the basis of these simulations, it has been proposed that the skewness of this curve can be used as a diagnostic tool to decide whether the significant p values within a certain domain of research suggest the presence of p-hacking or actually demonstrate that there is a true effect. Here we introduce a rigorous mathematical approach that allows the properties of p-curves to be examined without simulations. This approach allows the computation of a p-curve for any statistic whose sampling distribution is known and thereby allows a thorough evaluation of its properties. For example, it shows under which conditions p-curves would exhibit the shape of a monotone decreasing function. In addition, we used weighted distribution functions to analyze how 2 different types of publication bias (i.e., cliff effects and gradual publication bias) influence the shapes of p-curves. The results of 2 survey experiments with more than 1,000 participants support the existence of a cliff effect at p = .05 and also suggest that researchers tend to be more likely to recommend submission of an article as the level of statistical significance increases beyond this p level. This gradual bias produces right-skewed p-curves mimicking the existence of real effects even when no such effects are actually present.
Translational Abstract
Statistical tests are a major tool in empirical sciences. Specifically, these tests are used to assess whether certain effects are real or merely reflect random noise. The outcome of such tests is the so-called p value. When a study results in a very small p value, the data are interpreted as suggesting that the observed effect is real. p values are also important because they provide a basis for meta-analysis, which is a summary of many published studies on a particular topic. We analytically derive the statistical distribution-the p-curve-of p values for various statistical tests and for various effect sizes. In agreement with previous research using computer simulations, our analysis shows that the p-curves available for meta-analysis would typically exhibit certain distinctive shapes when the studies on a topic produce real effects. Contrary to the prevailing opinion, however, we show that a variety of other shapes are also |
---|---|
ISSN: | 1082-989X 1939-1463 |
DOI: | 10.1037/met0000125 |