p-Hacking by Post Hoc Selection With Multiple Opportunities: Detectability by Skewness Test?: Comment on Simonsohn, Nelson, and Simmons (2014)

Simonsohn, Nelson, and Simmons (2014) have suggested a novel test to detect p-hacking in research, that is, when researchers report excessive rates of "significant effects" that are truly false positives. Although this test is very useful for identifying true effects in some cases, it fail...

Full description

Saved in:
Bibliographic Details
Published in:Journal of experimental psychology. General 2015-12, Vol.144 (6), p.1137-1145
Main Authors: Ulrich, Rolf, Miller, Jeff
Format: Article
Language:English
Subjects:
Data Interpretation, Statistical
Effects
Experimentation
Humans
Research Design - statistics & numerical data
Researchers
Statistical Significance
Statistical Tests
Statistics
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Simonsohn, Nelson, and Simmons (2014) have suggested a novel test to detect p-hacking in research, that is, when researchers report excessive rates of "significant effects" that are truly false positives. Although this test is very useful for identifying true effects in some cases, it fails to identify false positives in several situations when researchers conduct multiple statistical tests (e.g., reporting the most significant result). In these cases, p-curves are right-skewed, thereby mimicking the existence of real effects even if no effect is actually present.
ISSN:0096-3445
1939-2222
DOI:10.1037/xge0000086