Reply to Rouder (2014): Good frequentist properties raise confidence

Established psychological results have been called into question by demonstrations that statistical significance is easy to achieve, even in the absence of an effect. One often-warned-against practice, choosing when to stop the experiment on the basis of the results, is guaranteed to produce signifi...

Full description

Saved in:
Bibliographic Details
Published in:Psychonomic bulletin & review 2014-04, Vol.21 (2), p.309-311
Main Authors: Sanborn, Adam N., Hills, Thomas T., Dougherty, Michael R., Thomas, Rick P., Yu, Erica C., Sprenger, Amber M.
Format: Article
Language:English
Subjects:
Bayes Theorem
Bayesian analysis
Behavioral Science and Psychology
Biological and medical sciences
Cognitive Psychology
Confidence
Data Interpretation, Statistical
Fundamental and applied biological sciences. Psychology
Humans
Hypotheses
Influence
Models, Statistical
Psychology
Psychology. Psychoanalysis. Psychiatry
Psychology. Psychophysiology
Psychometrics. Statistics. Methodology
Research Design - standards
Statistics. Mathematics
Studies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Established psychological results have been called into question by demonstrations that statistical significance is easy to achieve, even in the absence of an effect. One often-warned-against practice, choosing when to stop the experiment on the basis of the results, is guaranteed to produce significant results. In response to these demonstrations, Bayes factors have been proposed as an antidote to this practice, because they are invariant with respect to how an experiment was stopped. Should researchers only care about the resulting Bayes factor, without concern for how it was produced? Yu, Sprenger, Thomas, and Dougherty ( 2014 ) and Sanborn and Hills ( 2014 ) demonstrated that Bayes factors are sometimes strongly influenced by the stopping rules used. However, Rouder ( 2014 ) has provided a compelling demonstration that despite this influence, the evidence supplied by Bayes factors remains correct. Here we address why the ability to influence Bayes factors should still matter to researchers, despite the correctness of the evidence. We argue that good frequentist properties mean that results will more often agree with researchers’ statistical intuitions, and good frequentist properties control the number of studies that will later be refuted. Both help raise confidence in psychological results.
ISSN:1069-9384
1531-5320
DOI:10.3758/s13423-014-0607-4