Null Hypothesis Significance Testing Defended and Calibrated by Bayesian Model Checking

Significance testing is often criticized because p-values can be low even though posterior probabilities of the null hypothesis are not low according to some Bayesian models. Those models, however, would assign low prior probabilities to the observation that the p-value is sufficiently low. That con...

Full description

Saved in:
Bibliographic Details
Published in:The American statistician 2021-07, Vol.75 (3), p.249-255
Main Author: Bickel, David R.
Format: Article
Language:English
Subjects:
Bayesian analysis
Calibration
Conditional probability
Hypotheses
Hypothesis testing
Lower bounds
Model checking
Model testing
Null hypothesis
Objective Bayes factor
p-Value calibration
Probability
Regression analysis
Relative belief ratio
Reproducibility
Reproducibility crisis
Statistical methods
Statistics
Upper bounds
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Significance testing is often criticized because p-values can be low even though posterior probabilities of the null hypothesis are not low according to some Bayesian models. Those models, however, would assign low prior probabilities to the observation that the p-value is sufficiently low. That conflict between the models and the data may indicate that the models needs revision. Indeed, if the p-value is sufficiently small while the posterior probability according to a model is insufficiently small, then the model will fail a model check. That result leads to a way to calibrate a p-value by transforming it into an upper bound on the posterior probability of the null hypothesis (conditional on rejection) for any model that would pass the check. The calibration may be calculated from a prior probability of the null hypothesis and the stringency of the check without more detailed modeling. An upper bound, as opposed to a lower bound, can justify concluding that the null hypothesis has a low posterior probability.
ISSN:0003-1305
1537-2731
DOI:10.1080/00031305.2019.1699443