The secret lives of experiments: Methods reporting in the fMRI literature

• Published in: NeuroImage (Orlando, Fla.) Fla.), 2012-10, Vol.63 (1), p.289-300
• Main Author: Carp, Joshua
• Format: Article
• Language: English
• Subjects: Analysis methods; Bias; Biomedical Research - statistics & numerical data; Brain; Brain Mapping - statistics & numerical data; Brain research; Experimental design; fMRI; Humans; Literature Based Discovery - methods; Magnetic Resonance Imaging - statistics & numerical data; Medical research; Methods reporting; Neurosciences; NMR; Nuclear magnetic resonance; Periodicals as Topic - statistics & numerical data; Reproducibility; Reproducibility of Results; Science; Sensitivity and Specificity; Statistical power; Studies
• ISSN: 1053-8119; 1095-9572
• DOI: 10.1016/j.neuroimage.2012.07.004

Replication of research findings is critical to the progress of scientific understanding. Accordingly, most scientific journals require authors to report experimental procedures in sufficient detail for independent researchers to replicate their work. To what extent do research reports in the functional neuroimaging literature live up to this standard? The present study evaluated methods reporting and methodological choices across 241 recent fMRI articles. Many studies did not report critical methodological details with regard to experimental design, data acquisition, and analysis. Further, many studies were underpowered to detect any but the largest statistical effects. Finally, data collection and analysis methods were highly flexible across studies, with nearly as many unique analysis pipelines as there were studies in the sample. Because the rate of false positive results is thought to increase with the flexibility of experimental designs, the field of functional neuroimaging may be particularly vulnerable to false positives. In sum, the present study documented significant gaps in methods reporting among fMRI studies. Improved methodological descriptions in research reports would yield significant benefits for the field. ► This study evaluated methods reporting practices across 241 recent fMRI articles. ► Few studies reported sufficient methodological detail for independent replication. ► Methods were highly variable across studies, increasing the risk of false positives. ► Widespread adoption of reporting guidelines would improve fMRI research.