Valid Inference in Random Effects Meta-Analysis

• Published in: Biometrics 1999-09, Vol.55 (3), p.732-737
• Main Authors: Follmann, Dean A., Proschan, Michael A.
• Format: Article
• Language: English
• Subjects: Anticholesteremic Agents - therapeutic use; Approximation; Biometry; Cholesterol - blood; Clinical trials; Confidence interval; Confidence Intervals; Control groups; Gaussian distributions; Humans; Hypercholesterolemia - blood; Hypercholesterolemia - drug therapy; Inflation rates; Meta analysis; Meta-Analysis as Topic; Myocardial infarction; Myocardial Ischemia - prevention & control; P values; Permutation test; Permutation tests; Randomization test; Randomized Controlled Trials as Topic - statistics & numerical data; T tests; t-test
• ISSN: 0006-341X; 1541-0420
• DOI: 10.1111/j.0006-341X.1999.00732.x

The standard approach to inference for random effects meta-analysis relies on approximating the null distribution of a test statistic by a standard normal distribution. This approximation is asymptotic on k, the number of studies, and can be substantially in error in medical meta-analyses, which often have only a few studies. This paper proposes permutation and ad hoc methods for testing with the random effects model. Under the group permutation method, we randomly switch the treatment and control group labels in each trial. This idea is similar to using a permutation distribution for a community intervention trial where communities are randomized in pairs. The permutation method theoretically controls the type I error rate for typical meta-analyses scenarios. We also suggest two ad hoc procedures. Our first suggestion is to use a t-reference distribution with k - 1 degrees of freedom rather than a standard normal distribution for the usual random effects test statistic. We also investigate the use of a simple t-statistic on the reported treatment effects.