Methods for Estimating the Sampling Variance of the Standardized Mean Difference

• Published in: Psychological methods 2023-08, Vol.28 (4), p.895-904
• Main Authors: Suero, Manuel, Botella, Juan, Durán, Juan I.
• Format: Article
• Language: English
• Subjects: Confidence Limits (Statistics); Effect Size (Statistical); Mean; Meta Analysis; Sampling (Experimental); Statistical Estimation; Test Bias
• ISSN: 1082-989X; 1939-1463
• DOI: 10.1037/met0000446

One of the most widely used effect size indices for meta-analysis in psychology is the standardized mean difference (SMD). The most common way to synthesize a set of estimates of the SMD is to weight them by the inverse of their variances. For this, it is necessary to estimate the corresponding sampling variances. Meta-analysts have a formula for obtaining unbiased estimates of sampling variances, but they often use a variety of alternative, simpler methods. The bias and efficiency of five different methods that have been proposed and that are implemented in different computerized calculation tools are compared and assessed. The data from a set of published meta-analyses are also reanalyzed, calculating the combined estimates and their confidence intervals, as well as estimates of the specific, between-studies variance, using the five estimation methods. This test of sensitivity shows that the results of a meta-analysis can change noticeably depending on the method used to estimate the sampling variance of SMD values, especially under a random-effects model. Some practical recommendations are made about how to choose and implement the methods in calculation resources. Translational AbstractMeta-analysis is a methodology to synthesize and analyze the available evidence on a very specific question that usually expresses the relationship between two variables. This relationship is reflected in some index of effect size. In this article we refer to the index called standardized mean difference. For decades, meta-analytical methods and techniques have been consolidated, which are exposed in many scientific and academic sources. In these sources, various methods are usually proposed to obtain estimates of one of the essential elements to carry out a meta-analysis, such as the sampling variance of the effect size estimator. Not all methods and formulas for estimating the sampling variance of the standardized mean difference are equally advisable. In this article, the five most used methods are reviewed and compared, indicating computer programs for meta-analysis in which they are implemented. We show the magnitude of bias they show and their efficiency. We also show its potential impact by fitting random effects meta-analytic models, which are the most popular in psychology. We conclude with some recommendations in this regard, related to the choice of the estimator and the interpretation of the results obtained when applying the different methods.