Cautionary note on calculating standardized effect size (SES) in randomization test

• Published in: Community ecology 2018-06, Vol.19 (1), p.77-83
• Main Author: Botta-Dukát, Z.
• Format: Article
• Language: English
• Subjects: Animal Ecology; Biodiversity; Community & Population Ecology; Life Sciences; Marine & Freshwater Sciences; Microbial Ecology; Plant Ecology
• ISSN: 1585-8553; 1588-2756
• DOI: 10.1556/168.2018.19.1.8

In community ecology, randomization tests with problem specific test statistics (e.g., nestedness, functional diversity, etc.) are often applied. Researchers in such studies may want not only to detect the significant departure from randomness, but also to measure the effect size (i.e., the magnitude of this departure). Measuring the effect size is necessary, for instance, when the roles of different assembly forces (e.g., environmental filtering, competition) are compared among sites. The standard method is to calculate standardized effect size (SES), i.e., to compute the departure from the mean of random communities divided by their standard deviations. Standardized effect size is a useful measure if the test statistic (e.g., nestedness index, phylogenetic or functional diversity) in the random communities follows a symmetric distribution. In this paper, I would like to call attention to the fact that SES may give us misleading information if the distribution is asymmetric (skewed). For symmetric distribution median and mean values are equal (i.e., SES = 0 indicates p = 0.5). However, this condition does not hold for skewed distributions. For symmetric distributions departure from the mean shows the extremity of the value, regardless of the sign of departure, while in asymmetric distributions the same deviation can be highly probable and extremely improbable, depending on its sign. To avoid these problems, I recommend checking symmetry of null-distribution before calculating the SES value. If the distribution is skewed, I recommend either log-transformation of the test statistic, or using probit-transformed p-value as effect size measure.