Effect sizes in cross-cultural research

Comparison* is a cornerstone of cross-cultural research. A standard methodology in conducting these comparisons begins with the selection of measures of psychological constructs that produce quantitative data from two or more cultures or countries. The usual statistical analysis of data from two cultures involves testing a null hypothesis (H0) that conflicts with the research hypothesis either by positing that a correlation between two variables is zero in the population or that there is no difference between the means of two populations. Differences are tested by comparing variance among the culture means relative to the variance within the cultures, typically using t or F tests. When the chance probability of obtaining t or F values is sufficiently low (≤5%), the result is considered statistically significant. The p level represents the probability that a result at least as extreme as the obtained result would occur if the H0 were true. This attained p value primarily indicates the strength of the evidence that the H0 is wrong (but the p value does not by itself indicate sufficiently how wrong H0 is).

Statistical significance does not necessarily reflect differences among people of the different cultures, however. The sole computation of ts or Fs precludes our ability to interpret meaningful differences among people, because p values merely indicate the strength of the evidence against the null hypothesis of no difference between population means. Statistical significance, assuming no Type I error, only reflects some unknown, nonzero difference between the population means. Furthermore, the larger the sample sizes, the easier it is for smaller differences to become statistically significant. Therefore, a statistically significant difference may actually reflect a trivially small difference between population means. Interpretations of cultural differences among people based on “statistically significant” findings may be based on “practically insignificant” differences between means. “Practically insignificant” means that the nonzero difference between culture means is so small that it is of little or no practical significance. A synonymous phrase would be “substantively insignificant.”