The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication.

Abstract

Repeated investigations of the same phenomenon typically yield effect sizes that vary more than one would expect from sampling error alone. Such variation is even found in exact replication studies, suggesting that it is not only because of identifiable moderators but also to subtler random variation across studies. Such heterogeneity of effect sizes is typically ignored, with unfortunate consequences. We consider its implications for power analyses, the precision of estimated effects, and the planning of original and replication research. With heterogeneity and an interest in generalizing to a population of studies, the usual power calculations and confidence intervals are likely misleading, and the preference for single definitive large-N studies is misguided. Researchers and methodologists need to recognize that effects are often heterogeneous and plan accordingly. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 156 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.