Accuracy of Effect Size Estimates From Published Psychological Experiments Involving Multiple Trials

Abstract

The reporting of exaggerated effect size estimates may occur either through researchers accepting statistically significant results when power is inadequate and/or from repeated measures approaches aggregating, averaging multiple items, or multiple trials. Monte-Carlo simulations with input of a small, medium, or large effect size were conducted on multiple items or trials that were either averaged or aggregated to create a single dependent measure. Alpha was set at the .05 level, and the trials were assessed over item or trial correlations ranging from 0 to 1. Simulations showed a large increase in observed effect size averages and the power to accept these estimates as statistically significant increased over numbers of trials or items. Overestimation effects were mitigated as correlations between trials increased but still remained substantial in some cases. The implications of these findings for meta-analyses and different research scenarios are discussed.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 8 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.