Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions?

Abstract

Literature-based meta-analysis is a standard technique applied to pool results of individual studies used in medicine and social sciences. It has been criticized for being too malleable to constrain results, averaging incomparable values, lacking a measure of evidence's strength, and problems with a systematic bias of individual studies. We argue against using literature-based meta-analysis of RCTs to assess treatment efficacy and show that therapeutic decisions based on meta-analytic average are not optimal given the full scope of existing evidence. The argument proceeds with discussing examples and analyzing the properties of some standard meta-analytic techniques. First, we demonstrate that meta-analysis can lead to reporting statistically significant results despite the treatment's limited efficacy. Second, we show that meta-analytic confidence intervals are too narrow compared to the variability of treatment outcomes reported by individual studies. Third, we argue that literature-based meta-analysis is not a reliable measurement instrument. Finally, we show that meta-analysis averages out the differences among studies and leads to a loss of information. Despite these problems, literature-based meta-analysis is useful for the assessment of harms. We support two alternative approaches to evidence amalgamation: meta-analysis of individual patient data (IPD) and qualitative review employing mechanistic evidence.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 16 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.