Statistical Multiplicity in Systematic Reviews of Anaesthesia Interventions: A Quantification and Comparison between Cochrane and Non-Cochrane Reviews

Georgina LL Imberger · Alexandra Damgaard Vejlby · Sara Bohnstedt Hansen · Ann Merete Møller · Jørn Wetterslev

43 citations0/0 files storedPLoS ONE

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Statistical Methods in Clinical Trials Statistical Methods in Epidemiology

Abstract

BACKGROUND: Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews. METHODS/PRINCIPAL FINDINGS: We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews. CONCLUSION/SIGNIFICANCE: Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 43 citations exist upstream.

References

2008Cochrane Handbook for Systematic Reviews of Interventions 1995Empirical Evidence of Bias 1995Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials 1999Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement 1998Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?2008Assessing Risk of Bias in Included Studies 2001Investigating and dealing with publication and other biases in meta-analysis 2007Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis

Cited by

2012Sistemas de apoio à decisão: mapeamento e análise de conteúdo 2024Research Note: Trial sequential analysis in systematic reviews with meta-analysis 2024Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses – the METSA systematic review 2022Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.