When may systematic reviews and meta-analyses be considered reliable?

Abstract

This Invited Commentary accompanies the following original article: Stevanovic A, Rossaint R, Fritz HG, et al. Airway reactions and emergence times in general laryngeal mask airway anaesthesia. A meta-analysis. Eur J Anaesthesiol 2015; 32:106–116. In a systematic review in this issue of the European Journal of Anaesthesiology, Stevanovic et al.1 compare the event rate of upper airway adverse events and recovery after desflurane in comparison with other volatile anaesthetics and propofol in patients undergoing general surgery with a laryngeal mask airway (LMA). The authors found desflurane comparable with other agents with regard to upper airway reactions, but desflurane appeared to offer a significantly faster emergence from general anaesthesia. Furthermore, the authors advocate the need for large multicentre trials because large inter-study variations were detected. In light of such uncertainty, does inclusion of less than 600 patients in most of the primary and secondary outcome statistics across 13 trials in this systematic review justify drawing firm conclusions or even partially answer the clinical questions of interest? To answer the above questions, one should look beyond the numbers and reflect on some fundamental issues. Systematic reviews and meta-analyses serve the important purpose of identifying potential benefits or harms of an intervention and ideally provide the answers in a rapid and reliable manner. They are often the cornerstone of clinical practice guidelines and may serve as a justification for funding of further trials.2 Authors of systematic reviews are obliged to consider the limitations of the included trials in relation to their design, conduct, analysis and presentation of data. Bias in ‘quality’ assessment of trials included in meta-analyses often leads to result discrepancies between different systematic reviews examining identical questions and between meta-analyses and subsequent large trials.3 This is often due to inclusion of biased results from randomised controlled trials (RCTs). Quality assessment of RCTs can be carried out in many ways such as the use of scales and checklists that assess the trials and provide a score.4 But because most of these scales have methodological shortcomings and often lack transparency, the Cochrane collaboration discourages the use of such scales/scores and recommends the use of the Cochrane risk of bias assessment tool to allow readers to decide on whether they disagree with the authors on the judgements made.5,6 The latter strategy requires a risk of bias judgement by incorporating trial conduct details such as random sequence generation, allocation concealment, blinding of personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and other bias.7,8 Over the years, the quality of published systematic reviews has remained at best questionable despite the introduction of QUOROM (QUality Of Reporting Of Meta-analyses) and the PRISMA (Preferred Reporting Items of Systematic reviews and Meta-Analyses) Statements.9,10 The leading medical journals are partly responsible for the lack of adherence to these quality requirements because they often fail to inform authors and peer-reviewers of their responsibilities and fail to demand adherence to a research-quality-of-reporting statement in the manuscript.11 This remains the case despite ample evidence to support the fact that introduction and adherence to QUOROM and PRISMA statements have improved the quality of systematic reviews in medical journals.12,13 As a consequence, the validity and contribution of published systematic reviews to clinicians is rightly being put to question by many.14 Interpretation of evidence in case of scarce available data is challenging. Inclusion of a small number of trials or few patients in meta-analysis leads to an increased risk of random error that ultimately may lead to false conclusions.15–17 This uncertainty increases further as more statistical tests are carried out either due to a large number of primary and secondary outcomes within a systematic review or due to emergence of additional data. The latter is often referred to as ‘multiplicity due to multiple and repeated significance testing’.18,19 As systematic reviews and meta-analyses should be subject to updates and thus significance testing, the risk of type I error increases. The actual risk of type 1 error may be as high as 10 to 30%.20,21 Meta-analyses often provide evidence of substantial heterogeneity across included trials (e.g. methods, populations, interventions, duration of intervention) and issues such as between-trial variation may further reduce the precision of their findings.22,23 Computation of statistical power (sample size calculation) is essential in order to carry out a randomised clinical trial based on an expected event rate in the control group with adjustment by an anticipated relative risk reduction in the intervention group and with maximum risks of type I and II errors. Despite a general misperception that systematic reviews and meta-analyses provide high statistical power, the contrary is often the case.24 Furthermore, inclusion of studies without adequate power (small sample sizes) in meta-analyses may act paradoxically and not only contribute very little information but also reduce the overall precision and power of meta-analyses, and as a consequence perhaps even delay decisions on the benefits or harms of an intervention.24,25 Thus, extrapolation of similar thinking from RCTs to systematic reviews translates into application of measures to calculate appropriate and adequate power in order to be able to accept or refute positive findings or nonsignificant results. Inadequate power (small number of patients) and small event rates in meta-analysis tend to overestimate intervention effect estimates not only because of bias but equally importantly due to risk of random error and lack of precision.22 Indeed, evidence suggests that the majority of large treatment effects emerge from small studies, and as additional trials are conducted, these effect sizes are often substantially reduced.26 The required information size in meta-analysis (number of participants) in order to provide a reliable and conclusive estimate can be calculated by application of sequential methods such as trial sequential analysis (TSA).17,18,23 Sequential methods may help to provide information on when firm evidence is reached in a cumulative meta-analysis. In addition, they provide information on a lack of presence of clinical effect in a meta-analysis as early as possible. As a consequence, large numbers of apparently statistically conclusive meta-analyses may become inconclusive (false positives or false negative) due to lack of power.18,21,28 This problem is not to be underestimated because the estimated proportion of false-positive meta-analyses has been reported to be as high as 16 to 37%, which is an alarmingly large figure and has a severe impact on our clinical decision-making and for treatment of our patients.29 When addressing the level of statistical significance, the use of P values and unadjusted 95% confidence intervals (CIs) is generally the norm and the assumption seems to be that a meta-analysis in itself provides all the necessary information and the required number of randomised patients. However, this may be a naive and dangerous presumption because P values and CI are closely related and their interpretation may require estimates of how much information is needed. As in the case with interim-analyses in RCTs, statistics in meta-analyses also need adjustment. Sequential methods such as TSA increase the validity and reliability of CIs when sequentially adjusted.30 The authors of the systematic review in this issue of EJA should be commended on summarising the evidence of the effect of desflurane compared with other anaesthetics on adverse airway events.1 However, we miss an evaluation of the information required for the meta-analysis to be conclusive. Given that all included trials were of low risk of bias, a required information size of 2220 randomised patients would have been necessary to detect or reject a 40% relative risk reduction, equalling an NNT of 23, with the usual choice of type 1 and 2 errors and a diversity of 41%. At present, the meta-analysis of the primary outcome of adverse airway events is merely an interim analysis in this process and the TSA-adjusted 95% CI is 0.32 to 3.92, allowing for both considerable harm and benefit. Further, as only two trials described allocation sequence generation and allocation concealment, all the trials were certainly not free of bias risk and it would have been interesting to estimate the possible bias effect comparing trials with the lowest risk of bias to trials with a high risk of bias. These considerations suggest that the authors are indeed correct in the quest for a new large trial with a low risk of bias, but the authors could have substantiated their demands of how much information was actually gathered in relation to the needed information and how much the risk of bias may have influenced the meta-analysis results. We believe that it is time to move beyond merely reflecting on unadjusted P values and CI in meta-analyses as if all the necessary information has been reached, and promote the application of sequential methods in systematic reviews adopting an ‘interim analysis’ perspective on meta-analysis. We strongly urge journals, editors and peer-reviewers to exert more attention to quality indicators and demand adherence to the PRISMA guideline and risk of bias assessment in all systematic reviews as a mandatory requirement prior to publication of systematic reviews. Uncited reference 27 Acknowledgements relating to this article Assistance with the Commentary: none. Financial support or sponsorship: none. Conflicts of interest: none. Comment from the Editor: this Invited Commentary was accepted by the editors but was not sent for external peer review.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 44 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.