Power-enhanced funnel plots for meta-analysis: The sunset funnel plot

Abstract

Background and Objectives: The funnel plot is the most widely used diagnostic plot for meta-analysis. Numerous variants exist to visualize small-study effects, heterogeneity, and the sensitivity of the meta-analytic summary estimates to new evidence (Langan, Higgins, Gregory, & Sutton, 2012). What is currently missing is a funnel plot variant which incorporates information on statistical study-level power to detect an effect of interest. To fill this gap, we here introduce the sunset funnel plot, which, in essence, is a power-enhanced funnel plot (Figure 1). Visual funnel plot examinations for small-study effects include checks whether smaller studies in particular (i.e., those with larger standard errors and associated lower analytic power) tend to yield larger effect sizes. When such an association evidently is driven by conventional criteria of statistical significance, then publication bias is considered to be a likely explanation for the phenomenon, and preferred to other causes, such as true heterogeneity or chance alone (Peters, Sutton, Jones, Abrams, & Rushton, 2008). Information on the power of studies can further support such evaluations of potential publication bias. The test for excess significance (Ioannidis & Trikalinos, 2007) is a widely used evidentiality test to check whether there is a higher number of statistically significant studies than expected, considering their power to detect an effect of interest. Such an excess of significant findings indicates bias in the set of studies under consideration. In the same spirit, if an implausible excess of significant, but at the same time underpowered, studies is visible and potentially drives small-study effects in the funnel plot, this can further weaken the credibility of these results and indicate bias. In addition, significant effects observed in low-powered studies more likely are false positive findings (Forstmeier, Wagenmakers, & Parker, 2017). Power can therefore be seen as an indicator for the replicability of research findings. Indeed, for a set of studies, the deviation of (or, gap between) the proportion of actually observed significant studies and twice the median study power has been proposed as the R-index of replicability (Schimmack, 2016). All in all, study-level power is one useful information to assess the credibility and evidentiality of a set of studies potentially included in a meta-analysis. Consequently, a power-enhanced funnel plot is one means to visualize and communicate this information by incorporating information on study-level power in the well-known, classic funnel plot display. Methods: The sunset (power-enhanced) funnel plot assumes normally distributed effect sizes and regards variances of these effect sizes as known. These assumptions are common in the context of meta-analysis and standard effect sizes for meta-analysis are suitable for the sunset funnel plot as well (e.g., Cohen d, Hedges g, log OR, Fisher’s z-transformed r). For a true population effect size δ, the power of a two-sided Wald test with significance level α testing the null hypothesis δ = 0 is given by Power = 1 - Φ(z1-α/2 - δ/SE(d)) + Φ(-z1-α/2 - δ/SE(d)) with Φ the cumulative distribution function of the standard normal distribution, z1-α/2 the 1-α/2 quantile of the standard normal distribution, and SE(d) the standard error of the study effect size d. The sunset (power-enhanced) funnel plot visualizes these power estimates corresponding to specific standard errors on a second y-axis and with color-coded power regions (Figure 1). Color regions range from an alarming dark red for highly underpowered studies to a relaxing dark green for appropriately powered studies to detect the underlying true effect of interest. The color palette used in the graphic display is vividly remindful of a colorful sunset; hence, the denomination sunset funnel plot. Figure 1: Sunset (power-enhanced) funnel plot, using data from a published meta-analysis (Mathie et al., 2017) comparing homeopathic treatment with placebo. 95% confidence contours are shown, with the black vertical reference line marking the observed summary effect (fixed-effect model) used for power analysis. Significance contours at the .05 and .01 levels are indicated through dark shaded areas. Power estimates are computed for a two-tailed test with significance level .05. R code to reproduce the figure: https://osf.io/967bh/?view_only=e659e4eb1cfa46c2bfe4c8ceb622e922. The underlying true population effect size can either be determined theoretically (e.g., by assuming a smallest effect of interest), or empirically, using meta-analytic estimates of the summary effect. For the latter, the fixed-effect model estimator is one natural default choice, giving less weight (and therefore being less sensitive) to small, biased studies, as compared to random-effects meta-analytic modeling. A number of related power-based statistics can be presented alongside the power-enhanced funnel plot and support its evaluation. These include (i) the median power of studies, (ii) the true underlying effect size necessary for achieving certain levels of median power (e.g., 33% or 66%), (iii) the results of the test for excess significance (Ioannidis & Trikalinos, 2007), and (iv) the R-index as measure for the expected replicability of findings (Schimmack, 2016). To create sunset (power-enhanced) funnel plots and to compute statistics related to these, we provide the tailored function viz_sunset in the package metaviz (Kossmeier, Tran, & Voracek, 2018) within the statistical software R (R Core Team, 2018), and a corresponding online application available at https://metaviz.shinyapps.io/sunset/. Results: For the following illustration example, we use data from a recent published meta-analysis on the effect of homeopathic treatment vs. placebo for numerous medical conditions (Mathie et al., 2017). In this systematic review and meta-analysis, bias assessment suggested high risk of bias for the majority of the 54 randomized controlled trials (RCTs) considered for meta-analysis; only three RCTs were judged as reliable evidence. For illustration purposes, we use the totality of these 54 effect sizes. Visual examination of the corresponding funnel plot shows clear small-study effects, such that imprecise, smaller studies (those with larger standard errors) report larger effects in favor of homeopathy than more precise, larger studies (those with smaller standard errors). This association seems to be driven by studies reporting imprecise, but significant estimates, in particular. Incorporating power information in these considerations (with the fixed-effect estimate δ = -0.25 in favor of homeopathy) additionally reveals that a non-trivial, implausible high, and thus worrisome, number of the significant studies evidently are drastically underpowered (with power values lower than 10%) to detect this effect of interest, thus further suggesting bias (Figure 1). Accordingly, there is an excess of significant findings among the primary studies included in this meta-analysis (15 nominally significant studies observed, but, under these circumstances, only 9.45 significant studies expected; p = .047). The median power of this set of primary studies merely amounts to 14.3% (IQR: 11.1-20.6%), and the true effects needed to reach typical (i.e., median) power levels of 33% or 66% would be substantial (absolute δ values of 0.43 or 0.67, respectively). The expected replicability of findings, as quantified with the R-Index, is extremely low (0.8%). Conclusions and Implications: We introduce the sunset (power-enhanced) funnel plot as a new, useful display for the meta-analytic visualization toolbox. First and foremost, the sunset funnel plot allows to incorporate power considerations into classic funnel plot assessments for small-study effects. In the same spirit as testing for an excess of significant findings (Ioannidis & Trikalinos, 2007), the credibility of findings can further be critically examined by checking whether small-study effects are especially driven by an implausible large number of significant, but at the same time underpowered, studies. Second, the display allows to visually explore and communicate the distribution and typical values of study power for an effect of interest. This visualization is not only informative for meta-analyses, but also in the broader context of meta-scientific investigations into the power of studies of whole scientific fields (e.g., Szucs, & Ioannidis, 2017). Third, changes of power values for a set of studies can be visually examined by varying the true underlying effect. This directly corresponds to questions regarding the necessary true effect size, such that the power of individual or typical studies would reach desired levels. Software to create sunset (power-enhanced) funnel plots is provided. References: Forstmeier, W., Wagenmakers, E. J., & Parker, T. H. (2017). Detecting and avoiding likely false‐positive finding: A practical guide. Biological Reviews, 92, 1941-1968. Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. Kossmeier, M., Tran, U. S., & Voracek, M. (2018). metaviz [R software package]. Retrieved from https://github.com/Mkossmeier/metaviz Langan, D., Higgins, J. P., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. Journal of Clinical Epidemiology, 65, 511-519. Mathie, R. T., Ramparsad, N., Legg, L. A., Clausen, J., Moss, S., Davidson, J. R., ... McConnachie, A. (2017). Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: Systematic review and meta-analysis. Systematic Reviews, 6, 63. R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://ww

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 8 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.