Trials and tribulations of a meta‐analyst

Abstract

He said. She said. They said. And then they said again. By its very definition, science is a self-correcting process. Scientific breakthroughs are not made in leaps and bounds, but data accumulates incrementally. Many studies are performed, and the results all go into helping to accumulate our body of knowledge in a particular area. Randomised controlled trials (RCT) have become the gold standard for evaluating the effectiveness of interventions 1. But what if the trials contradict each other? Can we simply do a head count, and say, five studies find an effect, and two do not, can we conclude that the certain drug, device or method is effective? Well, no. In order to settle controversies arising from apparently conflicting studies, we use statistical analysis of findings to formally assess the degree of conflict and the reasons for the different results seen 2. We actually need a study of the studies, where results from all the RCTs are pooled, and that, essentially, is what a meta-analysis is 3. While meta-analyses are thought to have their roots in astronomy, the first medically-oriented study of studies was published in 1904, by the statistician Karl Pearson, who collated data from several studies looking at the effects of typhoid inoculation 4. A quick search through medical databases will show that meta-analyses have exploded in recent years, with over 40 000 papers indexed in PubMed alone (last search July 2015). Despite the recent exponential growth in meta-analytical publications, the statistical techniques used are relatively new, and a survey of clinician scientists showed that the statistics used in meta-analyses are not widely understood 5. The objective of a systematic review is to present a balanced and impartial summary of the existing research, and to provide a meta-analysis of net benefit, or harm, aggregated over all the included studies. Since meta-analyses combine the results of many trials, they have more power to detect small, but perhaps clinically significant, effects, and can give a more precise estimate of the size of the effect. Results in a meta-analysis are usually presented as Forest Plots. An example of a Forest Plot for continuous outcomes is given in Fig. 1. Each square represents the mean results from one study, with the tails showing the margin of error (the confidence interval, CI) around a result. Fig. 1 shows data from 13 separate studies. If any part of the square or the confidence interval lines touch the vertical line depicting zero, then there is no difference in effect between the treatment and control group at the selected significance level, for instance, a p value <0.05 for 95% CI. The diamond at the bottom represents the combined result from all 13 studies. A long, thin diamond means that there is a lot of uncertainty, but the diamond will be broad and short if the evidence for the effect is strong 6, although if the diamond cross the line of equality, regardless of its shape, there is strong evidence for the absence of effect. There are several different ways we can 'pool' and meta-analyse data from different studies. Data shown in Fig. 1 are the results from a systematic review, where data from separate studies are plotted separately, and then all pooled together into a combined result 7. Another way to meta-analyse data is to conduct a cumulative meta-analysis, where the results from the first study are plotted, then the results are pooled with the second study, then these two studies are pooled with the third one, and so on, so that changes as each study is added can be determined. A network, or multiple treatment meta-analysis, is one where the meta-analysts explore comparisons between treatments which were not attempted by the original investigators 8. For example, you may be interested in comparing two interventions, A and C, although these two interventions may never have been directly compared to each other in an RCT. If both A and C have been compared to a common comparator, B (perhaps a placebo), then A and C can be compared indirectly 9. A prospective meta-analysis is one where different groups of researchers agree with each other before conducting their original studies how to pool and meta-analyse their data. Prospective meta-analyses may be less prone to bias as many decisions may be made before the results of the original studies are known 10. According to the above descriptions of meta-analyses, it would appear that this type of publication should be the last word on any disagreements with regard to treatments or interventions. But meta-analyses can be prone to bias, and subjective decisions are involved in every step in conducting such a review 11. Before a meta-analyst can dissect the data from individual studies, the primary outcome, or the question to be answered, must be decided upon. If you are looking at studies of drug A to reduce post-operative pain, several outcome measures may be reported in the original studies, including pain scores and reduction in postoperative opioid requirement, as well as reduction in postoperative nausea and vomiting. Due to the differences in the nature of these measurements, different statistical tests must be applied. Outcome measures might be dichotomous, such as dead/alive, yes/no or success/failure; in such cases, the measure of treatment effect should be expressed as relative risk, odds ratio, or absolute risk difference. If the outcome measure is continuous, for example, blood pressure, or postoperative morphine requirement, then the pooled mean difference may be used. Hazard ratios are often used if the outcome measure is time-to-event, or survival time, examples of this type of outcome would be time to death, reoccurrence or time to first request of analgesic. Although many primers 12-14 have been published to aid the meta-analyst in the quest to transform a haystack of data into a needle of relevance, many data and statistical issues (some of which are particularly widespread in anaesthesia research) have not been addressed. Here, we outline some specific issues we came across when collating data for a meta-analysis on the efficacy of a certain analogue of gamma-aminobutyric acid (drug A) in reducing postoperative pain, in the hope that future such trials could incorporate more comprehensive and standardised outcome reporting. The haystack was a large one, and after exhaustive searching expeditions of trial registries and science-publication databases, a total of 74 trials passed the inclusion criteria. The outcomes we were interested in were two-hour and 24-hour postoperative pain scores, and total morphine consumption 24 hours after surgery. Two-hour postoperative pain measurements were chosen as it was considered the first time after surgery where any reductions in postoperative pain would be less likely to be due to the sedative effects of anaesthesia. Twenty-four hour postoperative pain assessed at this time-point is thought to be predictive of chronic postoperative pain 15. The secondary outcomes were postoperative nausea and vomiting, sedation, and visual disturbances. One important thing to note at the outset is that although the meta-analyst may be interested in these particular primary and secondary outcomes, and have stated them as such in the meta-analysis, the same measures may not have been the main outcomes in the original trial. All of the chosen outcomes for this meta-analysis posed statistical issues. The visual analogue scale (VAS) used to score pain is a continuous score from 0 (no pain) to 10 (worst pain imaginable). Continuous data should be handled in a meta-analysis using a pooled mean difference or standardised mean difference, where the means and standard deviations (SD) of the treatment and control group for this effect are used to calculate the difference between these two groups. However, VAS pain scores are often very skewed, and rather than reporting these values as means and SDs, medians and interquartile ranges are given instead. This also applies to postoperative morphine consumption, where individual morphine requirement can vary so substantially that reporting these data using mean (SD) would totally distort the picture. A quick mathematical manipulation can be applied to transform values from medians to means 16, if the sample size is bigger than 25, and if the maximum and minimum values are known. Which poses another problem, since standard statistical texts usually advise the reporting of skewed data as medians and 'inter-quartile range' (IQR), which is the middle value of your group, and the first and third quartile of your data. Simply put, data might be reported thus, median (IQR) 1.24 (0.96 to 3.71), these figures denote that the middle value for the group is 1.24, and a quarter of all the values are below 0.96, while three-quarters of all the values are below 3.71. Such reporting does not lend to the calculation of means and standard deviations, and there follows frantic emails to the original trial authors to request data not given in the original manuscripts. But even if the original authors are amenable in providing their raw trial data for the purposes of inclusion into a meta-analysis, is it appropriate to apply an obviously parametric test such as standardised mean difference calculations to equally obvious skewed data such as VAS scores and morphine requirements? Theoretically, further statistical transformations can be applied to normalise skewed data 17, but that is a discussion for another article. Though pain scores at individual time-points are reported in many studies of postoperative pain, some researchers deem the 'average' pain felt over a certain period of time (often up to 24 hours after surgery), to be a more relevant measure of analgesic effect 18. In such cases, VAS scores collected at different time points are plotted on a graph, and simple integration will give the total 'area under the curve', which is reported as the average pain experienced over that period of time 18. The meta-analyst must attempt to extract two and 24-hour pain scores from this collective data if it is to be included with the other studies in a meta-analysis. The analysis of outcomes such as nausea, vomiting and sedation is also complicated by the heterogeneity of reporting. While some investigators will report these outcomes simply as number of subjects with/without nausea, vomiting or sedation, others may present this data as a continuous variable using various sedation, or nausea scales 19. We have also noticed during our own meta-analysis of drug A, that around half of the studies reported nausea and vomiting as a composite outcome (as postoperative nausea and vomiting, PONV), while the other half reported nausea and vomiting as separate outcome measures. No statistical techniques exist which can tease out data reported as composite outcomes, and it should be worth noting that not all patients who reported vomiting were nauseated, and vice-versa. The abstract of our meta-analysis of the efficacy of drug A to reduce postoperative pain touting the inclusion of 74 randomised controlled trials sounds impressive. Readers are advised not to fall into the most common trap of meta-analyses and systematic reviews. Just because 74 trials were included somewhere in the article, it does not imply that all 74 trials reported on all the outcomes studied in the meta-analysis. It could very well be that only half of the studies reported usable data for two-hour pain scores, while only a quarter of them reported on adverse effects of drug A. Though the meta-analyst will endeavour to find every single RCT conducted on drug A, identifying the trial does not pre-suppose usability of the data. If the original investigators are unwilling or unable to reveal raw data for certain outcomes, these parameters may have to be excluded for that trial, rendering the systematic review to be a little less 'systematic'. Though meta-analyses are the best weapons we have in our arsenal to navigate the infinite number of studies published every day, they are not perfect and could be subject to bias, both in terms of bias inherent in trials and reporting, as well as biased decisions made by the meta-analyst 20, 21. The quality of a meta-analysis is only as good as the quality of the studies which were meta-analysed 22, and we should do well to remember the old statistical adage: if you torture the data long enough, it will confess.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 14 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.