No open-access location is known for this record yet.
No open-access location is known for this record yet.
Systematic reviews provide the best estimates of the true effects (both beneficial and adverse) of medical interventions (1). In this era of evidence-based medicine, clinicians are increasingly using systematic reviews to keep up with new evidence and to guide their clinical decision-making. Yet the main challenges for clinicians are to translate the results of systematic reviews into clinical practice and to provide optimal patient care. This concept is known as “applicability.” Applicability addresses whether a particular treatment that showed an overall benefit in a study or systematic review can be expected to convey the same benefit to an individual patient (2). This paper outlines a framework for how quantitative systematic reviews (meta-analyses) should be reported and how they may be used to identify those individuals in whom the treatment is likely to do more good than harm. We illustrate the concepts by using data from systematic reviews of ondansetron for the treatment and prevention of postoperative nausea and vomiting (PONV). Throughout this paper, we use the terms “baseline” and “underlying risk” interchangeably. Underlying risk is defined as the risk of event for a patient under the control condition; it indicates the average risk of a patient if not treated (3). Limitations of Systematic Reviews and Meta-Analysis In a quantitative systematic review, results of primary studies are combined by using complex statistical methods to estimate a common pooled effect with increased precision (4). The overall results of a meta-analysis represent an “average” treatment effect. Before applying the results of systematic reviews to individual patients, clinicians must consider some of the limitations and biases (5). One of the weaknesses of meta-analysis is that it magnifies the problems of individual trials. In the case of systematic reviews of ondansetron, for example, the pooling of results from different studies may lead to inconsistencies in separate incidences of nausea, vomiting, and combined PONV because investigators do not often distinguish between nausea and vomiting but combine the symptoms of PONV into a single outcome (6). Second, if the overall conduct of the individual trial is poor, a summary treatment effect will generally be an overestimate of the “true” effect by up to 41%(7). Other specific methodological issues related to systematic reviews of ondansetron include variability of the underlying risk and event rates and small trial sample size, problems related to antiemetic comparisons without a placebo group and covert duplication of primary trials (8,9). Therefore, the interpretation of the relative efficacy and harm of antiemetic treatment can be difficult and a cautious attitude to accepting the results is warranted. Existing Methods for Applying Results to Individual Patients Despite the above limitations, systematic reviews and meta-analyses can highlight how effective a treatment is and identify subgroups of patients who would most benefit from an intervention. Systematic reviews and meta-analyses can guide clinical decision-making, but they are integrated with other factors in caring for an individual patient (10). Common methods of applying the overall results of meta-analysis to individual patients are outlined below but these have limitations. First, the number needed to treat (NNT) to achieve one unit of benefit is a common clinically useful measure. However, pooled NNT may be misleading because NNT are sensitive to factors that change the baseline risk, such as the outcome considered, trends in disease risk, and clinical setting (11). A solution to this is that NNT should be derived by applying the relative risk reductions from treatment estimated by the meta-analysis to relevant baseline risks for different types of patients (Table 1) (11).Table 1: Calculating Numbers Needed to Treat (NNT) for High- and Low-Risk GroupsAnother erroneous method is to apply the estimated average treatment effect to all patients that meet the eligibility criteria of the clinical trials (12). Not all of these individuals will experience a net benefit, and some that were not eligible for the trial may benefit from treatment (2). There is evidence to suggest that treatments found to be beneficial in a narrow range of patients can have broader application in actual practice (1). Differences between study participants and patients in real-world practice tend to be quantitative (differences in degree of risk of the outcome or responsiveness to therapy) rather than qualitative (no risk or adverse response to therapy) (1). Economic evaluation seeks to predict net change in benefits and costs arising from alternative approaches to providing a particular form of care (13) and can be useful in deciding the optimal strategy for treatment in a patient. The applicability of the results of an economic evaluation depends on the characteristics of the patients treated, the resources used that are related to outcome, the intrinsic effectiveness of the treatments compared, the viewpoint of the analysis and time scale of the decision to be made (14). However, a review of published economic evaluations showed that two-thirds of articles gave misleading conclusions about the relative costs of alternative treatments without supporting statistical evidence (15). Therefore, the reliability and validity of cost-effectiveness analyses of antiemetics to manage PONV and its subsequent applicability to individual patients may be questionable. Currently, there is no formal methodology for the conduct and reporting of systematic reviews of economic evaluations (13). Framework for Applying the Evidence from Systematic Reviews to Individual Patients An approach for assessing the applicability of results from systematic reviews to individual patients has been proposed by O’Connell et al. (2). This requires the following five questions to be addressed: 1. What are the beneficial and harmful effects of the intervention? 2. Are there variations in the relative treatment effect? 3. How does the treatment effect vary with baseline risk level? 4. What are the predicted absolute risk reductions for individuals? 5. Do the expected benefits outweigh the harms? The first three questions address the issue of transferability of the average treatment effect (2). The last two questions cover aspects of individualizing the treatment decision through estimating the expected absolute risk reduction based on an individual’s baseline risk and then taking into account the patient’s preferences in determining benefits and harms (2). When considering these five issues, there may be insufficient data (2). However, this will highlight the additional information required to be reported in future systematic reviews. In the next section, we use data from the systematic reviews of ondansetron to illustrate the five issues. Appraising Current Systematic Reviews of Ondansetron We identified seven quantitative systematic reviews of ondansetron for the prevention and/or treatment of PONV (16–22) by searching the MEDLINE (1966 to August 2000) and EMBASE (1980 to August 2000) electronic databases. One paper was a meta-analysis of individual patient data from three randomized controlled trials (16). Each systematic review was assessed according to the five criteria outlined above (Table 2). Table 3 shows how each systematic review met these criteria.Table 2: Criteria for Assessing Applicability of Systematic Reviews of Ondansetron for Management of Postoperative Nausea and Vomiting (PONV)Table 3: Applicability Criteria Reported by Systematic Reviews of Ondansetron for Management of Postoperative Nausea and VomitingWhat are the Beneficial and Harmful Effects of the Intervention? In considering the potential benefits and harms, all authors reported surrogate measures (nausea, vomiting, nausea and/or vomiting) but did not consider patient-relevant outcome measures (patient satisfaction, length of stay in the postanesthesia care unit or hospital, time to full recovery), as these were not reported in the primary trials. The incidences of nausea (16,18,22), vomiting (16,18–22), and nausea or vomiting (17) were measured. The incidence of headache was measured in three papers (18,19,22). In addition, Tramer et al. (18) measured the incidence of increased liver enzymes, hypotension, constipation, and abdominal cramps. Other side effects, such as sedation, anxiety, restlessness, and abdominal muscle movements, were considered in a meta-analysis that compared ondansetron with metoclopramide and droperidol (22). The severity of adverse effects was not reported. Are there Variations in the Relative Treatment Effect? To understand how an intervention works, consideration is given to determine whether treatment effect is constant or whether it varies according to patient, disease, intervention, and measure of effect used (2). Variations in relative treatment effect were best examined using individual patient data (16) rather than a series of meta-analyses of subgroups from primary randomized controlled trials. In the individual patient data meta-analysis, significant factors associated with the incidence of PONV were dose of ondansetron, neuromuscular blocker use, country, volatile anesthetic, type of surgery, and age, the last two factors showing a significant interaction (16). Subgroup meta-analyses of patients undergoing eye surgery, ear/nose/throat surgery, and anesthesia with or without propofol were conducted in one paper (22). Many authors examined the dose-response efficacy of ondansetron (17–21), different routes of administration (18,19), and timing of ondansetron administration in relation to surgery (19). There were variations in the optimal dose of ondansetron for prophylaxis; 4 mg (19), 8 mg in patients with a history of PONV (21), 4 mg and 8 mg both equally effective in patients with a history of motion sickness (20), 8 mg IV (18), 16 mg orally (18), or repeated 8 mg oral doses (16). For the treatment of established PONV, there was no clinically relevant dose response between 1 mg and 8 mg (17). Heterogeneity can be defined as the variability or differences between studies in the estimates of effect (23). All systematic reviews except one (16) tested for heterogeneity because if present, potential sources of heterogeneity should be identified, as this can affect the overall conclusion as well as the clinical implications of the review. An important source of heterogeneity is the underlying risk (24). As the underlying risk is not a measurable quantity; the best estimate is the observed risk of events in the control group (24). How Does the Treatment Effect Vary with Baseline Risk Level? Of the seven systematic reviews, only one (18) assessed how treatment varied with baseline risk. A sensitivity analysis of two predefined ranges of baseline risk—early PONV within 20% to 60% of control event rates and late PONV within 40% to 80% of control event rates was conducted in one systematic review of ondansetron used for prophylaxis of PONV (18). As there was little variation in the control group in the trials included in the systematic review of ondansetron for the treatment of PONV (17), the criteria of how treatment varies with baseline risk may not be relevant. Which patients will obtain a net benefit will depend on the harms associated with the treatment. This tradeoff and how it relates to the underlying risk is discussed in more detail in the next section. What are the Predicted Absolute Risk Reductions for Individuals? To judge whether treatment is worthwhile for an individual patient, an appropriate NNT is estimated. This was done in one systematic review (19) using the Cook and Sackett method of applying patient’s baseline risk to the overall NNT (25). Do the Expected Benefits Outweigh the Harms? None of the seven systematic reviews discussed patient preferences associated with risk-benefit ratio, as the original trials did not report these data. The tradeoffs between risks and benefits can involve patient-centered outcomes, such as patient satisfaction and quality of recovery after anesthesia (26), measures that have been developed and validated recently. Which Patients Will Benefit? We applied the risk-benefit approach (12) to individualizing treatment as an example of how an anesthesiologist may consider a risk-benefit ratio using data from a meta-analysis and a cohort study. We show how this approach can be applied to the prophylactic use of IV ondansetron 4 mg using data from a systematic review (18). For the purposes of this example, benefit was defined as reduction in postoperative vomiting in the first 48 h and harm was defined as the incidence of headache. In clinical practice, consideration is given to all clinically important adverse effects. Schmid et al. (27) showed that across a wide range of therapies, low-risk patients gain less absolute benefit compared with high-risk patients and as patient’s expected risk increases, the absolute risk reduction increases proportionally. We assessed the net benefit of prophylactic ondansetron by estimating the absolute risk reduction from the primary randomized controlled trials included in a systematic review (18). The overall risk reduction in postoperative vomiting associated with ondansetron was 20% (95% confidence interval [CI], 17%–24%) using a random-effects model that took into account within and between study variability. Therefore, the pooled NNT was 5 (95% CI, 4–6). The risk of vomiting in the placebo group, an estimate of the underlying risk (x axis) was plotted against reduction in absolute risk of vomiting (y axis, plot 1) and excess absolute risk of headache (y axis, plot 2). Patients should weight the benefit and harm against each other. This can be done by assigning monetary value for the two items. In this example, the monetary value given by patients in decreasing and eliminating emesis compared with experiencing headache was a 1:1 ratio (28). The mean risk of vomiting in the control group was 56% (95% CI, 54%–59%), and the excess risk of headache was 1% (95% CI, −1% to 3%). The point at which the line of benefit and line of harm crossed was the threshold. Net benefit occurred only when the line of benefit was above the threshold of 5% (Fig. 1). It is important to note that this threshold will vary according to the weighting of benefit to harm ratio.Figure 1: Benefit compared with harm for the prophylaxis of vomiting (0–48 h) by ondansetron 4 mg IV dose. Data taken from Tramer’s et al.’s systematic review (18). The number of risk factors is described in Koivuranta et al.’s study (29).We also plotted the risk of postoperative vomiting against the number of risk factors (female sex, previous PONV, duration of operation more than 1 h, history of motion sickness, and nonsmoking status) as estimated by Koivuranta et al. (29) in Figure 1. This score was chosen because it has moderate accuracy in predicting PONV (30). These points on the x axis were 7% (0 to 1 risk factors), 17% (2 risk factors), 25% (3 risk factors), 38% (4 risk factors), and 61% (5 risk factors). Figure 1 shows that patients with 0 or 1 risk factors of postoperative vomiting would not benefit greatly from ondansetron prophylaxis and that high-risk patients would benefit most from prophylaxis. Caution is needed in assessing the relationship between treatment effect and underlying risk as it can be misleading as a result of the “regression to the mean” phenomenon as the baseline risk forms part of the definition of treatment difference (24). In another words, even if there is no true relationship between baseline risk and treatment effectiveness, one is likely to be observed because of this statistical artifact. This problem is reduced if studies are large, if variation in true underlying risks is large, and if there are a large number of studies (24,31). A Bayesian procedure for random-effects meta-analysis has been proposed (3,24) to overcome this problem. Another alternative, if individual patient data are available for meta-analysis, is to relate treatment effects to individual patient covariates (predictors of risk). This would be directly useful for the clinician considering treatment for an individual patient (24). Conclusions Appropriate reporting of meta-analysis is required to allow evidence-based decisions to be made. In this study, all systematic reviews adequately addressed the issue of transferability but failed to predict the absolute risk reduction for individuals. Therefore, it is difficult for anesthesiologists to clearly assess whether the expected benefits of ondansetron outweigh the harms for individual patients. More effort is required in reporting the results of meta-analysis in a way that will help anesthesiologists to individualize treatment. The framework used in this paper may help to improve the reporting of results from quantitative systematic reviews. Currently, there is limited applicability of results of ondansetron meta-analyses to individuals. A guideline, “the QUOROM statement”(32), may improve the quality of reports of meta-analyses of randomized controlled trials. However, we note that there is no emphasis in the guideline on how to apply the results of meta-analysis to the clinical setting. Systematic reviews should address the following two questions: Does the treatment work? Which individuals will benefit most from the treatment? Depending on the availability of data from primary trials and observational prognostic studies, we believe that if the framework described is applied after considering some of the limitations, the divergence between research evidence and current clinical practice may narrow.
Loading…
Stored only when both papers are in the corpus — 5 citations exist upstream.
Imported from OpenAlex (CC0) — nothing here is generated.