No open-access location is known for this record yet.
No open-access location is known for this record yet.
Intervention research provides information regarding the potential health impacts of public health initiatives and is critical to guide evidence-informed public health decision-making and practice. According to translation models of research, testing of public health interventions should first occur under ‘ideal’ controlled conditions to maximize internal validity; increase confidence that intervention effects are attributable to the intervention and aid an understanding of casual relationships between the intervention and outcome assessed.1 Such studies are known as ‘explanatory’ or ‘efficacy’ trials and are usually characterized by restrictive sample selection, intervention delivery by experts and inflexible intervention protocols.2,3 If found to be effective, these interventions should next be tested under ‘real world’ conditions to assess the likely impact of an intervention if it was implemented in the community.1 These studies are known as ‘pragmatic’ or ‘effectiveness’ trials and have less restrictive sample selection procedures, flexible intervention delivery approaches and a greater focus on external validity.2,3 Consideration of whether studies are explanatory or pragmatic in their design may be particularly useful when synthesizing the impact of public health intervention trials as part of systematic reviews. Given the wide variation in study methodology, the characteristics that distinguish between explanatory and pragmatic trials are likely to influence both statistical heterogeneity and intervention effect estimates in meta-analysis. As systematic reviews of public health trials often report high heterogeneity, pooling data from trials grouped according to their explanatory or pragmatic design may increase confidence in the results obtained from meta-analyses. Examination of characteristics that distinguish between pragmatic and explanatory trials in systematic reviews may also improve the policy and practice relevance of systematic review findings.2,4,5 Specifically, a number of authors have proposed that the effects of interventions attenuate as trial designs become more pragmatic in nature due to the potential greater variability in delivery of such trials.5–7 Given that findings from pragmatic trials are more likely to closely approximate the public health impacts caused by an intervention if investments were made to introduce it into the community, examining intervention outcomes based on trial design is likely to provide more useful information for assessing the transferability of public health initiatives.8,9 The pragmatic–explanatory continuum indicator summary (PRECIS) was developed by a group of international researchers and methodologist to assist trialists distinguish between pragmatic and explanatory trial designs.2 PRECIS requires trialists to indicate on a visual scale (in the shape of wheel) where a trial falls along the pragmatic–explanatory continuum across 10 domains (see Table 1). An adapted version of the PRECIS tool has been used in two systematic reviews to classify trials as explanatory or pragmatic.4,10 Neither review examined public health interventions (one examined the effectiveness of acupuncture for treatment of lower back pain and the other examined clinical management of hypertension in primary care), nor did they describe the utility of the tool as a resource for systematic review authors. Description of pragmatic/explanatory trials based on the PRECIS domains. PRECIS, pragmatic–explanatory continuum indicator summary. Thorpe et al.2. Accessed 15 November 2013. Description of pragmatic/explanatory trials based on the PRECIS domains. PRECIS, pragmatic–explanatory continuum indicator summary. Thorpe et al.2. Accessed 15 November 2013. In this paper, we describe an approach to applying the PRECIS tool to categorize the explanatory and pragmatic nature of trials included in public health systematic reviews and provide some reflections on the potential utility of this tool for use in the conduct of systematic reviews. Specifically, using the Cochrane Review of Interventions for preventing obesity in children11 as a case study, we describe: (i) approaches for scoring and classifying trials as pragmatic or explanatory using the PRECIS tool; (ii) the inter-rater reliability of trial classifications; (iii) the availability of information needed to classify trials; (iv) the impact of missing information on trial classification and (v) the potential impact of study classification on estimates of intervention effect. The Cochrane obesity prevention review (2011) included all controlled studies published between 1990 and March 2010, which examined community-based interventions to prevent obesity in children.11 Overall, 55 articles were included in the review and 49 comparisons were included in the meta-analyses. Using an adapted version of the PRECIS tool,5 we conducted an audit of all 55 trials included in the Cochrane review11 to assess the pragmatic–explanatory design features of these studies. Instead of using the original ‘wheel’ scale to give a visual representation of trial characteristics, a quantitative scoring system was used, similar to that used by a number of other authors.4,5 For each of the 10 PRECIS domains, trials included in the review were independently scored by 2 authors on a 5-point scale from 0 (completely explanatory) to 4 (completely pragmatic) using the broad definitions provided by the tool developers (see Table 1 for definitions).2,5 Authors then met to discuss variations in scoring and reached a consensus where there were discrepancies. As no cut-off scores were provided by the original authors, we applied a scoring method for categorizing trials as (i) pragmatic; (ii) combination pragmatic/explanatory or (iii) explanatory, based on the distribution of the 55 included trials. We classified trials as predominantly explanatory if the average score for the trial was between 0 and 1.7; combined explanatory/pragmatic if the average score was >1.7 and ≤2.2 and mostly pragmatic if average score was >2.2 up to 4.0. When a publication did not clearly state a design characteristic for a trial, data for that specific PRECIS domain were treated as missing. To assess whether the treatment of missing data affected overall trial classification we ran the classification process twice, using two different methods. Method 1 involved imputing a score of ‘2’ for any domains where data were missing and deriving the overall average score by summing all 10 domain scores and dividing by 10. Method 2 involved no imputation for missing data, and the overall average score was derived by summing scores from each domain and dividing by number of domains with no missing data. To assess whether explanatory or pragmatic trial classification had an impact on estimates of intervention effect, we descriptively examined the standardized mean change in BMI/zBMI scores for all studies included in the meta-analyses by trial classification and age groups (similar to that used in the original Cochrane obesity prevention review). To examine inter-rater reliability between the two independent raters, a weighted kappa was calculated for each domain. The kappa ranged from 0.23 to 0.75 (Table 2), indicating low to substantial agreement across the domains.12 We achieved fair agreement (0.21–0.40) for one domain, moderate agreement (0.41–0.60) for four domains and substantial agreement for five domains (>0.60).12 Lowest kappa values were identified for the domain assessing follow-up intensity, whereas highest agreement was identified for the domain assessing practitioner's expertise in experimental intervention delivery. We noted that the definitions provided by PRECIS developers were at times difficult to apply to coding of public health interventions, particularly for the domains assessing outcome measures, analysis and flexibility of intervention delivery. For example, it was often difficult to code the flexibility of intervention delivery for trials that consisted of multiple components and were delivered by various practitioners, as is common in public health interventions. Number and proportion of studies that reported insufficient information within text for each PRECIS domain, and weighted kappa between reviewers 1 and 2 for each domain. aOne study coded as not applicable as no practitioner was responsible for delivering the control intervention. Number and proportion of studies that reported insufficient information within text for each PRECIS domain, and weighted kappa between reviewers 1 and 2 for each domain. aOne study coded as not applicable as no practitioner was responsible for delivering the control intervention. Across all 55 trials, missing data occurred for 6 of the 10 domains due to a lack of information provided in the included publications (Table 2).Missing data were most frequent for domains that assessed the monitoring of participant compliance and practitioner adherence to study protocol. The lack of information reported in published manuscripts is a well-known challenge for systematic reviewers in their examination of potential study characteristics that may influence heterogeneity and intervention effect.13,14 Missing data also present challenges for scoring systems such as this, where data are required in each domain in order to make an overall classification of a trial as explanatory or pragmatic. Six (10.9%) trials were categorized into different categories based on the two different methods of treating missing data (Table 3). As part of meta-analyses in systematic reviews, review authors should consider applying both scoring systems in a sensitivity analysis to test the assumptions of missing data on the scoring system used. Number of explanatory, combined and pragmatic trials using scoring methods 1 and 2. aMethod 1 involved imputing a value of 2 for missing data and dividing scores by all 10 domains. bMethod 2 involved summing scores from all domains with data and dividing by number of domains with no missing data. Number of explanatory, combined and pragmatic trials using scoring methods 1 and 2. aMethod 1 involved imputing a value of 2 for missing data and dividing scores by all 10 domains. bMethod 2 involved summing scores from all domains with data and dividing by number of domains with no missing data. Using Method 1 for trial classification, 49 comparisons were included in the meta-analysis. As can be seen in Table 4, pragmatic trials had the smallest effect size overall, and for each age group. The largest overall intervention effect size was found in trials that were explanatory in design; however, the combined overall effect of pragmatic trials on BMI was non-significant (overlap in 95% confidence intervals (CIs) between the different trial designs). The overall intervention effect size for pragmatic trials [−0.09 kg/m2 (95% CI −0.19, 0.01)] was not significant (95% CI included zero), while it was significant for both pragmatic [−0.21 kg/m2 (95% CI −0.35, −0.08)] and combination [−0.16 kg/m2 (−0.25, −0.06)] trial designs, suggesting that further explorations of how the individual domains included in the PRECIS tool impact on intervention effect may be warranted. Also worth nothing is that the distribution of explanatory, combination and pragmatic trials varied by age group. There were no pragmatic trials conducted in the 13–18-year-age group, 10 pragmatic trials conducted in the 6–12-year-age group and 1 in the 0–5-year-age group. These findings suggest that a gap may exist in the evidence base for obesity prevention trials for the younger (0–5 years) and older (3–18 years) age groups, where few pragmatic trials are available to inform policy and practice. Standardized mean change in body mass index from baseline to post-intervention for childhood obesity prevention interventions by age group and explanatory/combination/pragmatic trial designs (including 49 comparisons). aMean BMI change data obtained from Review manager using data from Waters et al.11 bClassification based on Method 1 of treating missing data. cNot estimable as no trials fell into this study classification. Standardized mean change in body mass index from baseline to post-intervention for childhood obesity prevention interventions by age group and explanatory/combination/pragmatic trial designs (including 49 comparisons). aMean BMI change data obtained from Review manager using data from Waters et al.11 bClassification based on Method 1 of treating missing data. cNot estimable as no trials fell into this study classification. The Agency of Health Care Research and Quality in the USA recommended in 2011 that authors of reviews more systematically describe and analyse the methodological design traits and resulting treatment effects of studies in order to guide application in real-world settings.15 In this case study, we demonstrate that the PRECIS is a useful and promising tool for authors of public health intervention systematic reviews, who wish to consider the extent in which explanatory and pragmatic study paradigms moderate intervention effect. While other authors have reported that intervention effect size tends to decrease when delivered under more real-world conditions,6,16 we did not observe any differences in intervention effect between the three classifications of studies included in this obesity prevention review. Further exploration of the impact of explanatory and pragmatic study designs with a larger number of studies, across a range of topics are needed to provide a better understanding of whether these designs moderate the intervention effects of public health interventions. For authors considering PRECIS assessment as an element within a review, we would recommend to plan on doing so at the stage of protocol development, to ensure that necessary information is collected during the process of data extraction for the main review. Indeed, reviewers may need to make an initial assessment during initial scoping of the review to determine whether a sufficient mix of both explanatory and pragmatic trials may be available on the topic to warrant the additional effort described. Reviewers may choose to report results of the PRECIS assessment together with other subgroup analyses in the review. A number of challenges, including high frequency of missing data and difficulty achieving substantial inter-rater reliability, may exist for systematic review authors who would like to use the PRECIS method. While not specific to public health interventions, a revision of the PRECIS tool with plans to address these described limitations is currently in progress and is likely to improve the utility of this tool for systematic reviews.16 Given difficulties with achieving high inter-rater reliability for a number of domains, we recommend that clearer domain definitions which are relevant to the specific topic examined be established, prior to coding. Additionally, our experience suggests that systematic review authors should be prepared to contact primary trial authors or identify published protocols of trials in order to obtain the necessary data required to assess a trial using the PRECIS tool. Additionally, as the developers of the original tool did not include any cut-offs for trial classification, the proposed cut-offs for classification of pragmatic and explanatory trials used in this case study should be further examined. More case studies are needed to increase our understanding on how to best apply and interpret findings from this tool, so that results become more relevant to informing public health policy and decision-making. Interventions to reduce ambient particulate matter air pollution and their effect on health. Infrastructure support was received by Hunter New England Population Health and Hunter Medical Research Institute, New Lambton for the conduct of the study. Dr L.W. is supported by a NHRMC Career Development Fellowship. The authors wish to acknowledge Jodie Doyle, Managing Editor, Cochrane Public Health Group, who provided comments on earlier drafts of this article.
Loading…
Stored only when both papers are in the corpus — 47 citations exist upstream.
Papers that cite the same references — the relational fallback while embeddings are still backfilling.
Imported from OpenAlex (CC0) — nothing here is generated.