The Registrar

Abstract

Improving the Quality of Research Reports in NEUROSURGERY®: The CONSORT, PRISMA, MOOSE, STARD, STROBE Statements and the EQUATOR Network Why Are We Doing This? What Are They? In a word—QUALITY—there is substantial evidence that there are significant deficiencies in the medical literature regarding the reporting of research studies. These shortcomings include the omission of valuable information about research methods or interventions, selective reporting of outcomes or risk, and ambiguous or confusing data. These errors can lead to incorrect conclusions, some with deleterious results for patients, misdirection of practitioners and poor decision-making by policy makers. These observations have lead to the development and dissemination of guidelines and checklists to assist researchers, authors, reviewers, editors and journals to meet certain standards of best practice. They are designed by experts in study design, epidemiology, biostatistics, and research methodology. The routine use of these research reporting guidelines by biomedical journals has become an important step in quality control. Reporting guidelines complement the journals' existing instructions to authors. Many journals require them. The published report of a study provides a clear explanation of the study methods and statistical techniques to allow independent verification of the results and much more. NEUROSURGERY® has taken a huge step in endorsing these guidelines and statements and will require them for submissions. A full report of this initiative follows below. Reporting Guidelines for NEUROSURGERY®: The CONSORT and PRISMA Statements Among the many sources of information used by practicing neurosurgeons in reaching clinical decisions—including training, personal experience or advice from peers, meeting presentations from experts, review articles, and material provided by industry vendors—clinical trials published in peer-reviewed journals hold a unique place because of high quality and broad, shared access with fellow clinicians. Prospective clinical trials, especially when randomized between treatment and control, provide the best evidence for medical decisions (when available), and a meta-analysis that combines several randomized clinical trials (RCTs) can provide even stronger evidence.1 Measured by citation patterns, RCTs in both medicine and surgery have more impact than papers that use weaker experimental designs, such as observational comparisons or case series,2,3 and many of the most-cited neurosurgical articles published in the last 20 years have been RCTs.4 Recent results from the Congress of Neurological Surgeons Integrated Medical LearningSM program showed that neurosurgeons strongly prefer RCT results for guidance in clinical controversies—by a 5-to-1 margin over any other source of clinical information (Barker et al, unpublished). A similar preference for RCTs is expressed by surgeons in other specialties.5,6 National review panels are more likely to approve funding for operations and procedures with RCT support.7 Most importantly, neurosurgical RCTs are frequently followed by obvious, rapid shifts in community practice patterns, both toward effective therapies8-11 and away from ineffective ones.12-14 Neurosurgical RCTs Are Scarce and Not Always Well Reported Although highly influential, RCTs are relatively uncommon in the neurosurgical literature, comprising far fewer published studies than weaker designs such as case series and case reports.15 RCTs form a smaller fraction of papers published in neurosurgical journals than in internal medicine, general surgery, or many surgical subspecialties.16, 17 An extensive literature has been devoted to the particular difficulties of conducting RCTs on surgical questions,18-20 but the specific reasons for the paucity of RCTs in neurosurgery are still elusive.15 Perhaps the perception that neurosurgical diseases are rare, and that mistaken therapies can result in severe consequences, could be contributing factors – although pediatric oncologists in the US face similar problems while achieving far higher enrollment rates of eligible patients onto clinical trials.21 Some evidence, however, suggests that the number of RCTs on neurosurgical topics has been slowly increasing.15, 22 Given the combination of the scarcity of neurosurgical RCTs and their singular value in shaping clinical decisions, it is particularly unfortunate that many neurosurgical RCTs show flaws in design and reporting. Vranos et al surveyed 108 RCTs on neurosurgical procedures published between 1966 and 2002; they found that many trials were underpowered, and few reported power calculations or allocation concealment adequately.23 In a more recent survey of 159 neurosurgical RCTs, Schöller et al found that almost half of trials had inadequate reporting of allocation concealment while only a third reported intent-to-treat analysis.24 These deficiencies in the quality of published neurosurgical RCTs are not unique or even unusual.25 Systematic reviews have found that similar flaws in conduct or reporting are endemic to RCTs reported in both medical and surgical journals, even those that are highly-cited.26-29 However, evidence suggests that trials evaluating nonpharmacological therapies, such as surgical procedures, face special challenges in clear and accurate trial reporting.30-32 For example, in a systematic review on randomized and non-randomized published studies of 4 orthopedic operations, reviewers found that nearly 30% of studies lacked details about how the operations had been performed that surgeons considered necessary to proper interpretation of the trials' results.33 Reporting Guidelines Can Improve the Quality of RCT Reports: The CONSORT Statement Concerns about the adequacy of trial reporting led several groups to formulate guidelines for reporting RCTs in the early 1990s, and by far the most widely accepted of these has become the “CONSORT statement” (Consolidated Standards of Reporting Trials).34 First published in 1996, the CONSORT statement has been updated twice, most recently in 2010.34 In its present form (www.consort-statement.org), the CONSORT statement includes a checklist of 25 items that should be included in all reports of RCTs. The group also provides an extensive “Explanation and elaboration” document,35 available on the group's website, which spells out the group's rationale for requiring each of the checklist items, and includes examples of correct reporting. Because the rationale for requiring most of the items is supported by empirical studies demonstrating that proper trial design reduces bias in assessing treatments, the CONSORT statement elaboration document can be read as a primer on modern clinical trial design and conduct as well as a guide to reporting. Additionally, the group's website includes an archive of literature references supporting the CONSORT items that has grown to about 700 articles. Not all RCTs are parallel-group comparisons of pharmacological agents, which was the trial design CONSORT was initially formulated to guide. Extensions of the CONSORT statement have been written that cover other types of RCTs, such as noninferiority and equivalence trials,36 pragmatic trials,37 and trials reported as abstracts.38,39 Of particular importance for surgeons are the extensions covering trials that use cluster randomization (eg, between a surgical and a nonsurgical treatment where the unit of randomization is the center or surgeon),40 on improving the description of harms resulting from experimental treatments,41 and on nonpharmacologic treatments, including recommendations for reporting details on provider (surgeon) characteristics such as training, certification, or volume of care.30 For example, one of the early criticisms of the ISAT trial on clipping vs coiling for ruptured aneurysms was that insufficient information had been provided on surgeons' levels of experience and clinical volume in the trial.42 With the CONSORT statement in hand, authors, reviewers and editors now have an important tool for evaluating and improving the quality of RCT reports. Investigations have shown that many inadequately reported RCTs were actually well-conducted, pointing out an opportunity for many authors for immediate improvement of future trial reports.43, 44 In addition, widespread recognition of the importance of the items in the CONSORT checklist can help improve trial design itself. Studies of RCTs published before and after the CONSORT introduction,45,46 and of RCTs published in journals that do or do not require CONSORT compliance in reports,47-50 show that the statement has improved trial report quality significantly.51 Surgical trial reports have improved in quality as well,52 whereas during the decade before the CONSORT statement the quality of surgical trial reports did not change.53 Almost 500 medical journals have now “endorsed” the CONSORT statement (www.consort-statement.org/about-consort/supporters/), meaning that they require authors of RCT manuscripts to report the items contained in the CONSORT checklist. Most associations of medical journals and editors have also endorsed the statement, including the Council of Science Editors (www.councilscienceeditors.org), the International Committee of Medical Journal Editors (www.icjme.org), and the World Association of Medical Editors (www.wame.org). With this issue, NEUROSURGERY® joins the list of journals endorsing the CONSORT statement. NEUROSURGERY® will also support the ICJME requirement that prospective therapeutic trials be registered in a public trials registry, such as the one maintained by the National Library of Medicine (www.clinicaltrials.gov) or the equivalent, to be considered for publication.54-56 Guidance for Reports Using Other Research Designs: PRISMA for Meta-analyses, and the EQUATOR Network RCTs are uncommon in neurosurgery,15 and while many aspects of the CONSORT checklist are applicable to other types of study design, guidelines and checklists have been prepared that are specifically tailored to many designs used in neurosurgical studies (see the Table). For example, the PRISMA statement (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; www.prisma-statement.org) provides guidance to authors of meta-analyses and systematic reviews.57 As for the CONSORT statement, a separate document describes the reasoning behind the individual items in the 27-item PRISMA checklist.58 The items provide guidance on searching methodology, procedures for combining individual study results, and on presentation of analyses. Of note, the PRISMA authors specify fixed-effects and random-effects as the two common methods of combining data from different studies; simply dividing the total number of events by the total number of patients (“pooling”) is an inappropriate method for meta-analysis. As for CONSORT, bibliometric studies confirm an increase in literature meta-analysis quality after publication of the predecessor to the current PRISMA statement.59 NEUROSURGERY® now endorses the PRISMA statement and will require the use of the checklist in preparing systematic review and meta-analysis manuscripts for submission.TABLE: Important Guidelines and SourcesaThe art of meta-analysis is even more arcane than that of performing a randomized trial, and many aspects of meta-analysis are very controversial. For example, whether to use non-randomized studies in a meta-analysis is a difficult question (that probably has no simple answer) – but a vital one for neurosurgery because of the scarcity of neurosurgical RCTs.33,60-63 In general, when studies other than RCTs are combined in a meta-analysis, special care must be taken to avoid trying inappropriately to pool what amounts to “apples and oranges”—that is, a careful heterogeneity analysis must be performed and adequately discussed. A guideline intended to help in preparation of meta-analyses of observational studies is MOOSE (Meta-Analysis of Observational Studies in Epidemiology), which should be used by authors to supplement PRISMA when non-randomized studies are included in a meta-analysis. With increasing frequency, authors are attempting to combine the results of single- or multi-institutional case series, and even individual case reports, using statistical methods. This practice is likely to produce results that are difficult or impossible to interpret because the heterogeneity in results from different single-arm studies is so large,64 and essentially impossible to explain from literature reports alone. A recent neurosurgical example illustrates the problem of heterogeneity between different study results in a case-series meta-analysis, with event rates ranging from 2% to 50% for the main endpoint within one arm of the comparison and from 0% to 45% in the other arm. Other types of bias also afflict studies that pool scattered information from noncomparative studies. For example, in one meta-analysis of case reports and small series of osteosarcomas of the head and neck, the authors found that postoperative radiation treatment was associated with shorter survival. They speculated that radiation was more often used in patients with positive surgical margins or gross residual disease, and recommended adjuvant radiation despite the results of their analysis.65 Given the many undetectable and insurmountable biases associated with treatment comparisons that use meta-analyses of noncomparative (single-arm) case series, NEUROSURGERY® will no longer accept such studies for publication. Most of the neurosurgical literature is composed not of RCTs and meta-analyses, but of retrospective studies, single-arm case series, and case reports. For some such study designs, published guidelines exist that can aid authors in designing and reporting studies, but there has been little consensus on their value and few if any journals require their use. The STROBE guidelines (Strengthening the Reporting of Observational Studies in Epidemiology)66,67 provide guidance that applies in part to many neurosurgical studies that are observational in nature, although the differences between treatment trials and risk factor studies are significant. For investigators reporting studies of diagnostic accuracy, the STARD statement (Standards for Reporting of Diagnostic Accuracy) can be of assistance, particularly when authors are unfamiliar with the specialized methods used in this literature.68 Few authors will have the time to keep track of the ever-increasing number of guidelines available to assist with specific study types, such as those governing acupuncture trials or trials of herbal medicine. The EQUATOR network website (www.equator-network.org) maintains a useful, up-to-date list of guidelines as they are published, with links to articles and checklists.69 Over 70 specific guidelines have appeared to date. Authors will also frequently be familiar with subject-specific guidelines to assist preparation of manuscripts in their own field of expertise, such as the 2007 standards for reporting surgically-based brain tumor therapeutic trials.70 Even experienced authors are likely to benefit from reviewing applicable checklists before submitting a manuscript, although most such guidelines are intended by their authors not to be used as a rigorous list of prescribed content. One group described their list's intended use as “most helpful if authors simply keep the general content of the guideline items in mind … [and] refer to the details of individual items … during the revision process.”71 Finally These initiatives bring us in line with the top tier of biomedical journals and should, more than ever, encourage the highest quality submissions in our specialty. We will provide the necessary support to assist authors, reviewers and editors as we navigate these changes. Our instructions to authors will be revamped and the PEGASUS (electronic submission) system will be updated to reflect the changes. There will be many helpful online links provided. We will allow time for everyone to acclimatize and the changes will take effect July 1, 2011. Welcome to the new and improved NEUROSURGERY®. Fred G. Barker, II, MD, FACS Section Editor, Evidence-Based Medicine Editorial Review Board, NEUROSURGERY® Boston, MassachusettsFIGURE: Caption not available.Nelson M. Oyesiku, MD, PhD, FACS Editor-in-Chief, NEUROSURGERY® Atlanta, Georgia January, 2011

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 8 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.