Editorial: Peer Review and the Editorial Process - A Look Behind the Curtain

Abstract

Peer review is fundamental to the editorial process, but the vital work of our reviewers is only one component of many in that process. As we begin the new year, this seems a suitable time to give both our readers and our authors a look behind the curtain: What do we know about peer review's efficacy, can it be improved, and what happens at Clinical Orthopaedics and Related Research® after the reviews have been completed? We can trace the origins of peer review as far back as the 18th century [6], but consistent, contemporary-looking peer review of scientific and medical manuscripts became common only in the late-20th century. Perhaps the most influential general interest science journal of our time, Nature, only instituted peer review in 1967 [9]. Today, peer review is ubiquitous among the better journals of our specialty, and at CORR®, we work hard to get it right. Thoughtful observers have raised questions about its efficacy [4, 11], but good evidence substantiates that peer review improves both article quality and clarity [3, 10]. Insofar as peer review is a decentralized system that depends in large measure on the work of volunteers, no doubt it can be improved, and this remains an area of active research. Simple reviewer training does not seem to make an impact [5], while certain labor-intensive add-ons might [1]. At CORR®, we are developing a systematic refinement to guide reviewers through the editorial process. The idea is to create a tool that can walk a reviewer through the key elements of each article in such a way that the reviewer does not need to remember lessons learned in a lecture or training session. We hope that this tool, which will be used during the “routine” review process, may result in improved quality and consistency of reviews compared to simple mentoring or the implementation of still-more-rigid guidelines. Since we depend on peer review to make good decisions, and because the process requires a considerable investment of time and effort, it is important to ask whether peer review can be improved upon. Alternatives to traditional review include open peer review (allowing authors to know the identities of reviewers), postpublication peer review, and portable peer review, among many others. We need to remain mindful of the shortcomings of traditional peer review, open minded about other options, and willing to invest in thoughtful experiments on the topic when appropriate. CORR® is currently involved with a large-scale effort to investigate modifiable nonscientific elements of manuscript presentation that can influence reviewers’ decisions, and we hope to have results to share on the topic in the next year or so. Although high-quality peer review is essential to a robust editorial process, it is neither a direct democracy nor an end in itself. On the former, thoughtful editors must do more than count votes, if for no other reason than reviewers often disagree about the applicability or validity of a paper's methods, research questions, and conclusions. Rather than tallying opinions, editors need to read the reviews in detail, and use their content to determine whether articles are interesting, important, and methodologically sound. Quality reviews provide essential subject matter expertise to guide good editorial decisions. But reviewers’ manuscript-disposition recommendations are advisory to the editors; they do not constitute a binding vote of any kind. One substantive review identifying an article's irremediable methodological flaws can and should sway a decision, even if two other more-cursory reviews missed those flaws and favored publication. While we commonly request additional reviews even after several have come in, we generally do so not to “break ties,” but to ensure we have a balanced perspective from the best possible experts on whether a paper is interesting enough to deserve the attention of our readers, or to verify the validity of subspecialized methodological elements. An effective editorial process begins with peer review, but it does not end there. Reviewers with special subject-matter knowledge help us decide when papers on unusual topics are important or interesting enough to warrant attention from our readers. They also offer invaluable technical guidance on complex or uncommon methodological elements. But most papers study familiar topics using straightforward study designs. Specifically, the large majority of orthopaedic papers evaluate treatment approaches, and the large majority of those are case series [2], the “bones” of which are relatively similar [8] regardless of whether the topic is hip replacement or carpal tunnel release. Selection bias and study inclusion criteria, loss to followup, and outcomes assessment are essential to the validity of studies about surgical treatments, and good editors must ask about these elements whether or not the reviewers have. Editors should ensure that studies do not overstate their conclusions, or fail to disclose and discuss key limitations. Finally, editors work with authors to ensure that the presentation is as clear as it can be. Corporate standards for methodological rigor, completeness, and clarity are central to the integrity of a journal's brand, and so are squarely within the purview of its editors. Editors who ask questions beyond those posed by reviewers are not violating the spirit of peer review, they are completing an important process. Scientists, reviewers, and editors, will inevitably err. The last step in the process - no less important than any other - is the evaluation of new work by a thoughtful community of readers. Ideas (including those in papers that have passed through peer review), in time, get replaced - either incrementally or in revolutionary fashion - through further experiments that may fill in the blanks, correct the errors, or even shift the paradigm [7]. Unless something changes substantially about the way scientists evaluate new ideas, these new ideas themselves probably will pass through peer review. We think that is a good thing. Often, the first inkling we get that an idea in an article is not quite right comes in the form of a letter to the editor. Send yours to [email protected]. Acknowledgment I am grateful to Lee Beadling BS, whose thoughtful suggestions improved this essay.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 16 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.