Substantial Agreement of Referee Recommendations at a General Medical Journal – A Peer Review Evaluation at Deutsches Ärzteblatt International

Christopher Baethge · Jeremy G. Franklin

32 citations0/0 files storedPLoS ONE

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Publishing and Scholarly Communication Academic Writing and Publishing Meta-analysis and systematic reviews

Abstract

BACKGROUND: Peer review is the mainstay of editorial decision making for medical journals. There is a dearth of evaluations of journal peer review with regard to reliability and validity, particularly in the light of the wide variety of medical journals. Studies carried out so far indicate low agreement among reviewers. We present an analysis of the peer review process at a general medical journal, Deutsches Ärzteblatt International. METHODOLOGY/PRINCIPAL FINDINGS: 554 reviewer recommendations on 206 manuscripts submitted between 7/2008 and 12/2009 were analyzed: 7% recommended acceptance, 74% revision and 19% rejection. Concerning acceptance (with or without revision) versus rejection, there was a substantial agreement among reviewers (74.3% of pairs of recommendations) that was not reflected by Fleiss' or Cohen's kappa (<0.2). The agreement rate amounted to 84% for acceptance, but was only 31% for rejection. An alternative kappa-statistic, however, Gwet's kappa (AC1), indicated substantial agreement (0.63). Concordance between reviewer recommendation and editorial decision was almost perfect when reviewer recommendations were unanimous. The correlation of reviewer recommendations and citations as counted by Web of Science was low (partial correlation adjusted for year of publication: -0.03, n.s.). CONCLUSIONS/SIGNIFICANCE: Although our figures are similar to those reported in the literature our conclusion differs from the widely held view that reviewer agreement is low: Based on overall agreement we consider the concordance among reviewers sufficient for the purposes of editorial decision making. We believe that various measures, such as positive and negative agreement or alternative Kappa values are superior to the application of Cohen's or Fleiss' Kappa in the analysis of nominal or ordinal level data regarding reviewer agreement. Also, reviewer recommendations seem to be a poor proxy for citations because, for example, manuscripts will be changed considerably during the revision process.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 32 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 1990High agreement but low Kappa: I. the problems of two paradoxes 2006Computing inter‐rater reliability and its variance in the presence of high agreement 1990High agreement but low kappa: II. Resolving the paradoxes 2009Comparisons of Citations in Web of Science, Scopus, and Google Scholar for Articles Published in General Medical Journals 1990The Philosophical Basis of Peer Review and the Suppression of Innovation 1990Peer Review in 18th-Century Scientific Journalism 1990The Evolution of Editorial Peer Review

Cited by

2017Inter-rater reliability and validity of peer reviews in an interdisciplinary field 2015Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research 2021A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018 2022Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length 2017The review process in tourism academia: An elaboration of reviewers' extrinsic and intrinsic motivations

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.