The language of peer review reports on articles published in the BMJ, 2014–2017: an observational study

Abstract

To analyse the words and expressions used in peer reviews of manuscripts that were later published as original research in the BMJ. Secondary aims were to estimate the differences in net sentiment between peer review reports on manuscripts subject to one or more rounds of peer review and and review reports on initially rejected manuscripts that were accepted after appeal. This observational study included all peer review reports published in the BMJ from September 2014 until the end of 2017. The study analysed the frequency of specific words in peer review reports for accepted manuscripts, identifying the most commonly occurring positive and negative words and their context, as well as the most common expressions. It also quantified differences in net sentiment in peer review reports between manuscripts accepted after appeal and manuscript accepted without appeal. The dataset consisting of 1716 peer review reports contained 908,932 word tokens. Among the most frequent positive words were “well”, “important”, “clear”, “while the negative words included “risk”, “bias”, and “confounding”. The areas where the reviewer makes the most positive and negative comments included: “well-written paper”, “well-written manuscript”, “this is an important topic”, “answers an important question”, “high risk of bias” and “selection bias”. The sentiment analysis revealed that manuscripts accepted after appeal had lower scores on review reports for joy and positive sentiment, in addition to having higher scores for negative words expressing sadness, fear, disgust and anger compared with manuscripts that were not initially rejected. Peer review comments were mainly related to methodology rather than the actual results. Peer review reports on initially rejected manuscripts were more negative and more often included expressions related to a high risk of bias.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 20 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.