Can peer review accolade awards motivate reviewers? A large-scale quasi-natural experiment

Houqiang Yu · Yian Liang · Yinghua Xie

15 citations0/0 files storedHumanities and Social Sciences Communications

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Publishing and Scholarly Communication scientometrics and bibliometrics research Psychology Research and Bibliometrics

Abstract

The utilization of accolade awards to motivate reviewers is widespread, but their effectiveness remains uncertain. We aim to explore how receiving an accolade award affects reviewers’ subsequent number of reviews. In contrast to small-scale, unrepresentative experiments, we perform a large-scale, global, and all-disciplinary analysis based on quasi-natural experiments. By integrating the Publons, ORCID, MAG, SciSciNet, and OpenAlex databases, a large dataset that tracks the reviewers’ annual number of reviews and their bibliometric indicators is compiled, encompassing 179,794 individuals. Among them, 6605 individuals who receive the “Publons Global Peer Review Award” in 2018 are constituted as the experimental group. Those non-winners are then matched to form a control group. Using propensity score matching (PSM), 711 and 762 reviewers are matched as the experimental and control groups respectively. An analysis employing the difference-in-differences (DiD) method is conducted to examine the impact of an accolade award on reviewers’ subsequent number of reviews. It is found that, following the receipt of an accolade award, reviewers, on average, reviews about four fewer manuscripts, with the reduction exhibiting a V-shaped pattern. Additional analyses are conducted to examine how individual differences and socio-economic factors influence the awarding effects. Besides, we analyze the mechanisms underlying the awarding effect and propose strategies to motivate reviewers. In brief, the sharp decrease in marginal utility of accolades, the voluntary nature of peer review, and the unexpected properties of accolade awards are the potential mechanisms that generate the negative effect. The academic community should reassess the existing incentive strategies for reviewers.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 15 citations exist upstream.

References

2020Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions 2012Self-Determination Theory 2001Motivation Crowding Theory 2011When and Why Incentives (Don't) Work to Modify Behavior 2018Science of science 2013The Matilda Effect in Science Communication 2020The limitations to our understanding of peer review 2014Measuring the effectiveness of scientific gatekeeping

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.