An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Abstract

ABSTRACT Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. Redaction reduced, but did not eliminate, reviewers’ ability to correctly guess features of identity. The primary, pre-registered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not pre-registered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators; redaction reduced the size of the difference by about half (e.g. from a Cohen’s d of 0.20 to 0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. The primary statistical test of the study hypothesis was not significant; the secondary analysis was significant. The findings support further evaluation of peer review models that diminish the influence of applicant identity.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 38 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.