Rating intervals: an experiment in peer review

Abstract

The National Institute of Health (NIH) peer review process for research grant applications is one of the largest and most respected systems of its kind in the world. Recently, however, the distribution of raw priority scores voted by NIH study sections has been skewed, and the rating behavior of individual review groups has been quite variable. These phenomena have made funding decisions more difficult. To achieve greater uniformity of rating behavior and a broader description of scores, an experiment was conducted involving 24 study sections. Standard adjectival descriptors and standard rating scales were used. On a random basis, half of the study sections were instructed to vote in units of 0.1 while the other half used an interval of 0.5. The results of this study have now been translated into standard practice at NIH.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 13 citations exist upstream.