Examining the Predictive Validity of NIH Peer Review Scores

Mark D. Lindner · Richard Nakamura

24 citations0/0 files storedPLoS ONE

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

scientometrics and bibliometrics research Research, Science, and Academia Psychology Research and Bibliometrics

Abstract

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 24 citations exist upstream.

References

2005Why Most Published Research Findings Are False 2013Power failure: why small sample size undermines the reliability of neuroscience 2011False-Positive Psychology 2005Why Most Published Research Findings Are False 2009Avoidable waste in the production and reporting of research evidence 1998HARKing: Hypothesizing After the Results are Known 2011Believe it or not: how much can we rely on published data on potential drug targets?2014Increasing value and reducing waste in research design, conduct, and analysis

Cited by

2017Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association 2020Reading Peer Review 2017Addressing Reproducibility: Peer Review, Impact Factors, Checklists, Guidelines, and Reproducibility Initiatives 2023Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.