Tracking Replicability as a Method of Post-Publication Open Evaluation

Joshua K. Hartshorne · Adena Schachner

107 citations0/0 files storedFrontiers in Computational Neuroscience

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Academic Publishing and Open Access Publishing and Scholarly Communication scientometrics and bibliometrics research

Abstract

Recent reports have suggested that many published results are unreliable. To increase the reliability and accuracy of published papers, multiple changes have been proposed, such as changes in statistical methods. We support such reforms. However, we believe that the incentive structure of scientific publishing must change for such reforms to be successful. Under the current system, the quality of individual scientists is judged on the basis of their number of publications and citations, with journals similarly judged via numbers of citations. Neither of these measures takes into account the replicability of the published findings, as false or controversial results are often particularly widely cited. We propose tracking replications as a means of post-publication evaluation, both to help researchers identify reliable findings and to incentivize the publication of reliable results. Tracking replications requires a database linking published studies that replicate one another. As any such database is limited by the number of replication attempts published, we propose establishing an open-access journal dedicated to publishing replication attempts. Data quality of both the database and the affiliated journal would be ensured through a combination of crowd-sourcing and peer review. As reports in the database are aggregated, ultimately it will be possible to calculate replicability scores, which may be used alongside citation counts to evaluate the quality of work published in individual journals. In this paper, we lay out a detailed description of how this system could be implemented, including mechanisms for compiling the information, ensuring data quality, and incentivizing the research community to participate.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 107 citations exist upstream.

References

1971Belief in the law of small numbers.1991Publication bias in clinical research 1989Journal of Individual Psychology 1962The statistical power of abnormal-social psychological research: A review.2003One Hundred Years of Social Psychology Quantitatively Described 2008Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias 2005Contradicted and Initially Stronger Effects in Highly Cited Clinical Research 1999The Nurture Assumption: Why Children Turn Out the Way they Do

Cited by

2012Rewarding Replications 2018Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field 2012Rising Publication Delays Inflate Journal Impact Factors 2019Predicting translational progress in biomedical research 2016Estimating the Reproducibility of Psychological Science 2014Evidence-Based Management and the Trustworthiness of Our Cumulative Scientific Knowledge: Implications for Teaching, Research, and Practice

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.