Reproducibility and Replicability in a Fast-paced Methodological World

Abstract

Methodological developments and software implementations progress in increasingly faster time frames. The introduction and widespread acceptance of preprint archived reports and open-source software make state-of-the-art statistical methods readily accessible to researchers. At the same time, researchers increasingly emphasize that their results should be reproducible (obtaining the same numeric results at a later stage), which is a basic requirement for assessing the replicability (obtaining the same conclusions at a later stage) of results. Although this age of fast-paced methodology greatly facilitates reproducibility and replicability, it also undermines them in ways not often realized by researchers. The goal of this paper is to draw researchers’ attention to these threats and to propose guidelines to help minimize their impact. Reproducibility may be influenced by software development and change over time: a problem that is greatly compounded by the rising dependency between software packages. Replicability is impacted by rapidly changing standards, researcher degrees of freedom, and possible bugs or errors in a code made by software developers or empirical researchers implementing an analysis. This paper concludes with a list of recommendations for methodologists to improve the reproducibility and replicability of their results.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 7 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.