No open-access location is known for this record yet.
No open-access location is known for this record yet.
Reproducibility is the cornerstone of science. Many scientific communities have been struck by the reproducibility crisis, and computer science is no exception. Its answer has been to require artifact evaluations along with accepted articles and award badges to reward authors for their efforts to support ‘reproducibility.’ Authors voluntarily submit artifacts associated with a submission to reviewers who decide their ‘reproducibility’ properties. We argue that the notion of ‘reproducibility’ considered by such badges is limited and misses important aspects of the reproducibility crisis. In this article, we survey almost 300 articles from five leading conferences on parallel and distributed systems held in 2023 (CCGrid, EuroSys, OSDI, PPoPP, and SC). For each article, we gather information about its artifacts (how it was shared, under which experimental setup, and how the software environment was generated and shared), as well as the reproducibility badges awarded. By reviewing the methods and tools used to create and share artifacts in a technical, in-depth, and article content-agnostic manner, we found that the state of practice does not address reproducibility in terms of artifact longevity and we expose eight observations that support this finding. To address the longevity of artifacts, we propose a new badge based on source code, experimental setup, and software environment. These criteria will allow rewarding artifacts expected to withstand the test of time. This work aims to shed light on the issue of long-term reproducibility in parallel and distributed systems and to start a discussion in the community towards addressing the issue.
Loading…
Stored only when both papers are in the corpus — 5 citations exist upstream.
Papers that cite the same references — the relational fallback while embeddings are still backfilling.
Imported from OpenAlex (CC0) — nothing here is generated.