Open-source code repository attributes predict impact of computer science research

Abstract

With an increased importance of transparency and reproducibility in computer science research, it has become common to publicly release open-source repositories that contain the code, data, and documentation alongside a publication. We study the relationship between transparency of a publication (as represented by the attributes of its open-source repository) and its scientific impact (as represented by paper citations). Using the Mann-Whitney test and Cliff's delta, we observed a statistically significant difference in citations between papers with and without an associated open-source repository. We also observed a statistically significant correlation (p < 0.01) between citations and several repository interaction features: Stars, Forks, Subscribers and Issues. Finally, using time-series features of repository growth (Stars), we trained a classifier to predict whether a paper would be highly cited (top 10%) with cross-validated AUROC of 0.8 and AUPRC of 0.65. Our results provide evidence that those who make sustained efforts in making their works transparent also tend to have a higher scientific impact.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 7 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.