Three Dimensions of Reproducibility in Natural Language Processing

Kevin Bretonnel Cohen · Jingbo Xia · Pierre Zweigenbaum · Tiffany J. Callahan · Orin Hargraves · Foster R. Goss +4 more

43 citations0/0 files storedLanguage Resources and Evaluation Conference (LREC)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Scientific Computing and Data Management Research Data Management Practices Natural Language Processing Techniques

Abstract

Despite considerable recent attention to problems with reproducibility of scientific research, there is a striking lack of agreement about the definition of the term.That is a problem, because the lack of a consensus definition makes it difficult to compare studies of reproducibility, and thus to have even a broad overview of the state of the issue in natural language processing.This paper proposes an ontology of reproducibility in that field.Its goal is to enhance both future research and communication about the topic, and retrospective meta-analyses.We show that three dimensions of reproducibility, corresponding to three kinds of claims in natural language processing papers, can account for a variety of types of research reports.These dimensions are reproducibility of a conclusion, of a finding, and of a value.Three biomedical natural language processing papers by the authors of this paper are analyzed with respect to these dimensions.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 43 citations exist upstream.

References

2015Estimating the reproducibility of psychological science 2016What does research reproducibility mean?2007Full publication of results initially presented in abstracts 1994Full publication of results initially presented in abstracts. A meta-analysis 2009Replicability is not Reproducibility:Nor is it Good Science 2008Enzyklopädie Philosophie und Wissenschaftstheorie 2012Replicates and repeats—what is the difference and is it significant?1999The fate of neuroradiologic abstracts presented at national meetings in 1993: rate of subsequent publication in peer-reviewed, indexed journals.

Cited by

2022French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English 2020Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites 2018Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process 2022A Metrological Perspective on Reproducibility in NLP*2019

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.