Article2013

HadoopProv: towards provenance as a first class citizen in MapReduce

Sherif Akoush · Ripduman Sohan · Andy Hopper

53 citations0/0 files stored

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Scientific Computing and Data Management Cloud Computing and Resource Management Distributed and Parallel Computing Systems

Abstract

We introduce HadoopProv, a modified version of Hadoop that implements provenance capture and analysis in MapReduce jobs. It is designed to minimise provenance capture overheads by (i) treating provenance tracking in Map and Reduce phases separately, and (ii) deferring construction of the provenance graph to the query stage. Provenance graphs are later joined on matching intermediate keys of the Map and Reduce provenance files. In our prototype implementation, HadoopProv has an overhead below 10% on typical job runtime (

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 53 citations exist upstream.

References

2011RAMP

Cited by

2016Big Data Storage 2020Improving reproducibility of data science pipelines through transparent provenance capture 2016Crossing analytics systems: A case for integrated provenance in data lakes 2021LIMA: Fine-grained Lineage Tracing and Reuse in Machine Learning Systems 2016Big Data Provenance: State-Of-The-Art Analysis and Emerging Research Challenges 2018A survey of simulation provenance systems: modeling, capturing, querying, visualization, and advanced utilization 2016Provenance as Essential Infrastructure for Data Lakes 2016Scaling SPADE to Big provenance

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.