Cloud-Native Repositories for Big Scientific Data

Abstract

Scientific data has traditionally been distributed via the "download model," in which scientists bring datasets to the personal computers for analysis. But this way of working suffers from major limitations as scientific datasets grow towards the Petabyte scale. This article discusses the potential of cloud computing to accelerate scientific research and defines the concept of a "cloud native data repository," distinct from a traditional data repository. We enumerate the objectives for such repositories-performance, reliability, cost-effectiveness, collaboration, reproducibility, creativity, downstream impacts, and access & inclusion-and use these objectives to define a set of best practices for cloud native data repositories, focusing on the importance of analysis-ready data, cloud-optimized formats, and loose coupling with data-proximate computing. We describe a prototype implementation of these principles by the Pangeo Project using open source scientific python tools. We conclude by discussing some practical challenges for the future development of cloud native data repositories.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 127 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.