Data-Intensive Services for Large-Scale Archive Access

Abstract

Recently many organizations have accumulated data from such various sources as web and network sensors and constructed large-scale archives. Some would like to publish their archives to public to facilitate the activities of other organizations, but the scale of the archives causes problems. Therefore, we propose the concept of data-intensive services, which publish large-scale archives. We show the architecture for data-intensive services and focus on the following fundamental functional properties: 1) enhancing search, 2) preprocessing, 3) and asynchronous transfer. We also developed a reference implementation of a framework for data-intensive services and applied it to a web archive that contains about 2 billion documents and greatly improved the access performance to the web archive at small development cost.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 5 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.