Real-time data warehouse: a solution and evaluation

Abstract

Real-time warehousing has been researched and tried in practice by several companies, but there should be a definition of an architecture and an evaluation of its merits. Based on both industry and theoretical insights, we define a data warehouse architecture for constant integration of data without compromising query performance, and we evaluate its capacity to provide realtime. For this real-time data warehouse (RTDW) we define a dynamic warehouse component and a static warehouse component to represent the recently integrated data and the rest of the data, respectively, with relevant choices concerning how the components merge together. We propose design choices concerning query computation mechanisms and evaluate the alternatives to conclude which is the most efficient implementation of those mechanisms. The realtime data warehouse architecture is evaluated with a realtime benchmark setup that considers online loading with simultaneous query workload processing. Results prove the validity of the architecture, compare different mechanisms and quantify its efficiency in the constant integration context. The performance is carefully analysed taking into account several important factors.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 10 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.