A Comparison and Generalization of Blocking and Windowing Algorithms for Duplicate Detection

Abstract

Duplicate detection is the problem of identifying pairs of records that represent the same real world object, and could thus be merged into a single record. To avoid a prohibitively expensive comparison of all pairs of records, a common tech-nique is to carefully partition the records into smaller sub-sets. If duplicate records appear in the same partition, only all pairs within each partition must be compared. Two competing approaches are often cited: Blocking methods strictly partition records into disjoint subsets, for instance using zip-codes as partitioning key. Windowing methods, in particular the Sorted-Neighborhood method, sort the data according to some key, such as zip-code, and then slide a window of fixed size across the sorted data and compare pairs only within the window.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 34 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.