Before this: Merge on evidence, review the rest
The idea
Comparing every record with every other grows quadratically. Blocking compares only records that share a cheap key, such as postcode or the first letters of a normalized name.
Choose blocking keys that rarely differ for true matches, and check how many true pairs a block would miss.
Worked example
Input: 10,000 customers
- Visual
(concept) Group by postcode → compare within groups
- SQL (DuckDB)
SELECT a.id, b.id FROM c a JOIN c b ON a.postcode = b.postcode AND a.id < b.id
- pandas
pairs = df.merge(df, on="postcode").query("id_x < id_y")
Result: Millions of pairs reduced to thousands
Common mistakes
- Comparing all pairs.
- Blocking on unreliable fields.
Check yourself
What is the risk of a blocking key?
True matches that differ on the key are never compared.
Practice it
Resolve records by normalized email.