Guide · updated 2026-11-04
Deduplication and entity resolution without false merges
Start with the grain
The grain says what one row represents. Order lines have the grain (order_id, line_id); an account table has one row per account. Duplicates are rows with the same key at that grain. Deduplicating on fewer columns deletes real rows.
Make the survivor deterministic
List the survivor rule as a complete order: latest instant first, then a business priority, then a unique tie-break such as the lowest id. Compare typed values, not text. Keep the losing rows in an exception table with a reason.
Entity resolution is about evidence
Different systems describe the same customer with different ids. Exact, normalized identifiers such as an email after trimming and lowercasing are strong evidence. Names, cities and birth years are weak evidence and shared by many people.
Prefer review over a false merge
A false merge mixes two customers' orders, addresses and consent, and it is hard to undo. Merge automatically only on strong evidence; send weaker candidate pairs and records without an identifier to a human review queue.