Data Cleaning Exercises

Reference · Visual workspace

Remove duplicates with a survivor rule in Visual workspace

Keep one row per key, chosen by a complete deterministic order, and report the others.

When to use it

Repeated records at the table's grain: re-sent rows, versions, the same customer from several systems.

How

Add step → Rows → Remove duplicates. Tick the key columns, add order columns for which row survives, and give a reason for the others.

One row per (order_id, line_id), keep first by version desc
  • Convert columns to their types first so 10 sorts above 9.
  • Missing key values group together; require a value first if they must not be merged.

Pitfalls

  • Deduplicating at the wrong grain.
  • An incomplete order that leaves ties to chance.
  • Comparing numbers or timestamps as text.

Lesson: Find the row grain before removing duplicates

Practise it: Deduplicate order lines at the right grain