Data Cleaning Exercises

Module 18 · Python and pandas cleaning · 6 min

merge with validate and indicator

You'll be able to: Use merge options that make join problems visible.

Before this: Override the pandas defaults that change your data

The idea

validate="many_to_one" raises if the right-hand keys repeat. indicator=True adds a _merge column showing left_only, right_only or both.

Use them together: fan-out fails loudly, orphans are easy to report.

Worked example

Input: payments × accounts

Visual
Look up accounts → unmatched quarantined
SQL (DuckDB)
LEFT JOIN accounts USING (account)
pandas
pay.merge(acc, on="account", how="left", validate="many_to_one", indicator=True)

Result: _merge == 'left_only' rows become unknown_account exceptions

Common mistakes

  • Silent fan-out.
  • Ignoring left_only rows.

Check yourself

What does indicator=True add?

A _merge column with left_only, right_only or both.

Practice it

Match payments to accounts.

Next: Referential integrity with explicit exceptions