Data Cleaning Exercises

Exercise · intermediate · 35 min

Clean an order-line export

Trim padded customer codes without losing leading zeros, parse amounts exactly, reject timestamps without an offset, quarantine unknown customers, and keep the latest copy of each duplicated line.

The rules

These rules decide what a correct result is. Your answer is checked on this dataset and on hidden datasets that follow the same rules.

Inputs
  orders(ingest_id, order_id, line_no, customer_code, order_ts, amount, status) — all text, any cell may be missing.
  customers(customer_code, region) — already validated: codes are trimmed and unique when present; a code may be missing.

Normalize
  customer_code: remove ASCII whitespace (space, tab, CR, LF) at both ends; an empty result is missing. Keep leading zeros.
  status: remove ASCII whitespace at both ends, then lowercase.
  amount: remove ASCII whitespace at both ends.
  line_no: whole number.

Quarantine each invalid row to exceptions with the FIRST failing reason, checked in this order:
  1. bad_timestamp     order_ts is missing, or is not exactly YYYY-MM-DD, "T" or one space, HH:MM[:SS[.f up to 6 digits]],
                       then an explicit offset (Z or ±HH:MM, at most ±14:00); hour 00–23, minute/second 00–59;
                       the date must exist on the calendar. A timestamp without an offset is invalid (never assume a zone).
  2. bad_amount        amount is missing or not: optional "-", 1–10 digits, optionally "." and 1–2 digits. Refunds (negative) are valid.
  3. bad_status        status is not one of: shipped, pending, cancelled.
  4. unknown_customer  customer_code is missing or has no matching customer. A missing code never matches a missing code.

Deduplicate the remaining rows
  The grain is (order_id, line_no). Keep the row with the latest order_ts instant; on a tie keep the smallest ingest_id
  (code-point order). Every other row goes to exceptions with reason "duplicate".

Outputs (row order does not matter)
  clean(order_id text, line_no int64, customer_code text, region text, order_ts timestamptz [UTC], amount decimal(12,2), status text)
  exceptions(ingest_id text, reason text)

The data

orders · 43 rows, first 12 shown

ingest_idorder_idline_nocustomer_codeorder_tsamountstatus
ing-1010-rtgSO-0001110056422026-11-01T01:30:00-05:00·42.10·pending
ing-1027-nwaSO-0001810056422026-02-30T10:00:00ZN/Apending
ing-1006-eedSO-0001630030242026-01-15T10:00:00+01:00007.5shipped!
ing-1014-uy3SO-0001120062812026-03-08T01:30:00-05:0012.50·Shipped
ing-1037-di1SO-0001210056422026-13-01T00:00:00Z100pending
ing-1033-rvdSO-000153·002518·2026-06-30T23:59:59.123456+05:300·Shipped
ing-1011-jdmSO-0001310030242024-02-29T12:00:00Z0shipped
ing-1008-qpfSO-000232·003024·2026-11-01T01:30:00-05:00007.5pending
ing-1039-cr5SO-000101·008814·2026-11-01T01:30:00-04:009999999999.99·Shipped
ing-1026-j7aSO-0001320088142024-02-29T12:00:00Z0shipped
ing-1007-m5fSO-0001720076002026-11-01T01:30:00-04:00-0cancelled·
ing-1013-niSO-0002430044752026-06-30T23:59:59.123456+05:305.PENDING

customers · 9 rows

customer_coderegion
001419South
002518Île-de-France
003024West
004475Île-de-France
005642São Paulo
006281São Paulo
007600West
008814São Paulo
missingUnassigned

What you practise

  • trimming
  • type conversion
  • quarantine
  • lookups
  • deduplication

Your answer is also checked on hidden datasets generated from the same rules.

Download this dataset (CSV) with its data dictionary

Sign in to bookmark this and keep private notes.