Exercise · intermediate · 35 min
Clean an order-line export
Trim padded customer codes without losing leading zeros, parse amounts exactly, reject timestamps without an offset, quarantine unknown customers, and keep the latest copy of each duplicated line.
The rules
These rules decide what a correct result is. Your answer is checked on this dataset and on hidden datasets that follow the same rules.
Inputs
orders(ingest_id, order_id, line_no, customer_code, order_ts, amount, status) — all text, any cell may be missing.
customers(customer_code, region) — already validated: codes are trimmed and unique when present; a code may be missing.
Normalize
customer_code: remove ASCII whitespace (space, tab, CR, LF) at both ends; an empty result is missing. Keep leading zeros.
status: remove ASCII whitespace at both ends, then lowercase.
amount: remove ASCII whitespace at both ends.
line_no: whole number.
Quarantine each invalid row to exceptions with the FIRST failing reason, checked in this order:
1. bad_timestamp order_ts is missing, or is not exactly YYYY-MM-DD, "T" or one space, HH:MM[:SS[.f up to 6 digits]],
then an explicit offset (Z or ±HH:MM, at most ±14:00); hour 00–23, minute/second 00–59;
the date must exist on the calendar. A timestamp without an offset is invalid (never assume a zone).
2. bad_amount amount is missing or not: optional "-", 1–10 digits, optionally "." and 1–2 digits. Refunds (negative) are valid.
3. bad_status status is not one of: shipped, pending, cancelled.
4. unknown_customer customer_code is missing or has no matching customer. A missing code never matches a missing code.
Deduplicate the remaining rows
The grain is (order_id, line_no). Keep the row with the latest order_ts instant; on a tie keep the smallest ingest_id
(code-point order). Every other row goes to exceptions with reason "duplicate".
Outputs (row order does not matter)
clean(order_id text, line_no int64, customer_code text, region text, order_ts timestamptz [UTC], amount decimal(12,2), status text)
exceptions(ingest_id text, reason text)The data
orders · 43 rows, first 12 shown
| ingest_id | order_id | line_no | customer_code | order_ts | amount | status |
|---|---|---|---|---|---|---|
| ing-1010-rtg | SO-00011 | 1 | 005642 | 2026-11-01T01:30:00-05:00 | ·42.10· | pending |
| ing-1027-nwa | SO-00018 | 1 | 005642 | 2026-02-30T10:00:00Z | N/A | pending |
| ing-1006-eed | SO-00016 | 3 | 003024 | 2026-01-15T10:00:00+01:00 | 007.5 | shipped! |
| ing-1014-uy3 | SO-00011 | 2 | 006281 | 2026-03-08T01:30:00-05:00 | 12.50 | ·Shipped |
| ing-1037-di1 | SO-00012 | 1 | 005642 | 2026-13-01T00:00:00Z | 100 | pending |
| ing-1033-rvd | SO-00015 | 3 | ·002518· | 2026-06-30T23:59:59.123456+05:30 | 0 | ·Shipped |
| ing-1011-jdm | SO-00013 | 1 | 003024 | 2024-02-29T12:00:00Z | 0 | shipped |
| ing-1008-qpf | SO-00023 | 2 | ·003024· | 2026-11-01T01:30:00-05:00 | 007.5 | pending |
| ing-1039-cr5 | SO-00010 | 1 | ·008814· | 2026-11-01T01:30:00-04:00 | 9999999999.99 | ·Shipped |
| ing-1026-j7a | SO-00013 | 2 | 008814 | 2024-02-29T12:00:00Z | 0 | shipped |
| ing-1007-m5f | SO-00017 | 2 | 007600 | 2026-11-01T01:30:00-04:00 | -0 | cancelled· |
| ing-1013-ni | SO-00024 | 3 | 004475 | 2026-06-30T23:59:59.123456+05:30 | 5. | PENDING |
customers · 9 rows
| customer_code | region |
|---|---|
| 001419 | South |
| 002518 | Île-de-France |
| 003024 | West |
| 004475 | Île-de-France |
| 005642 | São Paulo |
| 006281 | São Paulo |
| 007600 | West |
| 008814 | São Paulo |
| missing | Unassigned |
What you practise
- trimming
- type conversion
- quarantine
- lookups
- deduplication
Your answer is also checked on hidden datasets generated from the same rules.
Sign in to bookmark this and keep private notes.