Guide · updated 2026-11-04
Reproducible cleaning pipelines
Immutable raw data, idempotent steps and tests that make reruns safe.
Raw data is evidence
Never edit raw files. Rebuild every output from them so a fixed rule applies to all history.
Make steps deterministic
Use complete sort orders with unique tie-breaks, pass dates as parameters instead of reading the clock, and avoid random sampling.
Test invariants
Check row reconciliation, key uniqueness, required columns and allowed values on every run, and compare outputs before and after a rule change.
Practise it
- Lesson: Rules you can run twice
- Exercise: Deduplicate order lines at the right grain