Data Cleaning Exercises

Module 16

Reproducible pipelines

Immutable raw data, idempotent steps, new batches, tests and schema drift.

  1. Rules you can run twice

    Write cleaning steps that give the same result when re-run on cleaned data, and keep raw inputs untouched. · 7 min

  2. Deterministic steps

    Remove hidden sources of variation so a rerun on the same input gives the same output. · 7 min

  3. Schema checks on every run

    Detect schema drift at the start of a run and stop with a clear message. · 6 min

  4. Tests for cleaning code

    Write small tests that pin down rules and edge cases. · 7 min

Module 16 quiz

Practice questions: choose an answer and check it to see the explanation. Nothing here is timed or graded.

  1. 1. Where should cleaned data be written?
  2. 2. Why test a pipeline on a new batch, not only the file it was built on?
  3. 3. Why should cleaning rebuild outputs from raw input rather than editing yesterday's output?
  4. 4. A new delivery adds a column you didn't expect. What should the pipeline do?
  5. 5. A dedup step uses random sampling to break ties. What breaks?
  6. 6. Why keep raw files immutable?
  7. 7. How should a cleaning rule change be released?
  8. 8. A pipeline uses today's date to decide which rows are "recent". Reruns of last month give different results. Why?