Before this: Trim the right characters, and only those
The idea
Pipelines are re-run: a new batch arrives, a bug is fixed, a report is rebuilt. A step is idempotent when running it twice changes nothing the second time. Trim and lowercase are idempotent; "prepend a zero" or "subtract the fee" are not.
Keep the raw file immutable and write cleaned output somewhere else. Then any run can start from the same input, and you can always compare raw and clean to explain a change.
Test the pipeline on a new batch, not just the file you built it on. Hidden-dataset grading in these exercises is the same idea: rules must hold for data you haven't seen.
Worked example
Input: email: " Sam+Offers@Example.com"
- Visual
Trim (ASCII whitespace) → Change case (lowercase). Run the steps again on the result: nothing changes.
- SQL (DuckDB)
lower(trim(email, ' ' || chr(9) || chr(13) || chr(10)))
- pandas
df["email"].str.strip(" \t\r\n").str.lower()
Result: "sam+offers@example.com" both after one run and after two.
Common mistakes
- Overwriting the raw input with cleaned data.
- Steps that change already-clean values on re-run.
- Testing only on the file the pipeline was built on.
Check yourself
Is "add 1 to the version number" idempotent?
No. Each run changes the value again.
Why keep the raw file unchanged?
Every run can start from the same input, and differences between raw and clean stay explainable.
Practice it
Apply the declared email policy so that running it again on its own output changes nothing.