Before this: Rules you can run twice
The idea
Sources change without warning: a renamed column, a new one, numbers sent as text.
Check each input against the declared schema before cleaning: required columns present, unexpected columns handled by rule, types parseable.
Fail early with a message naming the file and the problem, instead of producing a half-empty output.
Worked example
Input: orders.csv missing 'amount'
- Visual
Append tables → schema check rejects the file
- SQL (DuckDB)
SELECT column_name FROM (DESCRIBE SELECT * FROM orders)
- pandas
missing = {"id", "amount"} - set(df.columns)
Result: Stop: orders.csv lacks required column amount
Common mistakes
- Checking only on the first run.
- Silently ignoring new columns.
- Vague error messages.
Check yourself
Why fail early?
A clear stop is cheaper than a wrong report discovered later.
Practice it
Append monthly files under a declared schema.