Guide · updated 2026-11-04
Appending monthly files and handling schema drift
Declaring required and optional columns, rejecting broken files and keeping lineage.
Declare the schema
List required columns, optional columns and their types. Align files by column name, not position.
Reject, don't half-load
A file missing a required column is rejected with a reason. Loading it with blanks hides the problem until someone reports wrong totals.
Keep lineage
Add the source file to every row and record each delivery's checksum so the same file is never loaded twice.
Practise it
- Lesson: Append files under a declared schema
- Lesson: Rules you can run twice
- Exercise: Append monthly files under a declared schema