Before this: Append files under a declared schema
The idea
When monthly files are stacked into one table, the origin of each row disappears unless you keep it. Add a source_file column during the append.
Record each delivery's name, size, checksum and row count. A repeated checksum means the same file was delivered twice; a row count far from usual means a truncated export.
Lineage lets you remove or reload one bad delivery without touching the rest.
Worked example
Input: jan.csv (3 rows), feb.csv (2 rows)
- Visual
Add step → Append tables → feb → (source file added automatically)
- SQL (DuckDB)
SELECT *, 'jan' AS source_file FROM jan UNION ALL BY NAME SELECT *, 'feb' AS source_file FROM feb
- pandas
pd.concat([jan.assign(source_file="jan"), feb.assign(source_file="feb")], ignore_index=True)
Result: 5 rows, each tagged with its file
Common mistakes
- Appending by position.
- Losing the source of each row.
- Loading the same delivery twice.
Check yourself
What does a repeated checksum tell you?
The same file was delivered twice; loading it again would duplicate rows.
Practice it
Append monthly files under the declared schema.