Before this: Lineage for appended files
The idea
The same export can arrive twice under different names. Compare checksums and row counts of deliveries.
Make loading idempotent: a delivery with a known checksum is skipped and reported.
Worked example
Input: feb.csv and feb_final.csv, same checksum
- Visual
(concept) delivery log → skip known checksum
- SQL (DuckDB)
SELECT md5(content) FROM deliveries
- pandas
hashlib.sha256(open(f, "rb").read()).hexdigest()
Result: feb_final.csv skipped as a repeat
Common mistakes
- Relying on file names.
- Appending twice.
Check yourself
What identifies a repeated delivery?
Its checksum.
Practice it
Append monthly files under a declared schema.