Before this: Profile before you change anything
The idea
Scope says which records belong: which period, which business units, which statuses. Grain says what one row means: an order, an order line, a daily reading.
Write both down at the start. Rows outside the scope are filtered with a declared rule and counted, not mixed with exceptions.
Worked example
Input: orders from 2025 and 2026; brief asks for 2026
- Visual
Add step → Filter rows → order year = 2026 (count the rest as out of scope)
- SQL (DuckDB)
SELECT * FROM orders WHERE order_ts >= TIMESTAMPTZ '2026-01-01 00:00:00+00'
- pandas
df[df["order_ts"] >= pd.Timestamp("2026-01-01", tz="UTC")]
Result: 2026 rows kept; 2025 rows counted as out of scope
Common mistakes
- Starting without a defined grain.
- Counting out-of-scope rows as errors.
Check yourself
Why separate out-of-scope rows from exceptions?
They aren't errors; mixing them hides real data problems.
Practice it
Deduplicate order lines at the right grain.