Before this: Flag outliers, don't delete them
The idea
A flag is a new column, such as needs_review, set by a stated rule: amount above 5000.00, quantity above the 99th percentile of last year's data.
Flagging keeps the population intact: totals still include the value, and reviewers can filter. Deleting would change the report and hide real events.
State the threshold, where it came from and how many rows it flagged.
Worked example
Input: amount: 120.00, 48000.00, -35.00
- Visual
Add step → Flag rows → needs_review where amount > 5000.00
- SQL (DuckDB)
CAST(amount AS DECIMAL(12,2)) > 5000.00 AS needs_review
- pandas
df["needs_review"] = df["amount"] > Decimal("5000.00")
Result: Only 48000.00 is flagged; the refund stays unflagged and included
Common mistakes
- Filtering instead of flagging.
- Comparing amounts as text.
- Thresholds with no documented source.
Check yourself
Is exactly 5000.00 flagged by 'greater than 5000.00'?
No; the rule is strictly greater than.
Practice it
Flag large transactions and keep refunds.