Before this: Flag outliers, don't delete them
The idea
An outlier is unusual; an error breaks a rule. A sale of 48,000 is an outlier; a quantity of -3 when quantities must be positive is an error.
Errors go to exceptions with a reason. Outliers stay in the data, flagged for review.
Context decides: a negative amount is an error in a price list and normal in a refund ledger.
Worked example
Input: quantity: 2, 480, -3 (rule: 1 to 500)
- Visual
Add step → Reject rows where quantity < 1; Flag rows quantity > 100
- SQL (DuckDB)
CASE WHEN q < 1 THEN 'quantity_out_of_range' END, q > 100 AS needs_review
- pandas
df["needs_review"] = df["q"] > 100
Result: -3 → exception; 480 → kept and flagged
Common mistakes
- Deleting outliers.
- Flagging rule violations instead of quarantining them.
Check yourself
Is a very large value an error?
Only if it breaks a declared rule; otherwise it is an outlier to flag.
Practice it
Validate quantities and prices against ranges.