Data Cleaning Exercises

Module 15 · Outliers and context · 6 min

Robust statistics

You'll be able to: Use medians and quantiles to describe data that contains extremes.

Before this: Flag outliers, don't delete them

The idea

A single extreme value pulls the mean far away; the median barely moves. For skewed data such as purchase amounts, report the median and a few quantiles.

Robust statistics describe the data without deleting anything.

Worked example

Input: amounts: 10, 11, 12, 13, 500

Visual
Add step → Group and total → median(amount)
SQL (DuckDB)
SELECT median(amount), quantile_cont(amount, 0.9) FROM t
pandas
df["amount"].median(), df["amount"].quantile(0.9)

Result: median 12; mean would be 109.2

Common mistakes

  • Reporting only the mean for skewed data.
  • Removing extremes to fix the mean.

Check yourself

Which is more robust to one extreme value, mean or median?

The median.

Practice it

Fill gaps with a median learned from training data only.

Fill gaps with a median learned from training data only · VisualFill gaps with a median learned from training data only · SQLFill gaps with a median learned from training data only · Python

Next: Fill gaps with a median learned from training data only