Data Cleaning Exercises

Reference · Python (pandas)

Flag rows in Python (pandas)

Add a true/false column that marks rows matching a condition, keeping every row.

When to use it

Unusual but possibly legitimate values that need human review.

How

A boolean Series assigned to a new column.

df["needs_review"] = df["amount"] > Decimal("5000.00")
  • Convert to bool explicitly if the comparison can produce <NA>.

Pitfalls

  • Deleting or capping instead of flagging.
  • Getting the boundary wrong (> versus >=).

Lesson: Flag outliers, don't delete them

Practise it: Flag large transactions, keep refunds