Data Cleaning Exercises

Reference · Python (pandas)

Check allowed values in Python (pandas)

Quarantine rows whose value is not in an approved list.

When to use it

Status codes, countries and other controlled vocabularies.

How

Series.isin(values) gives a boolean mask.

bad = ~df["status"].isin(["shipped", "pending", "cancelled"])
  • isin returns False for missing values, so they count as not allowed.

Pitfalls

  • Forgetting how missing values behave.
  • Checking before normalizing case and spacing.

Lesson: Quarantine with reasons, in rule order

Practise it: Validate quantities and prices against ranges