Data Cleaning Exercises

Reference · Python (pandas)

Convert a column's type in Python (pandas)

Parse text into a typed value (integer, exact decimal, date, timestamp with time zone, boolean) and decide what happens to values that don't parse.

When to use it

Every column that will be compared, summed or sorted as something other than text.

How

Use explicit conversions: astype("Int64") for nullable integers, decimal.Decimal for money, pd.to_datetime(..., utc=True, format=...) for timestamps.

df["amount"] = df["amount"].map(lambda s: Decimal(s).quantize(Decimal("0.01")))
  • pd.to_numeric(errors="coerce") turns bad values into NaN silently; count them before accepting that.
  • astype(float) is wrong for money.

Pitfalls

  • Letting a parser guess formats.
  • Silently turning bad values into missing ones.
  • Floating point for money.

Lesson: Exact decimals for money

Practise it: Flag large transactions, keep refunds