Data Cleaning Exercises

Module 18 · Python and pandas cleaning · 8 min

Nullable dtypes in practice

You'll be able to: Use Int64, boolean and string dtypes so integers stay exact and missing values stay explicit.

Before this: Override the pandas defaults that change your data

The idea

The classic int64 can't hold missing values, so pandas switches to float64 and 7 becomes 7.0. IDs and counts then print and compare wrongly.

Nullable dtypes fix this: Int64 (capital I), boolean and string all hold <NA>.

Convert explicitly: pd.to_numeric(s, errors="coerce").astype("Int64"), and check which values became <NA> during conversion.

Worked example

Input: duration_min: '10', '', '30'

Visual
Add step → Convert type → duration_min → Whole number (int64)
SQL (DuckDB)
TRY_CAST(NULLIF(trim(duration_min), '') AS BIGINT)
pandas
pd.to_numeric(df["duration_min"].str.strip().replace("", pd.NA), errors="coerce").astype("Int64")

Result: 10, <NA>, 30 as Int64 (not 10.0, NaN, 30.0)

Common mistakes

  • Accepting float ids.
  • Filling NaN with 0 to get ints back.
  • Not checking which values failed conversion.

Check yourself

What dtype does pd.Series([1, None]) get by default?

float64.

Practice it

Label unknown categories, keep missing numbers missing.

Label unknown categories, keep missing numbers missing · VisualLabel unknown categories, keep missing numbers missing · SQLLabel unknown categories, keep missing numbers missing · Python

Next: Label unknown categories, keep missing numbers missing