Data Cleaning Exercises

Module 19 · Analysis readiness · 6 min

Fit, then apply

You'll be able to: Learn cleaning statistics on training rows and apply the stored values elsewhere.

Before this: Fit on training data only

The idea

A fill value, a category list or a scaling factor is a fitted parameter. Fit it on training data, store it, and apply it to test and new data.

Refit only when you deliberately retrain, never on the data you evaluate.

Worked example

Input: train x: 2, 4, 6, missing; test x: 100, missing

Visual
Fill with median → from train
SQL (DuckDB)
SELECT median(x) FROM train  -- then COALESCE(test.x, 4)
pandas
m = train["x"].median(); test["x"] = test["x"].fillna(m)

Result: Test gap filled with 4, not 100

Common mistakes

  • Fitting on all rows.
  • Refitting on test data.

Check yourself

Where does the fill value come from?

Training rows only.

Practice it

Fill gaps with a median learned from training data only.

Fill gaps with a median learned from training data only · VisualFill gaps with a median learned from training data only · SQLFill gaps with a median learned from training data only · Python

Next: Fill gaps with a median learned from training data only