Before this: Missing is not empty (and neither is zero)
The idea
Some cleaning steps learn from the data: a median used to fill gaps, a scaling factor, a list of frequent categories. If they learn from the test set, information about the evaluation leaks into the inputs and results look better than they will in production.
Fit on training data, then apply the fitted value unchanged to test (and later to new batches). The test data must not change the fitted value, so a test row with an extreme value can't move the median.
Mark imputed values with a flag column. Analysts can then check whether imputation drives a result, and nobody mistakes a filled value for an observation.
Worked example
Input: train.x: 4, 7, 10, missing / test.x: missing, 100
- Visual
Convert x to int64 → Fill with median: column x, learn from table train, flag column imputed
- SQL (DuckDB)
WITH m AS (SELECT median(CAST(x AS BIGINT)) AS med FROM train WHERE x IS NOT NULL) SELECT id, COALESCE(CAST(x AS DECIMAL(12,2)), ROUND(m.med, 2)) ...
- pandas
med = train["x"].dropna().astype("int64").median()
Result: The median is 7 from train only; the test row with 100 does not change it. The missing test value becomes 7.00 with imputed = true.
Common mistakes
- Fitting on test or on all data.
- Re-fitting on each new batch without saying so.
- Not recording which values were imputed.
Check yourself
Why not compute the median over train and test together?
Test values would influence the fill, leaking evaluation data into the inputs.
What is the imputed flag for?
It keeps filled values distinguishable from observed ones in later analysis.
Practice it
Fill missing test values with a median learned from train only, and flag them.
Next: Fill gaps with a median learned from training data only