Reference · Python (pandas)
Fill missing values in Python (pandas)
Replace missing values with a stated value: a visible label for a category, or a fitted value with a flag for a measurement.
When to use it
Reporting labels such as "unknown"; imputation only when an analysis requires it and the fill is fitted on training data.
How
Series.fillna(value).
df["region"] = df["region"].fillna("unknown")- Fit statistics on the training frame and reuse the number for test data.
Pitfalls
- Filling measurements with 0.
- Fitting the fill value on test data.
- Not flagging imputed values.
Lesson: Fill labels, not measurements
Practise it: Label unknown categories, keep missing numbers missing