Glossary

Train/test split

Keeping evaluation data separate so nothing learned from it influences the model or the cleaning.

Statistics used for cleaning, such as a median for filling gaps, must be learned from the training rows only and then applied to test rows. Learning them from all rows leaks test information and makes results look better than they are.

Lesson: Fit on training data only
Practise: Fill gaps with a median learned from training data only

Related: Data leakage, Imputation