Before this: Map categories with an approved dictionary
The idea
After normalizing and mapping, some values remain: typos, new categories, junk. These are unmapped.
Send them to the exception report with reason unmapped and their raw value. A reviewer can add a variant to the dictionary or fix the source.
Fuzzy matching can propose candidates for the reviewer, but shouldn't decide by itself.
Worked example
Input: channel: email, e-mail, emial, fax
- Visual
Add step → Map values (approved dictionary) → unmapped: quarantine unmapped_channel
- SQL (DuckDB)
CASE WHEN m.approved IS NULL THEN 'unmapped_channel' END
- pandas
mapped = df["channel"].map(DICT); bad = mapped.isna() & df["channel"].notna()
Result: emial and fax → exceptions; email and e-mail → email
Common mistakes
- Nearest-match guessing.
- Dropping unmapped rows.
- Dictionaries keyed on un-normalized text.
Check yourself
Who decides whether 'emial' becomes email?
A reviewer, by adding it to the approved dictionary.
Practice it
Map channels with an approved dictionary.