Guide · updated 2026-11-04
Whitespace and invisible characters
Why keys that look identical don't match, and how to trim exactly what the rules say.
Not all spaces are equal
Besides the space and tab, data contains no-break spaces (U+00A0), ideographic spaces and zero-width characters. DuckDB's trim() removes spaces only; pandas .str.strip() removes all Unicode whitespace. Name the characters you remove.
Diagnose before fixing
When two values look equal but don't compare equal, check their lengths and code points. That shows invisible characters, decomposed accents or different dashes.
Trim only the ends
Trimming means removing characters from both ends. Inner spaces are part of the value unless a rule says otherwise. Changing them silently can merge different names or codes.
Practise it
- Lesson: Trim the right characters, and only those
- Lesson: Normalize Unicode, keep the accents
- Exercise: Normalize names without changing letters
- Exercise: Clean an order-line export