Before this: Trim the right characters, and only those
The idea
Values copied from web pages and PDFs carry characters you can't see: no-break spaces, zero-width spaces, soft hyphens.
Compare lengths: if 'A100' has length 5, something invisible is there. List code points to see exactly which character.
Then decide by rule: trim declared characters at the ends, and report anything inside the value rather than deleting it silently.
Worked example
Input: code: 'A100\u200b'
- Visual
Inspector → code → length
- SQL (DuckDB)
length(code), unicode(right(code, 1))
- pandas
df["code"].str.len(); [hex(ord(c)) for c in df.loc[0, "code"]]
Result: length 5; last code point 0x200b (zero-width space)
Common mistakes
- Assuming all invisible characters are whitespace.
- Deleting characters inside values without a rule.
- Debugging by eye.
Check yourself
Is a zero-width space removed by trimming Unicode whitespace?
Not necessarily: it is not classified as whitespace by many tools, so name it explicitly if the rules allow removing it.
Practice it
Trim Unicode whitespace and normalize names to NFC.