· 7 min read
Two strings look identical on screen, and every comparison says they differ. A search finds nothing though the word is visibly there. A code block will not run and the error points at a character you cannot see. All of these have the same cause.
| Character | Where it comes from | What it breaks |
|---|---|---|
| Non-breaking space | Word processors, web pages, PDFs | Splitting on spaces, trimming, comparisons |
| Zero-width space | Copying from web pages and chat apps | Search, comparison, character counts |
| Byte order mark | Files saved as UTF-8 with BOM | The first line of a config or CSV |
| Smart quotes | Word processors and editors that autocorrect | Code, CSV parsing, exact matching |
| Soft hyphen | Justified text, PDFs | Search, word matching |
| Non-standard whitespace | Copied tables and formatted output | Splitting and trimming |
It looks exactly like a space, it is wide like a space, and it is not a space. Code that splits on a space will not split there. A trim that removes whitespace may or may not remove it depending on the language and the function. A comparison against a normal space fails, and nothing about the screen explains why.
It arrives whenever text passes through something that cares about typography: a word processor keeping a number next to its unit, a web page preventing an awkward line break, a PDF preserving layout.
A BOM is three bytes at the very start of a file. Many editors hide it. The symptom is that the first column header of a CSV does not match, or the first key of a config is not found, while every other line works. If a problem is confined to the first line of a file, suspect a BOM before anything else.
A word processor turns straight quotes into curly ones. In prose that is correct typography. In a code snippet it produces a syntax error, and in a CSV it breaks quoting rules, because a curly quote is an ordinary character rather than a delimiter.
If a snippet copied from a document, a slide or a chat message will not run, check the quotes before anything else. It is the most common cause and the least visible.
For data that will be compared, parsed, stored as a key or used in a URL, normalise: convert every kind of space to a plain space, straighten quotes, remove zero-width characters and the BOM. For prose you are going to publish, be more careful. Non-breaking spaces are often deliberate, and removing them changes where lines break.
The cleaner here runs in the browser, which matters because the text being cleaned is often a pasted email or document rather than something public.
Related: How to clean AI-generated text before you publish it
Almost always an invisible character: a non-breaking space, a zero-width space or a curly quote in one of them. Compare the character counts first; a difference confirms it.
It is three bytes at the very start of a UTF-8 file that most editors hide. It attaches to the first field or key, so everything after the first line parses correctly.
In data, yes. In prose intended for publication, no. They are often deliberate typography, and removing them changes where lines break.