Invisible Unicode characters

The cast

These are not exotic. Every one of them arrives in normal work, usually pasted in from somewhere else.

The zero width joiner is worth a second look because it is load-bearing rather than junk. The family emoji 👨‍👩‍👧‍👦 is not one character, it is four emoji with U+200D between them: seven code points, 25 bytes in UTF-8, and a .length of 11 in JavaScript. Strip the joiners and your family falls apart into four people standing next to each other. Same story with U+200C in Persian and Hindi, where the non-joiner changes which word you wrote. Blanket-deleting invisible characters is not a safe cleanup.

U+FEFF has a split identity that explains a lot of confusion. It was originally ZERO WIDTH NO-BREAK SPACE, an actual formatting character, and got reused as the byte order mark. Unicode 3.2 (2002) resolved the conflict by introducing U+2060 WORD JOINER for the formatting job and reserving U+FEFF for the BOM role. Old text still contains it in the middle of lines, where nothing expects it.

Where they come from

Almost never from malice. The boring sources, in roughly the order we run into them:

• Copy-paste out of a PDF. PDFs carry soft hyphens from justified typesetting and no-break spaces around units and figure numbers. Search your codebase for a term you pasted from a spec and watch it not match.
• Word and Google Docs. Autocorrect inserts no-break spaces before punctuation in French locales, after abbreviations elsewhere, and both editors are fond of soft hyphens.
• macOS keyboard. Option+Space types U+00A0, and it is exactly one thumb slip away from a normal space. This one produces more broken YAML than any other source we know.
• Chat apps and issue trackers. Slack, Teams and various web editors insert zero-width characters to keep formatting or emoji intact, then hand them to you when you copy a snippet out of a message.
• Windows editors and Excel. Anything that writes a UTF-8 BOM by default, which is a large club. Excel in particular needs one to open a UTF-8 CSV correctly, a trade-off we go through in why Excel ruins your CSV files.

What they actually break

The reason these cost so much time is not that they are rare, it is that every layer of tooling handles them slightly differently.

String comparison. A trailing U+200B makes "foo" === "foo​" false (there is a real zero-width space at the end of the second literal, invisible in your browser too), and no amount of squinting at the diff will show you why. Trimming does not help either: in JavaScript, " ".trim() and "".trim() both return an empty string because the language counts no-break space and the BOM as whitespace, while "​".trim() returns the zero-width space unchanged. It is not in Unicode’s whitespace category, so \s in a regex misses it too. Python behaves the same way: "\xa0".isspace() is true, "​".isspace() is false.

YAML. Indentation must be plain spaces. One no-break space in the leading whitespace and the parser reports a character that cannot start any token, pointing at a line that looks perfect. YAML is generous with ways to be surprised, and we collected the other big one in the Norway problem.

JSON. A BOM in front of the opening { is not valid JSON. JSON.parse throws a syntax error at position 0, Go’s encoding/json refuses it, and the error message never says "BOM". Any strict parser reacts the same way, including the one behind our JSON to YAML converter, so if a file fails there and looks fine in your editor, check the first three bytes.

Everything else. A BOM before a shebang gives you "bad interpreter: no such file or directory". A no-break space in an HTML class attribute means the selector never matches. A zero-width space in an email address passes a naive regex and bounces at the SMTP server. A soft hyphen inside a slug produces a URL that works when clicked and 404s when typed.

Trojan Source: when invisible becomes a vulnerability

On 1 November 2021, Nicholas Boucher and Ross Anderson of the University of Cambridge published Trojan Source, tracked as CVE-2021-42574 for the bidirectional variant and CVE-2021-42694 for the homoglyph variant. The idea is simple enough to be uncomfortable.

Unicode has a bidirectional algorithm so that Hebrew and Arabic can be mixed with Latin text, and it has override characters (U+202A to U+202E, plus the isolates U+2066 to U+2069) that force a rendering direction. Those characters are legal inside comments and string literals. A compiler reads the token stream in logical order. Your editor, your terminal, your code review UI and your browser render it in display order. Put an override in the right place and the two orders disagree: a reviewer sees a return inside a comment or a condition that reads as harmless, and the compiler builds something else entirely.

The paper demonstrated it against C, C++, C#, Java, JavaScript, Python, Go and Rust. Not a bug in any one of them, a property of the encoding they all accept, which is why the disclosure went out to dozens of vendors at once. The responses arrived fast:

• GitHub started warning about bidirectional Unicode text in file views on 31 October 2021, the day before publication.
• Rust shipped 1.56.1 on 1 November 2021 with a deny-by-default lint for bidi control characters in source files.
• VS Code 1.63, the November 2021 release, added editor.unicodeHighlight.invisibleCharacters and editor.unicodeHighlight.ambiguousCharacters, both on by default. That is why your editor started boxing odd characters in yellow around then.

The homoglyph half (CVE-2021-42694) needs no invisible characters at all: define a function named with a Cyrillic е instead of a Latin e and you get two functions that look identical in review. If you review code for a living, the practical takeaway is that "I read the diff" is a weaker statement than it used to be, and a CI check for bidi and confusable characters is twenty lines.

The right-to-left filename trick

U+202E has a second career in malware, and it predates Trojan Source by years. Name a file Invoice + U+202E + fdp.exe and the override flips everything after it, so the file manager displays Invoiceexe.pdf. The extension is still .exe, the icon is whatever the binary carries, and the user sees a PDF. Windows Explorer renders the character faithfully because it is a legitimate part of the Unicode text model, which makes this a display convention being used exactly as designed.

The same shape of trick works in anything that shows a filename: mail clients, chat attachments, download lists. Mitigations mostly live in the client, either by refusing to render bidi controls in filenames or by showing the extension separately. If you build anything that displays user-supplied names, stripping U+202A to U+202E from that string before it hits the DOM is a cheap and complete fix for the display side.

Watermarks, leak tracing and AI text

Because they survive copy-paste and are invisible on screen, zero-width characters make a convenient covert channel. The technique is straightforward: encode a recipient ID in binary using two zero-width characters as 0 and 1, sprinkle them between words, and every copy of a document carries a unique fingerprint. If a leaked screenshot is retyped it is gone, but a leaked copy-paste keeps it. Several public write-ups and toy implementations of this exist; whether any given company does it to their internal documents is not something you can tell from the outside.

A more recent variant matters for anyone processing untrusted text with a language model. Researchers have shown that invisible code points can smuggle instructions past a human reviewer while still being tokenised by the model, which turns "paste this document into the assistant" into a prompt injection vector. Sanitising invisible characters before text reaches a model is now a reasonable default rather than paranoia.

Then there is the claim you have probably seen: that chatbot output is watermarked with zero-width characters. Be careful with that one. Through 2024 and 2025 people did find unusual typography in ChatGPT output, most notably the narrow no-break space U+202F where a normal space belonged, and read it as a hidden marker. There is no confirmation of such a watermark from the vendors, and the mundane explanation fits better: models trained on professionally typeset text learn professionally typeset spacing. Our reading of the evidence is that an invisible character tells you text went through some tool, and nothing reliable about which one.

How to hunt them down

Make the editor do it first. VS Code has flagged invisible and confusable characters by default since 1.63, JetBrains IDEs have the same inspection, and turning on render-whitespace catches the visible-width offenders like tabs and no-break spaces. Most of the time you never need the command line.

When you do, three tools cover it:

• grep -Pn '[\x{00a0}\x{00ad}\x{200b}-\x{200f}\x{202a}-\x{202e}\x{2060}\x{feff}]' -r . with GNU grep, or the same pattern with ripgrep, which supports the escapes natively. Adjust the ranges to the ones you actually care about.
• hexdump -C file | head for the first bytes of a file, which answers the BOM question in one second: ef bb bf at offset 0.
• cat -A file on Linux, where a no-break space shows up as M-BM- followed by a space and the soft hyphen as M-B M-­. Ugly, unambiguous.

For cleaning, target specific code points. In Python, unicodedata.category(c) returns Cf for format characters, and it is tempting to filter the whole category in one line. Don’t: that kills the emoji joiners and the Persian non-joiner along with the junk. Normalising to NFKC is the gentler move for prose, since it maps U+00A0 to a regular space and folds a pile of compatibility characters, while leaving the joiners alone.

No tool has saved us as much time as these two reflexes. First, when a comparison fails and the values look equal, print the lengths, then the code points; the answer arrives immediately instead of after twenty minutes of staring. Second, put a grep for bidi and zero-width characters in CI for source files, because the cost is one job step and the failure mode it catches is one you cannot see in review. And if the text is not invisible but garbled, that is a different problem with a different cause, which we untangled in mojibake explained.

Finding and removing invisible characters