One sentence, five languages, two vocabularies
The sentence is Article 1 of the Universal Declaration of Human Rights, a standard test text because official translations exist for practically every language. We counted it with the two OpenAI vocabularies that matter in practice: cl100k_base (GPT-4, GPT-3.5) and o200k_base (GPT-4o and later), using the same gpt-tokenizer library our token counter runs on.
$ node count-tokens.mjs english o200k: 13 cl100k: 13 german o200k: 14 cl100k: 16 hindi o200k: 25 cl100k: 84 thai o200k: 31 cl100k: 62 burmese o200k: 68 cl100k: 246
Same meaning, same information, and under cl100k_base the Burmese version costs 19 times the English one. You can reproduce this with your own text in the LLM token counter, which runs the real tiktoken vocabularies in your browser and draws every token as a chip, so you can see exactly where a language falls apart into pieces.
Why tokenizers favor English
Byte pair encoding starts from raw bytes and repeatedly merges the most frequent pairs in its training corpus until the vocabulary is full. Frequency decides everything. A string that appears constantly, like the or ing, ends up as a single token. A string the corpus rarely saw stays in small fragments, sometimes single bytes.
The training corpora behind these vocabularies are dominated by English web text. So English words get whole tokens, German gets syllable-sized pieces, and scripts like Myanmar, which the corpus barely contains, get chopped near the byte level. The tokenizer is not doing anything wrong. It is faithfully reflecting what it was fed.
There is a second, sneakier effect: a token is also the unit the model thinks in. Languages that fragment into byte crumbs give the model worse building blocks, which is part of why model quality and token efficiency tend to degrade together on low-resource languages.
The 15x premium has a paper
This is not a quirk we found in one sentence. Petrov, La Malfa, Torr and Bibi measured it across 350 languages in “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023) and found the same text costing up to 15 times more tokens than English, with Shan, Burmese and Dzongkha at the top of the scale. The paper’s framing stuck with us: a speaker of one of these languages pays more money for less context, waits longer per answer, and gets a worse model doing it.
Armenian or Cyrillic-script languages sit in the middle of that range, and Western European languages get off lightly. The premium is not exotic edge-case trivia. Hindi alone has over 600 million speakers, and it paid 6.5x under the vocabulary GPT-4 shipped with.
What that does to a real API bill
Token pricing multiplies through everything:
- Cost. A support bot handling Thai tickets pays for the Thai in every user message and every model reply. At 3x tokens, the same conversation volume is three times the spend.
- Latency. Output streams token by token. A reply that needs three times the tokens takes roughly three times as long to finish, and users notice streaming speed more than they notice anything else.
- Context. Retrieval chunks, few-shot examples and history budgets sized on English content silently hold far less when the content switches script. A “last 20 messages” window can become a “last 7 messages” window.
- Limits. Rate limits counted in tokens per minute are consumed faster for the same traffic.
If your product is multilingual, split your usage dashboards by language once. The distribution is usually a surprise.
What o200k_base fixed, and what it didn’t
OpenAI’s o200k_base vocabulary, introduced with GPT-4o, doubled the vocabulary size to around 200k entries and trained the merges on a more multilingual corpus. The measurement above shows what that bought: Hindi fell from 84 tokens to 25, Burmese from 246 to 68. Real progress, between 2x and 3.5x cheaper for those languages.
It did not buy equality. Burmese still costs 5x English for the same sentence, Hindi still 2x. A bigger vocabulary shifts the premium, it does not repeal the logic that produces it, because English is still the plurality of the training text.
What you can actually do about it
You cannot change the tokenizer, but you control what goes through it.
- Keep system prompts and instructions in English. The model follows English instructions fine while conversing in another language, and instructions are the part you send on every single request. This is usually the biggest single saving.
- Keep structural text ASCII. JSON keys, tool names and enum values in English cost little and never reach the user. Translated field names pay the premium for no one’s benefit.
- Trim payloads before prompting. Whitespace-heavy JSON pays the token premium too, so minifying the JSON you embed in prompts saves a measurable slice, and the counter shows that saving directly.
- Budget context per language. If you cap history or retrieval by tokens rather than by items, multilingual behavior stays consistent by construction.
- Measure with the real vocabulary before and after any of this, not with a characters-divided-by-four estimate, which only ever held for English.
When one character is three tokens
The floor under all of this is UTF-8. BPE tokenizers operate on bytes, and a character outside ASCII is two to four bytes before the tokenizer even starts merging, the same byte math that produces mojibake when a decoder guesses wrong. A Myanmar-script character is three bytes, so a rare word the vocabulary has no merges for bottoms out at three tokens per character. That is the mechanical reason the worst cases are all non-Latin scripts.
Emoji make the floor visible in one glyph: a single flag emoji is eight bytes, and a family emoji with skin tones can be over twenty, which the tokenizer then covers in several tokens.
Tokens across languages
Why does my prompt use more tokens in German than in English?
Because the tokenizer’s vocabulary was learned mostly from English text. Frequent English words get their own token, while German compounds and inflections are split into several pieces. In our measurement the effect on German is mild, around 10 to 25 percent more tokens for the same sentence. For languages in non-Latin scripts it gets dramatic.
Which languages use the most tokens?
Those written in scripts rare in the training data. Petrov’s NeurIPS 2023 study measured Shan, Burmese and Dzongkha at 15 times the English count.
Does translating prompts to English reduce API costs?
Yes, often substantially, and for a Burmese or Thai workload the input side can shrink several-fold. Whether it is worth it depends on the task: instructions and system prompts translate well, while user content usually has to stay in its language, and a translation step adds cost, latency and its own errors. Measure the split before rebuilding a pipeline around it.
What is tokenizer fertility?
The average number of tokens produced per word or per character in a language. A fertility of 2 means each word becomes two tokens on average. It is the standard metric for comparing how efficiently a tokenizer handles different languages.
Are responses slower in high-token languages?
Yes. Output is generated token by token, so an answer needing three times the tokens streams roughly three times as long.
Does Claude tokenize other languages the same way as GPT models?
Anthropic does not publish the Claude tokenizer, so exact counts are not reproducible offline. The pattern is the same in direction, English cheapest and rare scripts costliest, but the ratios differ per vocabulary. Anthropic’s API returns the real usage counts per request, which is the reliable source for Claude budgeting.
Why does the context window fill up faster in some languages?
The window is measured in tokens, not characters or words. A 128k-token window holds around 96,000 English words, but the same window holds far less Burmese or Hindi text, because each word costs more tokens there.
Do accents and umlauts increase token counts?
Slightly, in unlucky cases. Characters outside ASCII are more likely to fall onto rarer tokens or byte fallbacks, so café can cost one token more than cafe. It is measurable, not a savings strategy: stripping accents changes the words themselves.