LLM Token Counter

Every model bill is a token count times a price, and the token count is the only half you can change before sending. The counter above uses the same byte-pair vocabularies the OpenAI API uses, so its numbers are the billed numbers for GPT-5, GPT-4.1, GPT-4o, GPT-4 and GPT-3.5, not a characters-divided-by-four guess. For Claude it says "estimate", because nothing else would be honest.

What a token is, and why "4 characters" fails

A token is an entry in the model's vocabulary, and the vocabulary was built by byte-pair encoding. Start with the 256 byte values, scan a large corpus, merge the most frequent adjacent pair into a new entry, repeat until the table has the target size. o200k_base stopped at about 200,000 entries, cl100k_base at about 100,000, p50k_base at about 50,000. Frequent English words and word pieces got their own entries early (" the", " support", "ization"), rare strings never did and fall back to shorter pieces, in the worst case single bytes.

Encoding runs the merges in reverse. A regular expression first cuts the text into pre-tokens (a word with its leading space, a run of digits up to three long, a run of punctuation, a run of whitespace), then each pre-token is merged greedily from the rank table. That is why " tokens" with its leading space is one token, why "hello world" is two tokens and "HELLO WORLD" three, why a number splits every three digits, and why the same word can cost differently depending on what stands before it.

$ node -e "const {countTokens,encode}=require('gpt-tokenizer/encoding/o200k_base');for(const s of ['Count tokens before you pay for them.','{\"ok\": true}','日本語のテキスト','😀'])console.log(countTokens(s),JSON.stringify(encode(s)),JSON.stringify(s))"
8 [3417,20290,2254,481,2777,395,1373,13] "Count tokens before you pay for them."
5 [10848,525,1243,1343,92] "{\"ok\": true}"
6 [9048,40909,3385,16056,18368,38236] "日本語のテキスト"
1 [84083] "😀"

The "one token is about four characters" rule is correct for exactly one kind of text. Our cron parser article, 10,979 characters of English prose, encodes to 2,748 tokens, 4.00 characters per token. Everything else misses. The German sample runs 5.1 characters per token but 1.45 tokens per word instead of 1.36, TypeScript source 3.8 characters per token and 2 tokens per word, pretty-printed JSON 2.6 characters per token, the Japanese sample 1.6, a random UUID about 1.6 as well with a worst case near 1. A budget built on the four-character rule is 36 percent short on the JSON and 61 percent short on the Japanese text.

Which tokenizer counts for which model

Three OpenAI vocabularies cover every model people still call, and the right one is a function of the model, not of the text.

The counter above loads the first three on demand, which is the honest way to do it. o200k_base is 2.4 MB of merge ranks, cl100k_base 1.2 MB, p50k_base 0.6 MB, and downloading all of them for a visitor who wants one number would be rude. Each one is fetched once when you pick it and stays cached in the tab.

Everyone else has their own vocabulary. Anthropic published a tokenizer for Claude 2 and has not for Claude 3 onward, and Opus 4.7 switched to a new one that Anthropic's docs put at about 30 percent more tokens for the same text. The only exact count is the count_tokens endpoint, which is free, takes the same messages, system prompt and tool definitions as the real call, and never leaves the server. Gemini answers through countTokens, Google's Python SDK can also count locally, and Google's own guidance is 100 tokens for 60 to 80 English words. Llama 3 went the other way and adopted a tiktoken-style 128k vocabulary, Llama 2 still has a 32k SentencePiece one. Same text, five different numbers, and the differences are not noise. Vocabulary size is the main reason. A 200k vocabulary holds a merge for " Datenschutz" that a 50k vocabulary has to assemble from four pieces, which is why the German sample costs 77 tokens in o200k_base and 152 in p50k_base while English barely moves (1,358 against 1,325 per 1,000 words). For the Claude rows in the cost table the counter scales the o200k_base count by 1.3 for Opus 5 and Sonnet 5, the ratio of Anthropic's own words-per-token figure (about 555,000 words per million tokens) to o200k_base's (about 735,000), and leaves Haiku 4.5 at the plain count. For code, JSON and non-English text the gap goes either way, so treat those rows as a budget, not a bill, and confirm with the endpoint before committing to a number.

Cutting tokens, measured

Advice about token reduction is usually unmeasured. These numbers are from the counter above, o200k_base, on texts you can paste yourself.

Two things fall out of that table. Indent width is irrelevant, because a run of spaces is one token however long it is, so reformatting from four spaces to two saves nothing and the person who suggested it was guessing. And minifying JSON is the single cheapest cut there is, a third of the payload for zero change in meaning and one JSON.stringify without the indent argument, or a pass through the JSON minifier. The findings panel measures this for whatever JSON it finds inside your prompt, because "would save about a third" is an estimate and "would save 56 tokens" is a number you can act on.

The rest of the list, in order of how often it pays off. Replace identifiers the model only has to pass through. A random UUID is 20 to 26 tokens, id1 is one, and a lookup table on your side costs nothing. The SQL anonymizer and the code anonymizers next to it do exactly that for a whole query or file, table_1 and col_2 in place of every name of yours, with the lookup table kept as a key in your browser. Send files as files, because a Base64 image in the prompt costs about two thirds of a token per character and the model cannot decode it anyway, every provider has an image or file input for this. Strip boilerplate that arrives with copied text, including the invisible characters this page counts separately, because a zero-width space is a full token and a PDF paste can carry hundreds. Shorten the system prompt once and save on every call for the rest of the product's life. Summarise the history instead of resending it whole, which is the next section.

What counts toward the bill

Everything in the request, every time. The API is stateless, so the system prompt, the tool definitions and the full conversation history are sent, tokenized and billed on every turn, not once. A 1,200-token system prompt in a 20-turn conversation is 24,000 billed input tokens before the user has said anything new, and a chat with a growing history is quadratic, since turn 20 resends turns 1 through 19.

On top of the text, the chat format adds framing. OpenAI's cookbook formula is 3 tokens per message plus 3 for the reply, small per message and not small across a history of 40. Tool definitions are JSON schemas serialised into the prompt, and a verbose schema with long descriptions is the silent cost centre in most agent setups. Images are tokenized by tile size rather than bytes, so a screenshot is hundreds of tokens regardless of what is on it.

Output is priced separately and higher, four to eight times per token in the table above. The prompt above is the cheap half of a call with a long answer, and the single most effective cost control is a tighter max_tokens plus an instruction on length. Prompt caching cuts the other half. OpenAI bills cached input at 10 to 50 percent of the list price depending on the model (a tenth on GPT-5), Anthropic bills cache reads at a tenth and cache writes at 1.25 times, which turns the 24,000 repeated tokens from the first paragraph into the price of 2,400 on GPT-5. The cache only hits if the prefix is byte-identical, so put the stable parts first and the changing parts last.

Context windows, and why fits is not works

The context fit panel says whether the prompt goes through the door. 8k, 32k, 128k, 200k and 1M are the sizes that models are still sold under, and going over any of them is a hard error rather than a truncation. The panel measures the prompt alone. Real requests add history, tools and the answer, and the answer needs room of its own, so a prompt at 90 percent of the window is already a problem.

Filling the window is a separate question from it working. Long-context evaluations keep finding the same shape. A model that retrieves a fact perfectly from a short context starts missing it in a long one, and is worst when the fact sits in the middle (the "lost in the middle" result from 2023, reproduced since on newer models and in the RULER benchmark, which found the usable context of many models well below the advertised size). A 1M window is a capacity, not a promise. For anything that has to be reliable, shorter prompts with the relevant material near the start or the end beat a full window every time, and they are cheaper by exactly the tokens you left out.

Counting tokens in code

For OpenAI models the reference is tiktoken, the same library the API uses. tiktoken.encoding_for_model("gpt-4o") resolves the vocabulary from the model name, tiktoken.get_encoding("o200k_base") picks it directly, and len(enc.encode(text)) is the count. In JavaScript, js-tiktoken and gpt-tokenizer (the one bundled here) ship the same ranks, so their numbers match to the token, and the repro block above is the three-line version. Note that encode in both raises on special tokens like <|endoftext|> unless told otherwise, which is why pasted transcripts sometimes crash naive counters. The counter on this page encodes them as plain text, the way the API would receive them.

For Claude, client.messages.count_tokens(model=..., system=..., messages=..., tools=...) in the Python and TypeScript SDKs, or the raw POST /v1/messages/count_tokens. It is free, rate-limited separately from messages, and counts the request exactly as it will be billed. For Gemini, client.models.count_tokens(model=..., contents=...) in the google-genai SDK. In all three cases the usage field of a real response is the authoritative number after the fact, and logging it per call is how you find out which feature is eating the budget.

One parameter name causes more confusion than the rest together. max_tokens (OpenAI now calls it max_completion_tokens) limits the answer, not the request. The request limit is the context window, and input plus output together must fit in it. So a 120,000-token prompt to a 128k model with max_tokens: 16000 fails, a 120,000-token prompt with max_tokens: 4000 goes through, and a short prompt with max_tokens: 50 comes back cut off mid-sentence with finish_reason: "length" rather than with an error.

The verdict, and when it is wrong

Use o200k_base for every current OpenAI model and for any estimate where the exact tokenizer is unavailable. cl100k_base only for GPT-4, GPT-3.5 Turbo and the embedding models, p50k_base for nothing new.

The estimate is weakest for Claude on code-heavy or non-English prompts, where the vocabularies diverge by far more than the 1.3 factor covers. Call count_tokens before you budget there.

Tokens, counted and paid for