If you’ve ever had a prompt rejected for exceeding a context window, or watched an API bill come in higher than expected, the root cause is almost always the same: you were thinking in characters or words, and the model was thinking in tokens.
What a token actually is
A token is the smallest chunk of text a language model reads or generates — not a word, not a character, but a piece somewhere in between. Modern LLMs use subword tokenizers (Byte Pair Encoding or similar) that break text into frequently-occurring fragments. "tokenization" might become ["token", "ization"]. Common English words are usually a single token; rare words, code identifiers, and non-English text often split into several.
A few rules of thumb that hold reasonably well for English prose:
- ~4 characters per token
- ~0.75 words per token (so 100 words ≈ 133 tokens)
Those ratios fall apart fast for code, JSON, non-Latin scripts, or heavily-punctuated text — which is exactly the content developers paste into prompts most often.
Why the tokenizer matters more than the model
Every model family ships its own tokenizer, trained on its own corpus, and they don’t agree with each other:
- OpenAI’s
cl100k_base/o200k_base(GPT-4, GPT-4o family) tokenize English efficiently but split unfamiliar identifiers and Unicode aggressively. - Anthropic’s Claude tokenizer handles long-context documents with different segmentation, especially around whitespace and code.
- Open-source models (Llama, Mistral, etc.) each define their own vocabulary size and merge rules via SentencePiece or similar.
The same input string can produce meaningfully different token counts across these tokenizers — sometimes a 15-20% swing. If you’re estimating cost or context usage with a single “tokens ≈ words / 0.75” heuristic across model families, you’re carrying that error into every estimate.
The two places token counts bite you
1. Cost. Every major LLM API bills per input and output token, with input and output priced differently (output is usually 2-5x input). A prompt that “looks short” but contains a large system prompt, few-shot examples, or a pasted JSON schema can be 10x your assumption once tokenized.
2. Context window. Models cap total tokens per request — input plus output combined. Long conversation histories, RAG-retrieved chunks, and tool definitions all eat into that budget silently. Hit the ceiling and you get a hard error mid-request, often after you’ve already spent tokens getting there.
Measuring instead of guessing
The fix is simple: count tokens for the actual model you’re targeting before you ship the prompt, not after the invoice arrives. Our AI Token Counter runs real tokenizers for GPT, Claude, and other model families client-side, so you can paste a prompt and see the exact count per model side by side — useful when you’re choosing between providers or estimating cost for a feature before you build it.
If you want to see why a string counts the way it does — which characters or fragments become which tokens — the Tokenizer Visualizer renders the actual token boundaries as a heatmap. It’s the fastest way to understand why your carefully-minified JSON payload is costing more than the equivalent prose description.
For teams building RAG pipelines or long-context applications, two follow-up tools are worth bookmarking:
- Context Window Visualizer — see how your system prompt, chat history, and retrieved context stack up against a model’s context limit before you hit it in production.
- Embedding Cost Calculator — estimate the token and dollar cost of embedding a document corpus before you run the job.
The practical takeaway
Token count isn’t a detail to check after something breaks — it’s an input to your design decisions: which model to use, how much conversation history to keep, whether to compress a prompt or retrieve less context. Measuring it up front, per model, turns a guess into a number you can actually plan around.