What Is a Token?
A plain-language explainer covering what tokens are, how a model turns your text into them, and why tokens decide cost, limits, and some of the strange things models do.
Who this is for: anyone using AI tools who wants to understand what the model actually reads. No technical background needed. How it's organized: definition, then how tokenization works, then the numbers, then what tokens explain in everyday use.
The one-sentence version: a token is the unit of text an AI model actually reads and writes — the model never sees your words, only tokens.
Part 1: The Basics
What a token is
A token is a piece of text that can be a character, a part of a word, a whole word, or a punctuation mark [1]. Spaces count too: the same word with and without a leading space can end up as different tokens [1].
Every language model works in tokens, not in words. If a model's context window is 200,000 tokens, that is not 200,000 words — at roughly three-quarters of a word per token, it is closer to 150,000 [1]. Tokens are the currency the whole system runs on: usage is measured in them, prices are quoted per million of them, and every limit the model has is expressed in them.
What actually happens when you press send
Four mechanics explain most of what tokens do and why they matter.
1. Your text is cut into tokens before the model sees it. A component called the tokenizer splits the input into tokens and replaces each one with a number — a token ID [1][4]. The model processes the IDs, and when it answers, it generates output token by token [1]. The tokenizer runs first, last, and always.
2. The model never reads your words. This follows directly from mechanic 1, and it is worth stating twice. A model sees the tokens its tokenizer chose, not the letters a human sees [1] — so tasks that depend on individual characters, like counting the letters in a word, can be harder than they sound.
3. The tokenizer is trained, not programmed. Most modern models use byte-pair encoding (BPE). Training starts from every possible byte as the base vocabulary, then repeatedly finds the most frequent pair of neighboring tokens in the training data and merges them into one new token [4]. Common words earn their own single token; rare words get split into several pieces. There is no dictionary of words inside — just a table of merges learned from data.
4. Every model has its own encoding. The same text can produce different token counts on different models, because each model ships with its own tokenizer and encoding [1]. A token ID from one model means nothing to another. The effect reaches inside a single product family: Anthropic notes that Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same input text than earlier models [3]. A count measured against one model does not transfer to another.
One thing that follows from this. When a provider ships a new tokenizer, every cached estimate, prompt budget, and cost calculation quietly goes stale. Recount against the model you plan to use [3].
Part 2: The Numbers That Matter
For English text, a few approximations go a long way [1]:
| Measure | Approximation |
|---|---|
| 1 token | About 4 characters |
| 1 token | About three-quarters of a word |
| 100 tokens | About 75 words |
These are estimates, not counts. Sentence structure, capitalization, and language all shift the ratio [1].
Not all tokens cost the same, and not all tokens are visible. API usage breaks into four categories [1]:
| Category | What it counts |
|---|---|
| Input tokens | What you send — the prompt, documents, tool definitions. Also called prompt tokens |
| Output tokens | What the model generates. Also called completion tokens |
| Cached input tokens | Input reused through prompt caching; often billed at a lower rate |
| Reasoning tokens | Internal thinking a reasoning model does before its visible answer |
Reasoning tokens deserve a second look. They never appear in the answer text, but they are billed as output tokens [1] — so a short visible answer can use more tokens than its displayed text suggests [1]. If a response seems slow or expensive for its length, this is often why.
Counting tokens before you send
You do not have to guess. Most providers expose the tokenizer directly:
- Interactive: OpenAI's web Tokenizer shows exactly how any text splits into tokens [2].
- Programmatic: OpenAI's tiktoken library counts plain text in your own code; for a full request, a dedicated token-counting API includes the structural tokens that message framing, tools, and files add [1][2].
- Before sending to Claude: Anthropic's token-counting endpoint accepts the same input you would send to the model — system prompt, tools, images, documents — and returns an estimate of the input tokens [3].
Every count is an estimate, and a plain-text count never includes everything a real request adds [1][3]. Treat the numbers as a budgeting tool, not a receipt.
Part 3: What Tokens Explain
Token mechanics explain several things people otherwise find arbitrary.
Cost. API pricing is quoted per million tokens, and input, cached input, and output tokens carry different rates [1]. Comparing models on price per token alone is misleading: models tokenize the same text differently and generate different amounts of output and reasoning [1]. The honest comparison is total tokens times total rates on a representative task.
Limits. A context window is a token budget, and a maximum output is another one [1]. For reasoning models, the visible answer and the hidden reasoning share the same output budget [1]. Filling the window with input leaves no room for the answer.
Rare characters and emoji. Text outside the tokenizer's base vocabulary tends to fragment into more tokens. Byte-level BPE prevents outright failure — any character, including emoji, can still be represented — but less common text still splits into more pieces [4]. And since tokenization is deterministic, duplicated content is tokenized and billed again every time it appears [1].
Spelling and letter tasks. Questions that depend on individual characters — counting letters, spotting typos, reading acrostics — are harder for models than they sound, because the tokenizer's split may not align with the characters a human sees [1]. The model is not careless; it is working from tokens.
Non-English text. A tokenizer trained on an English corpus performs poorly on languages like Japanese, because the use of spaces and punctuation is very different [4]. The practical effect: token efficiency varies by language, and the same idea can consume a different amount of window and cost depending on the language it is written in [1][4].
Part 4: Why Tokens Matter
Tokens are the unit everything else is denominated in. Three things follow directly:
- The price you pay is in tokens, not words. Word-based intuition undershoots cost, especially with reasoning models [1].
- The window you fill is in tokens, not pages. Whether your material fits is a token question, and a different model can give a different answer for identical text [1][3].
- The quirks you notice are token quirks. Letter puzzles and non-English cost gaps trace back to what the tokenizer chose [1][4].
When something misbehaves, look at the tokens first. Most tools that expose the tokenizer let you paste the exact input and see the split — it takes seconds and usually explains the behavior [2].
Common Misconceptions
| Belief | Reality |
|---|---|
| "A token is a word." | Often a fragment. One token is about three-quarters of a word in English [1]. |
| "Token counts are the same across models." | Each model has its own encoding; the same text counts differently on each [1]. |
| "Only the visible answer is billed." | Reasoning tokens are invisible but billed as output [1]. |
| "Emoji and rare characters break models." | Byte-level BPE represents any character; uncommon text just splits into more tokens [4]. |
| "Non-English and emoji cost the same as English." | Text outside the tokenizer's training data splits into more tokens [4]. |
| "Once measured, a token count is final." | New tokenizer releases change counts; recount against the target model [3]. |
| "The tokenizer follows spelling rules." | It follows learned merge frequencies from training data, not a dictionary [4]. |
Glossary
| Term | Meaning |
|---|---|
| Token | The unit of text a model reads and generates; character, word fragment, word, or punctuation [1] |
| Tokenizer | The component that splits text into tokens and maps them to IDs [1][4] |
| Tokenization | The process of splitting text into tokens [4] |
| Byte-pair encoding (BPE) | The dominant tokenization algorithm: start from a base vocabulary, learn merges of frequent pairs [4] |
| Byte-level BPE | BPE on raw bytes, so any character can be represented, including emoji [4] |
| Merge rule | A learned rule joining two tokens into one [4] |
| Encoding | The specific tokenizer scheme a model uses [1] |
| Input / prompt tokens | Tokens sent to the model [1] |
| Output / completion tokens | Tokens the model generates [1] |
| Cached input tokens | Reused input tokens billed at a lower rate [1] |
| Reasoning tokens | Invisible internal tokens billed as output [1] |
| Context window | The total token budget for one request [1] |
| tiktoken | OpenAI's library for counting tokens programmatically [2] |
References and Helpful Links
- Understanding and counting tokens (OpenAI Help Center). The primary reference for token estimates, input/output/cached/reasoning categories, limits, and pricing [1].
- OpenAI Tokenizer and tiktoken. See how any text splits into tokens, and count tokens in code [2].
- Token counting (Anthropic Docs). Counting tokens before sending, and how newer tokenizers change counts [3].
- LLM Course, Chapter 6: Tokenizers (Hugging Face). How tokenizers are trained; byte-pair encoding step by step [4].
- Companion guide: What Is a Prompt? — where tokens appear first in the pipeline, and what the model does with them after tokenization.