What Is a Token in AI? Tokens, Context Windows and Cost, Explained

What a token is

Before a language model reads your prompt, a component called the tokenizer cuts the text into pieces from a fixed vocabulary and replaces each piece with a number. Those pieces are tokens. Common words are a single token. Rare or long words are split into several. Spaces are usually attached to the word that follows, and punctuation marks are tokens of their own.

Text	Typical split	Tokens
The cat sat.	The · cat · sat · .	4
tokenization	token · ization	2
unbelievably	un · bel · iev · ably	3–4
3.14159	3 · . · 141 · 59	3–5
print("hi")	print · (" · hi · ")	4

Illustrative. The exact split differs from one tokenizer to another.

The model's entire job is to predict the next token ID given the previous ones. It produces one token, appends it to the sequence, and repeats. Everything a chatbot writes is generated this way, one token at a time, which is why responses stream out in small bursts rather than appearing all at once.

Why tokens and not words or letters?

Letters would make sequences very long and force the model to relearn spelling constantly. Whole words would need a vocabulary of millions of entries and still fail on typos, names and new terms. Sub-word tokens are the compromise. Most tokenizers are built with byte-pair encoding (BPE) or a close relative: start from individual bytes, then (following the method introduced for machine translation by Sennrich et al.) repeatedly merge the most frequent adjacent pair in a large text corpus until the vocabulary reaches a target size. Frequent strings end up as single tokens; anything else can still be spelled out from smaller pieces, so no input is ever "out of vocabulary."

Vocabulary size has grown over time. GPT-2 used about 50,000 tokens; Llama 3 uses about 128,000; the Qwen family about 150,000; Gemma about 260,000. A larger vocabulary represents text, especially non-English text, in fewer tokens. You can see the vocabulary for any open model in the tokenizer.json file of its Hugging Face repository, for example Qwen/Qwen3-8B.

Tokens vs words: conversion rules of thumb

Content	Approximate tokens
1 English word	≈ 1.3 tokens
1 page of English prose (≈ 500 words)	≈ 650–700 tokens
A 10-page report	≈ 7,000 tokens
A 300-page novel (≈ 90,000 words)	≈ 120,000 tokens
French, German, Spanish	≈ 1.2–1.5× the English count for the same meaning
Source code	Often 1.5–2× prose of the same length, because of symbols and indentation

Two consequences people trip over. First, token counts are not portable between models: the same document might be 10,000 tokens for one model and 12,000 for another. Second, models are clumsy with anything below the token level. Counting the letters in a word or reversing a string is hard for an LLM precisely because it never sees the letters, only the chunk IDs. OpenAI's tokenizer playground is a quick way to see the splits for yourself.

Tokens and the context window

A model's context window is the maximum number of tokens it can consider at once. It includes everything: the system prompt, tool definitions, the full conversation history, any documents you pasted, and the answer being written. When the total exceeds the window, the oldest content is dropped or the request fails.

Model (BestLLMfor catalog)	Context window	Roughly equivalent to
DeepSeek R1 Distill 32B	32,768 tokens	≈ 50 pages
gpt-oss 20B, Llama 3.3 70B, Mistral Small 3.2 24B	128,000 tokens	≈ 190 pages
Qwen 3 8B / 14B / 32B	131,072 tokens	≈ 200 pages
Gemma 4 12B, Qwen 3.8 27B	262,144 tokens	≈ 400 pages

The advertised window is a ceiling, not a default. Local runtimes typically allocate far less unless you ask, because context costs memory. What those numbers mean in practice, and why quality degrades in very long contexts, is covered in LLM context windows explained.

What a token costs

On a cloud API: dollars

Hosted models bill per million tokens, with output priced several times higher than input, because generating is sequential while reading is parallel. "Reasoning" models also bill for the hidden thinking tokens they produce before answering, which can multiply the output count. To estimate a monthly bill from your own usage, and the point at which owning hardware becomes cheaper, use the cloud vs local cost calculator.

On your own GPU: memory

Locally, tokens are free to generate but not free to hold. For every token in the conversation, the model stores a key and a value vector in each layer: the KV cache. Its size per token is fixed by the architecture.

Model	Layers × KV heads × head dim	KV cache per token (FP16)	Per 8,000 tokens	Per 32,000 tokens
Llama 3.1 8B	32 × 8 × 128	0.13 MB	1.0 GB	4.2 GB
Qwen 3 8B	36 × 8 × 128	0.15 MB	1.2 GB	4.7 GB
Qwen 3 14B	40 × 8 × 128	0.16 MB	1.3 GB	5.2 GB
Qwen 3 32B	64 × 8 × 128	0.26 MB	2.1 GB	8.4 GB
Llama 3.3 70B	80 × 8 × 128	0.33 MB	2.6 GB	10.5 GB

Computed as 2 × layers × KV heads × head dimension × 2 bytes, from each model's published configuration. Quantizing the cache to 8-bit halves these figures.

This is why a model that loads fine can still run out of memory halfway through a long document: the weights fit, the tokens did not. The trade-offs between context length and VRAM are worked through in context window vs VRAM cost.

Tokens per second

Speed is measured in tokens too. Two different rates matter. Prompt processing (prefill) is how fast the model reads your input, typically hundreds to thousands of tokens per second. Generation is how fast it writes, typically 10 to 150 tokens per second on consumer hardware. A person reads at roughly 5 to 8 tokens per second, so anything above about 10 feels live; agents and coding tools that generate a lot of unseen text benefit from 30 and up. Benchmarks and what affects them are in LLM tokens per second.

Five practical ways to spend fewer tokens

1. Trim the system prompt. It is re-sent with every single turn.
2. Start a new chat for a new topic. Old history is paid for again on each message.
3. Paste the relevant excerpt, not the whole file. Or use retrieval so only matching passages enter the context.
4. Disconnect tools you are not using. Each tool definition costs 100 to 300 tokens per turn; see what MCP is for the numbers.
5. Cap the output. Set a maximum response length, and ask for a specific format rather than an open-ended essay.

Context sizes and model figures on this page come from the BestLLMfor catalog, available through our public API (CC BY 4.0) and open-source MCP server.