What a Context Window Actually Is
A context window is the model's working set: every token it can attend to at once. That budget is shared between your input (system prompt, chat history, retrieved documents, code) and the output it generates. When people say a model has a "128K context," they mean it can process up to about 128,000 tokens total before older tokens must be dropped or the request is rejected.
Two misconceptions are worth killing early. First, the window is not memory: once a conversation scrolls past the limit, the earliest tokens are gone unless your tool re-sends them. Second, tokens are not words or characters. Modern transformers process text as tokens produced by a subword tokenizer, the mechanism formalized in the original Attention Is All You Need architecture. A rough rule for English is 1 token ≈ 0.75 words, so 128K tokens is around 96,000 words — a short novel. Code, JSON, and non-English text tokenize less efficiently, so a 10,000-line file can burn far more of your budget than its word count suggests.
The Memory Tax: KV Cache
The reason you can't just crank context to a million tokens on a desktop GPU is the KV cache. During generation the model stores the key and value vectors for every token it has already seen, so it doesn't recompute them each step. That cache lives in VRAM alongside the model weights, and it grows linearly with context length.
For a typical 8B model using grouped-query attention (roughly 32 layers, 8 KV heads, 128 head dimension), the FP16 KV cache costs about 128 KB per token. Multiply that out and the numbers get serious fast:
| Context length | Tokens | KV cache (FP16, ~8B GQA) |
|---|---|---|
| 8K | ~8,000 | ~1 GB |
| 32K | ~32,000 | ~4 GB |
| 128K | ~128,000 | ~16 GB |
| 256K | ~256,000 | ~32 GB |
That is on top of the weights themselves. An 8B model at Q4_K_M is roughly 4.6 GB; add a full 128K FP16 cache and you need over 20 GB — more than most consumer cards carry. Quantizing the KV cache to Q8 halves it, and Q4 quarters it, at some quality cost. Exact per-token size depends on layer count, head configuration, and whether the architecture uses GQA or full multi-head attention, so plug your specific model and target length into the VRAM calculator rather than guessing. If you are already tight on memory, our quantization guide covers where to trade weight precision versus cache precision.
Why Huge Contexts Degrade
Even when a model advertises 128K and your GPU can hold it, using all of it is rarely a good idea. Long-context accuracy is not uniform across the window. The well-documented "lost in the middle" effect — measured in this arXiv study — shows that models reliably use information at the very start and very end of a long input but frequently miss facts placed in the middle. Recall can degrade sharply long before the token limit is reached.
There are structural reasons. Attention cost scales quadratically with sequence length, so training on very long sequences is expensive and comparatively rare; most models see far more short examples than 100K-token ones. Techniques that stretch a model's window after training (position-interpolation and RoPE scaling variants) extend the maximum the model will accept without teaching it to reason as well over that full span. The result is a gap between the marketed number and the length at which answers stay trustworthy — sometimes called the effective context window.
- Dilution: a single relevant sentence in 100K tokens of noise is easy to overlook.
- Recency bias: instructions early in a huge prompt may be quietly overridden by later text.
- Latency: prompt processing time and time-to-first-token climb with every extra thousand tokens you stuff in.
The practical takeaway: retrieval that hands the model 4K of relevant tokens usually beats dumping 100K of loosely related ones.
How Much Context Do You Actually Need
Match the window to the job, not to the spec sheet. Most interactive chat and coding turns fit comfortably in 8K–16K. Summarizing a long document or working across several files pushes you toward 32K–64K. Whole-repository analysis or long agent runs are where six-figure windows earn their keep — and where you pay for them in VRAM and speed.
- Chat and Q&A: 8K is plenty; you rarely reference more than a few turns back.
- Coding assistant: 16K–32K covers a file plus its imports and tests.
- Document/RAG: 32K with good chunking beats 128K with none.
- Agents and codebases: 64K+, but budget the memory before you commit.
On a 12GB or 16GB card, the honest move is to run a smaller context than the maximum and reclaim that VRAM for a larger, more capable model or a faster quant. Our leaderboard and 12GB benchmark results report how models behave at the context lengths people actually use locally, not just their theoretical ceiling. If you are renting compute or comparing against an API, the cost calculator makes the token math concrete. Whatever you pick, confirm the exact advertised window and any KV-cache flags against the model card and your runtime's docs — those defaults change between releases.
Frequently asked questions
How many words is a 128K context window?
Roughly 96,000 English words, since one token averages about 0.75 words. Code, JSON, and non-English text tokenize less efficiently, so the same 128K tokens will hold noticeably fewer lines of source or characters of Unicode text.
Does a bigger context window use more VRAM?
Yes, and it grows linearly with length because the KV cache stores key and value vectors for every token in the window. For a typical 8B model a full 128K FP16 cache is around 16 GB on top of the weights, which is why long contexts often exceed consumer GPU memory.
Why do LLMs get worse with very long context?
Models are trained mostly on shorter sequences and tend to recall information at the start and end of a long input while missing facts in the middle. Attention also scales quadratically with length, so the effective, reliable context is usually shorter than the advertised maximum.
Is a larger context window always better?
No. A smaller window with good retrieval that supplies only relevant tokens usually beats a huge window filled with loosely related text, and it costs far less VRAM and latency. Match the window to the task instead of always maxing it out.
By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.