What Is a Good Tokens-per-Second Speed?

Start with reading speed, not the spec sheet

A token is roughly three-quarters of an English word. A comfortable adult reading pace is about 200-300 words per minute, which works out to around 4-7 tokens per second. So the honest baseline is blunt: if a model generates faster than you can read, extra llm tokens per second stop mattering for a chat you actually read line by line.

Where higher speed earns its keep is everything you don't read token by token: long code blocks you scan, summaries you skim, and agentic loops where the model calls tools and re-generates many times before you see a final answer. In those cases total wall-clock time is what you feel, and that is throughput times length. Treat reading speed as the floor, not the goal, and be suspicious of any benchmark that quotes one number without saying what it measured. Our leaderboard and methodology pages break the numbers apart so you can compare like with like.

Prompt speed and generation speed are two different numbers

Every request has two phases. Prefill (prompt processing) reads your entire input at once; it is compute-bound and highly parallel, so it can run at hundreds or thousands of tokens per second and mostly determines your time to first token. Decode (generation) produces the answer one token at a time; it is limited by memory bandwidth, and this is the smaller number people usually mean by "tokens per second."

The distinction is practical. A short question against a small model feels instant because prefill is trivial. Feed the same model a 6,000-token document for retrieval-augmented generation and the wait shifts almost entirely to prefill, even though your quoted decode speed never changed. Always look at both:

What counts as good: a practical scale

Numbers below are for interactive, single-user, decode speed on consumer hardware. They are perception thresholds, not model scores, so they hold up regardless of which model or backend you run.

Most people over-optimize this. If you mainly chat, 15 tok/s is plenty; chasing 60 buys you nothing you can read. If you run agents or generate long files, aim higher because the total token count is large. See the best local models for code for where that trade-off bites.

What changes your tokens per second

Four things move the number, roughly in order of impact:

• Does it fit in VRAM? This is the cliff. A model that fits entirely on the GPU runs at full speed; spill even a few layers to system RAM (CPU offload) and decode can drop several-fold because CPU memory bandwidth is far lower.
• Quantization. A smaller weight format means fewer bytes to move per token, so lower precision usually decodes faster and frees VRAM. Q4_K_M is the common sweet spot; see quantization explained.
• Model size. More parameters means more memory traffic per token, so a 32B model is inherently slower to decode than an 8B, all else equal.
• Context length. A longer prompt and a growing KV-cache add memory pressure and slow both phases, which is why speed measured at 512 tokens flatters real use.

To predict whether a model fits, use these standard rules of thumb: FP16 weights are about 2 GB per billion parameters, Q8 about 1.07 GB/B, and Q4_K_M about 0.58 GB/B. Add roughly 20% for KV-cache and overhead at 8K context.

These are weight sizes; add ~20% for context. On a 16 GB card an 8B or 14B at Q4_K_M fits with room to spare, while a 32B at Q4_K_M spills over and forces offload, which is exactly where tokens per second collapse. Our VRAM calculator does this math for a specific config, and the best models for 8 GB covers smaller cards.

How to measure it honestly on your own machine

Vendor and benchmark claims rarely match your setup, so measure locally. Most runtimes print per-request stats: Ollama reports prompt and generation rates in its verbose output, and llama.cpp prints prefill and decode timings directly. Check the official library for the exact flags and model tags, since command details change.

Four habits keep the numbers honest. Run a warm-up request first so model-load time doesn't pollute the result. Report prefill and decode separately, never a blended figure. Test at a realistic context length, not an empty prompt. And run several times and take the median, because thermal throttling and background load create outliers. Once you have clean numbers, our benchmarks give a reference point measured the same way on one card.

Frequently asked questions

How many tokens per second do I need for a local LLM?

For chat you actually read, about 7-10 tokens per second matches comfortable reading speed, and 15-20 feels snappy. Agents, long code generation, and batch jobs benefit from more because they produce many tokens you don't read line by line. Below 5 tok/s is workable only for tasks you leave running.

What's the difference between prompt speed and generation speed?

Prompt speed (prefill) is how fast the model reads your input and sets your time to first token; it is compute-bound and often very fast. Generation speed (decode) is how fast it produces the answer, one token at a time, and is limited by memory bandwidth. Long prompts make prefill dominate, so quote both numbers.

Why is my local LLM suddenly so slow?

The most common cause is that the model no longer fits entirely in VRAM, so some layers spill to system RAM and decode speed drops sharply. A longer context, a larger quant, or a bigger model can each push you over the edge. Check VRAM usage and try a smaller quant like Q4_K_M.

Does quantization make tokens per second faster?

Usually yes. A lower-precision format moves fewer bytes per token and uses less VRAM, so it decodes faster and is more likely to fit fully on the GPU. Q4_K_M is the common balance of speed, size, and quality; extreme quants save more memory but can hurt output quality.

---

By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.