Guide · 2026-05-16

Q4 vs Q5 vs Q8 Quantization — Real Quality Loss Tested

We benchmarked Q4_K_M, Q5_K_M, and Q8_0 across coding, reasoning, and long-context tasks. Here is the actual quality loss, in numbers, not vibes.

By Mohamed Meguedmi · 11 min read

The internet keeps telling you Q4_K_M is "good enough" and Q8 is "near lossless." Both statements are half-true and depend on the model size, the task, and how much you care about the tail of the output distribution. The BestLLMfor editorial team ran a controlled benchmark suite across three weight classes (8B, 32B, 70B) and four tasks. This guide gives you the actual numbers — and a verdict you can act on today.

What "Q4", "Q5", and "Q8" actually mean

Modern GGUF quantization is not the old uniform integer rounding that scared everyone in 2023. The "K-quants" family — introduced by Georgi Gerganov and the llama.cpp community — uses block-wise mixed-precision encoding with super-block scales. The trailing letters tell you the variant inside each tier: _K_S (small, more aggressive), _K_M (medium, the default), _K_L (large, slightly heavier).

The headline number is the average bits per weight, not a uniform precision:

• Q4_K_M — ~4.83 bits/weight. Mixes Q4 and Q6 blocks; attention.wv and feed_forward.w2 tensors get the Q6 treatment.
• Q5_K_M — ~5.69 bits/weight. Q5 base, Q6 for the sensitive tensors.
• Q8_0 — 8.5 bits/weight. Uniform Q8 with FP16 scale per 32-element block. No K-quant trickery needed at this precision.

For the deep dive, read the original llama.cpp K-quants PR and the HuggingFace GGUF spec. The implementation detail that matters in practice: K-quants are not all-or-nothing. A "Q4" model still contains millions of Q6 weights in the layers that perplexity tests have shown to be most sensitive.

The test setup

All measurements were captured between 2026-04-22 and 2026-05-12 on identical inference nodes. Methodology is fully documented in our benchmark methodology page; raw runs are reproducible from the BestLLMfor public benchmark API (CC BY 4.0).

• Inference engine: llama.cpp build b4521 (CUDA 12.4) — no speculative decoding, deterministic sampling (temperature 0, top_p 1.0) for benchmark runs.
• Models: Qwen3-Instruct 4B, Llama 3.3 8B, Qwen3-Coder 32B, Llama 3.3 70B. FP16 baselines pulled from the official HuggingFace model cards.
• Benchmarks: WikiText-2 perplexity, MMLU-Pro (5-shot), HumanEval+ (pass@1, n=20), and a 16k-token needle-in-a-haystack retrieval set.
• Hardware: single RTX 4090 24 GB for ≤32B, dual RTX 4090 for 70B. Identical thermals, identical driver (NVIDIA 555.42).

We report the delta from each model's own FP16 score, not absolute numbers. That isolates the quantization effect from architectural differences.

VRAM and disk footprint — the part everyone cares about first

Model	Q4_K_M (GB)	Q5_K_M (GB)	Q8_0 (GB)	FP16 (GB)
Qwen3 4B	2.5	2.9	4.3	8.1
Llama 3.3 8B	4.9	5.7	8.5	16.1
Qwen3-Coder 32B	19.8	23.4	34.8	65.5
Llama 3.3 70B	42.5	49.9	74.9	141.1

Add 1.5–4 GB for KV cache at 8k context (more at long contexts, more again on multi-GPU). The takeaway: Q5_K_M vs Q8_0 is a 30–32% smaller artifact every single time. Q4_K_M vs Q8_0 is roughly 43% smaller. These ratios hold consistently from 4B all the way to 70B, which is convenient when you size up to a bigger model later.

If you want to model the cost side (cloud GPU rental vs owning a card), our cost calculator takes quant level and tokens/month as inputs and tells you when local hardware breaks even. French readers: the same calculator is mirrored on quelllm.fr.

Quality loss — the actual numbers

Each cell is the delta from that model's FP16 baseline. Negative = worse. Perplexity is reported as a percentage increase (lower is better, so positive means worse).

Model / Quant	PPL Δ (%)	MMLU-Pro Δ	HumanEval+ Δ	16k Needle Δ
Qwen3 4B — Q4_K_M	+3.8%	-2.7 pts	-6.4 pts	-8.1 pts
Qwen3 4B — Q5_K_M	+1.6%	-1.1 pts	-2.2 pts	-3.0 pts
Qwen3 4B — Q8_0	+0.4%	-0.3 pts	-0.5 pts	-0.6 pts
Llama 3.3 8B — Q4_K_M	+2.1%	-1.4 pts	-3.1 pts	-4.0 pts
Llama 3.3 8B — Q5_K_M	+0.9%	-0.5 pts	-1.0 pts	-1.3 pts
Llama 3.3 8B — Q8_0	+0.2%	-0.1 pts	-0.2 pts	-0.3 pts
Qwen3-Coder 32B — Q4_K_M	+1.3%	-0.9 pts	-1.8 pts	-2.4 pts
Qwen3-Coder 32B — Q5_K_M	+0.5%	-0.3 pts	-0.7 pts	-0.9 pts
Qwen3-Coder 32B — Q8_0	+0.1%	-0.0 pts	-0.2 pts	-0.2 pts
Llama 3.3 70B — Q4_K_M	+0.9%	-0.6 pts	-1.1 pts	-1.5 pts
Llama 3.3 70B — Q5_K_M	+0.3%	-0.2 pts	-0.4 pts	-0.5 pts
Llama 3.3 70B — Q8_0	+0.05%	-0.0 pts	-0.1 pts	-0.1 pts

Three patterns jump out, and they all repeat in independent published work like the "Effect of Quantization on LLM Performance" study (arXiv 2402.16775):

1. Quantization hurts small models proportionally more. The same Q4_K_M setting costs Qwen3 4B 6.4 points on HumanEval+ but only 1.1 points at 70B. The mechanism is intuitive — smaller models have less redundancy to absorb rounding error.
2. Code generation is the most sensitive task. HumanEval+ deltas are consistently ~2x worse than MMLU deltas. Code requires exact tokens; reasoning often has multiple acceptable paths.
3. Long context amplifies everything. The 16k needle test is the worst case in every row, because retrieval errors compound across attention layers that have been rounded.

Speed: does smaller mean faster?

Counter-intuitive answer: on GPU, the speedup from going lower than Q5_K_M is small or negative once the model already fits in VRAM. Memory bandwidth dominates, and dequantization overhead eats some of the win. Measurements below are tokens/sec for single-stream generation, batch size 1, 512-token output, on a single RTX 4090.

Model	Q4_K_M tok/s	Q5_K_M tok/s	Q8_0 tok/s
Llama 3.3 8B	118	112	96
Qwen3-Coder 32B	41	38	27
Llama 3.3 70B (2x 4090)	17	15	10

Q4_K_M gives roughly 5–8% more tokens/sec than Q5_K_M. That is rarely worth the quality regression. The real speed-vs-quality bargain is Q8_0 → Q5_K_M, which buys you ~40% throughput at almost no quality cost. On Apple Silicon and CPU the bandwidth advantage of Q4 widens, but the conclusion holds: Q5_K_M is still the better default unless you are memory-bound.

When Q4 wins anyway

Q4_K_M earns its keep in three concrete scenarios:

• You are running ≥32B models on 24 GB VRAM and Q5 simply will not fit alongside your context window. Qwen3-Coder 32B at Q5_K_M plus 32k context exceeds 24 GB once you add the KV cache; Q4_K_M leaves headroom.
• You are serving many concurrent users. Higher VRAM headroom means larger batch sizes, and batching throughput dominates per-token quality for chat-style workloads.
• You are running on Apple Silicon with unified memory pressure. The bandwidth gap between Q4 and Q5 is more visible on M-series chips. Ollama's default tag for qwen3-coder is Q4_K_M for exactly this reason.

Outside those cases, Q5_K_M is the editorially recommended default.

When Q8 is worth the disk

Pick Q8_0 when:

• You are using the model as a distillation teacher or for generating synthetic training data. Tail probabilities matter, and Q8 preserves them.
• You run tool-using agents that emit strict JSON / function-call payloads where one bad token cascades. Q8's ~0.1-point HumanEval delta translates into measurably fewer schema-violation retries.
• You are auditing or comparing models for a customer-facing recommendation where a fraction of a benchmark point is editorially relevant.

For everything else — daily coding, drafting, summarization, RAG over your docs — Q8's extra fidelity is statistically real and practically invisible.

Putting it on disk: a 4-step install

1. Pick the file. On HuggingFace, look for a repo published by bartowski, QuantFactory, or the model author. Pick the *-Q5_K_M.gguf variant. Verify the SHA256 against the model card.
2. Download with hf-transfer enabled for faster pulls: HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download <repo> <file>.gguf --local-dir ./models.
3. Load in llama.cpp: ./llama-server -m ./models/<file>.gguf -c 8192 -ngl 99. The -ngl 99 offloads every layer it can to GPU.
4. Sanity check. Run a 10-prompt smoke test against your usual workload before swapping out a higher-quant version. The quelllm-mcp open-source server ships a ready-made eval harness.

The verdict

Your situation	Pick	Why
Default daily driver, 7B–32B on a single 24 GB GPU	Q5_K_M	Best quality/VRAM ratio in every measured slice.
32B+ with long context (≥16k) on 24 GB VRAM	Q4_K_M	Only quant that leaves room for KV cache.
Agentic tool-use, JSON-strict outputs, distillation	Q8_0	Tail-probability fidelity matters.
4B model for edge / on-device	Q8_0 or FP16	Small models punish quantization. Pay the storage.
70B+ on multi-GPU	Q4_K_M or Q5_K_M	Quality loss is <1%. Both are excellent.

One sentence to take with you: start at Q5_K_M, drop to Q4_K_M only when VRAM forces it, and reach for Q8_0 only when you can name the tail-probability reason you need it.

Reviewed by the BestLLMfor editorial team. Last updated 2026-05-16. Raw benchmark data is available under CC BY 4.0 via the BestLLMfor public API.