LLM quantization, explained.
TL;DR: quantization stores a model’s weights with fewer bits — 4 instead of 16 — shrinking it ~4x with a quality loss most people never notice. It’s the only reason 27B-class models run on consumer GPUs at all. Default to Q4_K_M; step up to Q5/Q6 if you have spare VRAM, down to Q8 only when quality is critical and memory is not.
Why quantization exists
A model’s weights are billions of numbers, natively stored in 16 bits each — 2 GB per billion parameters. A 27B model at FP16 is ~54 GB: no consumer GPU holds that. Store each weight in ~4.5 bits instead and the same model drops to ~16 GB — inside a 24GB card with room for context. That compression is quantization, and it’s why the entire local-LLM scene exists.
How it works (no math degree required)
Weights are rounded onto a coarser grid. Done naively that destroys quality, so modern schemes work block by block: each small group of weights gets its own scale factor, keeping the rounding error locally small. The “K-quants” used in GGUF files go further, spending more bits on the tensors that matter most and fewer elsewhere. Result: a 4-bit model that behaves remarkably like its 16-bit parent.
Decoding the names
| Name | ≈ bits/weight | What it means |
|---|---|---|
| Q8_0 | 8.5 | Near-lossless; ~1.07 GB per B params — the quality reference |
| Q6_K | 6.6 | Practically indistinguishable from Q8 in normal use |
| Q5_K_M | 5.7 | Comfortable middle ground when you have VRAM to spare |
| Q4_K_M | 4.8 | The default. Best quality-per-GB trade for most models |
| Q4_0 | 4.5 | Older, simpler 4-bit scheme — prefer Q4_K_M when both exist |
| Q3_K_M / Q2_K | 3.9 / 3.4 | Noticeable degradation — last resort to squeeze into small VRAM |
| IQ4_XS / IQ3… | 4.3 / ~3.5 | Newer "importance" quants — better quality at low bits, slightly slower |
In Q4_K_M: Q4 = ~4-bit, K = k-quant block scheme, M = medium variant (S/M/L trade size vs quality). GGUF is the file format that carries these — the one Ollama, LM Studio and llama.cpp all read. GPTQ, AWQ and EXL2 are sibling formats for GPU-only runtimes; same idea, different packaging.
What it costs in quality
Measured honestly: Q8 is a rounding error from FP16. Q5/Q6 sit within noise on most benchmarks. Q4_K_M loses a little — occasionally visible on hard reasoning and long code — and is still where the community lives, because the alternative is usually running a smaller model at higher precision, which loses more. Q3 and below degrade visibly. One counter-intuitive rule: a bigger model at Q4 beats a smaller model at Q8 in almost every real comparison.
The size math, ready to use
| Model | Q4_K_M | Q8_0 | FP16 |
|---|---|---|---|
| 9B | ~6 GB | ~10 GB | ~18 GB |
| 12B | ~7 GB | ~13 GB | ~24 GB |
| 27B | ~16 GB | ~29 GB | ~54 GB |
| 31B | ~18 GB | ~33 GB | ~62 GB |
| 70B | ~42 GB | ~75 GB | ~140 GB |
Weights only — add KV cache and overhead (~20% at 8K context, more beyond) to know what actually fits: that’s exactly what the LLM VRAM calculator computes. Then pick the model itself from the leaderboard or your hardware’s ranking.
Frequently asked questions
What does Q4_K_M mean?
A GGUF quantization label: Q4 = weights stored in ~4 bits (4.8 including block scales), K = the k-quant block scheme, M = the medium size/quality variant. It's the community default because it offers the best quality per gigabyte for most models.
Does quantization make a model dumber?
Slightly, and less than you'd think: Q8 is near-lossless, Q4_K_M loses a little on hard reasoning, Q3 and below degrade visibly. A larger model at Q4 almost always beats a smaller one at Q8.
What's the difference between GGUF and GPTQ or AWQ?
They're competing packaging formats for quantized models. GGUF (llama.cpp ecosystem) runs on CPU, GPU or mixed, and is what Ollama and LM Studio use. GPTQ, AWQ and EXL2 target GPU-only runtimes. Same core idea; pick the format your runtime reads.
Which quantization should I download?
Q4_K_M by default. If the model still leaves several GB of VRAM free, take Q5_K_M or Q6_K. Only go Q3/Q2 when nothing else fits — and consider a smaller model instead. Check fit first with a VRAM calculator.
By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.