What GGUF actually is
GGUF stands for GPT-Generated Unified Format. It is a single-file binary container that packs a model's weights, tensors, tokenizer, and metadata into one .gguf file you can download and run. It is the native format of llama.cpp, the C/C++ inference engine that powers most local-LLM tooling, including Ollama and LM Studio.
GGUF replaced the older GGML format in 2023. The key improvement is that everything the runtime needs lives in one file with a structured, extensible key-value header — no separate config, tokenizer, or vocabulary files to track. The file is designed to be memory-mapped, so the runtime loads weights lazily and can share them across processes instead of copying the whole model into RAM at startup.
In practice, "a GGUF" means a model you can pull, drop on disk, and run on a CPU or a consumer GPU without a Python stack. That portability is why it dominates local inference.
GGUF and quantization
GGUF and quantization get mentioned together, but they are not the same thing. GGUF is the container; quantization is what you do to the weights before you store them in it. You can have an unquantized FP16 GGUF, but the reason the format matters to local users is that it ships with a mature set of low-bit quantization schemes baked in.
The naming looks cryptic but follows a pattern. Q4_K_M means 4-bit weights using the "K-quant" method at the medium size/quality tier; Q8_0 is a simpler 8-bit scheme; Q5_K_S is 5-bit small. More bits means larger files and less quality loss. For most people on a single GPU, Q4_K_M is the default sweet spot, and Q5_K_M or Q6_K buy back accuracy if you have the VRAM to spare.
If the quant suffixes are new to you, start with our quantization explainer — the trade-offs there apply directly to which GGUF file you should download.
GGUF vs safetensors vs GPTQ
GGUF is not the only way to distribute a model. The two you will see most often alongside it are safetensors and GPTQ, and they solve different problems.
| Format | Primary use | Quantized? | Typical runtime |
|---|---|---|---|
| GGUF | CPU + GPU local inference | Yes, built-in (Q2–Q8) | llama.cpp, Ollama, LM Studio |
| safetensors | Safe weight storage/loading | No (full precision) | Transformers, vLLM |
| GPTQ | GPU-only quantized inference | Yes, 3–4 bit | Transformers, vLLM, ExLlama |
safetensors is a storage format from Hugging Face that fixed the security problems of Python pickle files: it holds raw tensors and nothing executable, but it does not quantize and is not a runtime format. It is what you fine-tune, and what most models are published in first.
GPTQ is a GPU-only quantization method aimed at throughput. It can be excellent when the model fits entirely in VRAM, but it does not gracefully offload layers to CPU. GGUF's advantage is exactly that: it splits work across GPU and CPU, so you can run a model that is slightly too big for your card. For that split in detail, see Ollama vs llama.cpp.
How big is a GGUF file?
File size follows directly from parameter count and bits per weight. Multiply the billions of parameters by the per-B factor for the quant, then add roughly 20% for the KV cache and overhead at an 8K context window to estimate VRAM.
| Quant | GB per B params | 7B file size | 7B VRAM @ 8K |
|---|---|---|---|
| FP16 | 2.0 | ~14 GB | ~17 GB |
| Q8_0 | 1.07 | ~7.5 GB | ~9 GB |
| Q4_K_M | 0.58 | ~4.1 GB | ~4.9 GB |
These are rules of thumb; actual files vary a little by architecture. Use our VRAM calculator to check a specific model and context length before you download. On an 8 GB card, a 7B model at Q4_K_M fits comfortably, while a 13B needs offloading or a smaller quant — see the best models for 8 GB VRAM.
How to run a GGUF, and when to use it
You do not need to compile anything to use a GGUF. The simplest path is a runtime that wraps llama.cpp. With Ollama, models are pulled by name and the GGUF is fetched and cached for you; LM Studio gives a GUI over the same engine; and raw llama.cpp gives the most control over offload and sampling.
Because tags and filenames change, avoid hard-coding a specific file. The stable pattern is: pick a repository, choose a quant that fits your VRAM, and let your runtime download it. Check the model card for the exact quant tags, and confirm the file's checksum and license before running it.
The Hugging Face Hub lists GGUF files with per-quant sizes; its GGUF documentation and the format specification are the authoritative references if you want the byte-level details.
Frequently asked questions
Is GGUF the same as GGML?
No. GGUF replaced the older GGML format in 2023 and is not backward compatible. GGUF adds a structured, extensible metadata header so weights, tokenizer, and config all live in one self-describing file.
Which GGUF quant should I download?
For most single-GPU setups, Q4_K_M is the default sweet spot between size and quality. If you have spare VRAM, Q5_K_M or Q6_K reduce quality loss; Q8_0 is near-lossless but roughly twice the size of Q4_K_M.
Can I run a GGUF model without a GPU?
Yes. GGUF is designed for CPU inference through llama.cpp and runs on plain RAM, which is a major reason the format exists. A GPU speeds things up by offloading layers, but it is not required.
GGUF vs safetensors — which is better?
They serve different jobs, so neither is strictly better. Use GGUF to run models locally on consumer hardware, and use safetensors for full-precision storage, fine-tuning, and GPU serving with the Transformers stack.
By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.