Guide · 2026-05-16

ExLlama vs vLLM vs llama.cpp — Production Inference Compared

Three engines, three philosophies. We benchmark ExLlamaV2, vLLM, and llama.cpp on real production workloads and tell you which one to ship.

By Mohamed Meguedmi · 11 min read

The local-LLM serving stack has consolidated. After two years of churn, three engines now cover roughly 95% of self-hosted production deployments: vLLM for GPU-backed APIs, ExLlamaV2 for latency-sensitive consumer-GPU workloads, and llama.cpp for everything portable. This guide cuts through the marketing benchmarks and tells you which one to ship — with numbers.

The Three Engines in One Paragraph

vLLM is a Python-first, CUDA-heavy inference server from UC Berkeley built around PagedAttention and continuous batching. It is the de facto standard for serving open-weight models at scale and powers production inference at Anyscale, Mistral, and most managed-inference providers. ExLlamaV2 (and the newer ExLlamaV3) is turboderp's hand-tuned CUDA kernels for the EXL2/EXL3 quantization formats, optimized for single-user latency on consumer NVIDIA cards. llama.cpp is Georgi Gerganov's portable C/C++ inference library powering Ollama, LM Studio, Jan, and roughly every "run an LLM on your laptop" tool. It uses GGUF quantization and runs on CPU, Metal, CUDA, ROCm, Vulkan, and SYCL.

Architecture: Why These Engines Behave Differently

The performance gaps between these three engines are not micro-optimizations. They come from fundamentally different scheduling and memory models.

Capability	vLLM	ExLlamaV2	llama.cpp
Language	Python + CUDA C++	Python + CUDA C++	C/C++
Batching	Continuous (paged)	Dynamic (paged in v0.1+)	Continuous (server only)
KV-cache	PagedAttention	Paged KV	Contiguous + flash-attn
Multi-GPU	Tensor + pipeline parallel	Tensor parallel	Layer split (RPC tensor split)
Quantization	AWQ, GPTQ, FP8, INT4, BitsAndBytes	EXL2 (2–8 bpw), EXL3	GGUF (Q2_K → Q8_0, IQ-series)
CPU inference	Limited (CPU offload)	None	First-class
Backends	CUDA, ROCm, TPU, Neuron	CUDA only	CUDA, Metal, ROCm, Vulkan, SYCL, CPU
OpenAI-compatible API	Built-in	Via TabbyAPI	Via llama-server

The three architectural choices that matter most in production:

• PagedAttention (vLLM) treats the KV-cache like virtual memory — pages can be allocated, freed, and shared between requests. This is why vLLM can serve 50+ concurrent users on a single A100 without OOM, while a naive transformer pre-allocates worst-case KV for each request.
• EXL2 quantization (ExLlamaV2) stores different layers at different bit-widths, packed into a single file. A 4.65 bpw EXL2 model is genuinely smaller and faster than a Q4_K_M GGUF on the same hardware — but only on NVIDIA, only with batch size 1–4.
• GGML graph compilation (llama.cpp) generates a portable compute graph at load time and dispatches to whatever backend is present. This is why a single GGUF file runs identically on a Mac, a Raspberry Pi 5, and an RTX 5090.

Benchmark: Throughput on a Single GPU

Benchmarks below aggregate the llama.cpp performance discussion, the Red Hat vLLM vs llama.cpp report, and our internal regression suite (methodology at /methodology/). Model: Llama-3.1-8B-Instruct, 4-bit quantization equivalent (AWQ for vLLM, EXL2 4.65 bpw for ExLlamaV2, Q4_K_M for llama.cpp). Prompt: 512 input tokens, 256 output tokens. Hardware: NVIDIA RTX 4090, 24 GB VRAM, CUDA 12.4.

Engine	Batch 1 (tok/s)	Batch 8 (tok/s aggregate)	Batch 32 (tok/s aggregate)	TTFT batch 1 (ms)
ExLlamaV2 0.2.x	148	720	OOM	41
vLLM 0.7.x	97	1,840	4,200	78
llama.cpp b4400	91	410	Crashes > 16	62

The pattern is consistent across model sizes from 7B to 70B: ExLlamaV2 wins single-stream latency by a clear margin, vLLM wins concurrent throughput by 2–10×, and llama.cpp is the slowest of the three on dedicated GPU but the only one that survives a 96 GB Mac Studio.

Multi-GPU: Where llama.cpp Falls Apart

If you have 2× RTX 3090, 2× RTX 4090, or a server with 4–8× A100/H100, the choice narrows immediately. llama.cpp implements multi-GPU as layer splitting: layers 0–15 on GPU 0, layers 16–31 on GPU 1, etc. Only one GPU is computing at any moment. The community discussion documented this extensively in 2025 and the conclusion holds: on a 2× 3090 setup running Llama-3 70B, vLLM with tensor parallelism delivers roughly 2.3× the throughput of llama.cpp split mode, and ExLlamaV2 TP lands in between.

Practical rule: if you bought a second GPU specifically to serve LLMs, you bought it for vLLM or ExLlamaV2. Layer-split llama.cpp is fine for occasionally running a model that doesn't fit on one card, but it is not a production serving topology.

Concurrency: The 5-User Test

The most-cited 2026 stress test on Towards AI showed Ollama (a llama.cpp wrapper) collapsing at 5 concurrent users while vLLM maintained sub-second TTFT through 50+. Our own load test on Qwen3-32B AWQ (vLLM) vs Qwen3-32B Q4_K_M (llama-server) on a single H100 80 GB:

Concurrent users	vLLM p50 TTFT	vLLM p99 TTFT	llama-server p50 TTFT	llama-server p99 TTFT
1	89 ms	112 ms	74 ms	98 ms
8	180 ms	340 ms	1,400 ms	3,100 ms
32	410 ms	780 ms	Timeout	Timeout
64	720 ms	1,380 ms	—	—

llama-server has continuous batching since mid-2024 and the gap is smaller than it was, but vLLM's paged KV-cache and prefix caching still pull ahead decisively above ~8 concurrent users. For comparing total cost of ownership at different concurrency targets, the cost calculator lets you plug in TPS and amortize across requests.

The Install & Operate Reality

Performance is one axis. Operational cost is the other. The story here is reversed.

llama.cpp: trivial to deploy

# Single binary, no Python, no CUDA toolkit
curl -L https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-bin-linux-x64.tar.gz | tar xz
./llama-server -m qwen3-32b-q4_k_m.gguf -ngl 99 --port 8080

vLLM: opinionated, Python-heavy

pip install vllm  # 4+ GB of dependencies
vllm serve Qwen/Qwen3-32B-AWQ \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

ExLlamaV2 via TabbyAPI: middle ground

git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI && pip install -e .[cu121]
python main.py  # OpenAI-compatible on :5000

vLLM's dependency surface is its operational tax. Upgrading CUDA, PyTorch, or transformers regularly breaks vLLM in subtle ways. llama.cpp's static binary attitude is the opposite — it ships nightlies and the worst case is recompiling. ExLlamaV2 sits between: less stable than llama.cpp, less fragile than vLLM, but the EXL2 quantization step itself takes several hours on consumer hardware.

Quantization: The Hidden Lock-In

Each engine prefers its own quantization format and the formats are not portable.

• vLLM ingests GPTQ, AWQ, FP8 (Hopper+), Marlin, and BitsAndBytes. AWQ 4-bit is the production-default; FP8 dynamic is best on H100/H200. See the vLLM quantization docs.
• ExLlamaV2 uses EXL2 (or EXL3 since 2025), a measurement-guided mixed-precision format produced by turboderp's exl2 converter. EXL2 at 4.65 bpw typically matches Q5_K_M GGUF in perplexity while being faster to generate.
• llama.cpp uses GGUF — by far the broadest quantization menu, from Q2_K (extreme compression) through IQ4_XS (good perplexity at low bpw) to Q8_0 (near-lossless). The official guidance is on the HuggingFace GGUF page.

This matters because you cannot pre-quantize once and serve everywhere. Choose the engine first, then download or produce the matching format. The BestLLMfor public API (CC BY 4.0) exposes a per-engine compatibility matrix you can query directly, and the open-source quelllm-mcp server makes the same data available to any MCP-aware client.

The Verdict Table

Scenario	Pick	Why
Serving 8B–70B to 10+ concurrent users on a server GPU	vLLM	Continuous batching, tensor parallelism, FP8 on Hopper
Single-user chat on RTX 3090/4090/5090	ExLlamaV2	Fastest tok/s at batch 1, EXL2 fits more model per VRAM
Apple Silicon (M-series Mac)	llama.cpp	Only engine with first-class Metal backend
CPU-only or hybrid CPU+GPU server	llama.cpp	Layer offload + AVX-512/AMX kernels
Windows desktop deployment	llama.cpp	vLLM on Windows is WSL-only and brittle
Multi-GPU consumer build (2× 3090, 2× 4090)	ExLlamaV2 or vLLM	Both support tensor parallelism; ExLlamaV2 is leaner on RAM
Datacenter H100/H200/MI300 fleet	vLLM	FP8, ROCm, prefix caching, speculative decoding
Edge device, Raspberry Pi, NPU	llama.cpp	Only realistic option
Maximum raw speed on a 4090 for one user	ExLlamaV2	50–85% faster than llama.cpp at batch 1

The Editorial Recommendation

If you are building a product, default to vLLM. Pay the Python tax, get continuous batching, FP8, prefix caching, and a maintained OpenAI-compatible server. If you are building a desktop application or anything that ships to end-user hardware, default to llama.cpp — it is the only engine that runs on every platform your users actually own. Reach for ExLlamaV2 when you have one or two NVIDIA cards, you control the deployment, and you care about per-token latency more than aggregate throughput.

The most common mistake we see in 2026 is teams running Ollama (llama.cpp under the hood) in production behind a load balancer, then being surprised when p99 latency explodes at moderate concurrency. Ollama is a fantastic developer experience; it is not a serving stack. More on our editorial methodology at /about/.