Guide · 2026-05-16

We Switched From Ollama to llama.cpp — What Broke

After three months running llama.cpp in production, here is the honest list of what we lost, what we gained, and whether the migration was worth the pain.

By Mohamed Meguedmi · 11 min read

Why we left Ollama in the first place

Ollama solved the cold-start problem for local LLMs — pull a model name, get a working chat endpoint, move on. For the first 18 months of this site's existence, that tradeoff was correct. It stopped being correct around late 2025, for three measurable reasons.

First, the lag on new architectures kept widening. When GLM-4.6 shipped, llama.cpp had a working GGUF converter within 48 hours; the Ollama registry took 11 days. For Qwen3-VL the gap was 21 days. If your workflow depends on testing the model the day it drops — which ours does — that lag is fatal.

Second, Ollama forked its inference engine. As discussed on Hacker News in August 2025, the project moved its default runner away from llama.cpp toward a Go-based engine for new model families. That means "Ollama uses llama.cpp under the hood" is no longer a complete description — and the two diverge on tokenization, KV-cache layout, and sampler defaults in ways that affect output.

Third, the abstraction was costing us real VRAM. On a 24 GB card running Qwen3-Coder 32B Q4_K_M with 32k context, Ollama reserved 1.8–2.4 GB more than a hand-tuned llama-server invocation. That is the difference between fitting an 8k draft model for speculative decoding and not.

The five things that broke on day one

1. Automatic model loading

Ollama loads a model on first request and evicts it after OLLAMA_KEEP_ALIVE (default 5 minutes). llama-server loads exactly one model at startup and serves it until killed. If you were relying on ollama run qwen3:32b from a script, that pattern is dead. You either keep a persistent server per model (RAM-expensive) or you bolt on llama-swap, which intercepts the OpenAI model field and hot-swaps backends. We chose the latter and have not regretted it.

2. The /api/chat endpoint

Half our internal tools were hitting Ollama's native /api/chat with its idiosyncratic streaming format (newline-delimited JSON, done: true terminator). llama.cpp ships an OpenAI-compatible /v1/chat/completions and a legacy /completion endpoint, but nothing matches Ollama's wire format. We rewrote 14 call sites. Budget half a day per non-trivial integration.

3. Modelfiles

The FROM ... / SYSTEM ... / PARAMETER ... Modelfile DSL has no equivalent in llama.cpp. System prompts move into the request body or a Jinja template passed via --chat-template-file. Sampler parameters move to CLI flags or per-request JSON. This is mostly a documentation problem, but if you had ten custom Modelfiles, you now have ten YAML configs for llama-swap and ten template files.

4. OpenAI client defaults

The OpenAI Python SDK against Ollama silently tolerates a lot — unknown fields, missing max_tokens, half-formed tool schemas. llama-server is stricter. We hit three production bugs in week one where requests that "worked" against Ollama returned 400s from llama.cpp because the schemas were genuinely malformed. That is a feature, not a regression, but it will surface latent bugs.

5. Home Assistant and other downstream integrations

The official Home Assistant Ollama integration assumes the Ollama API. We swapped to the generic OpenAI conversation agent pointed at llama-swap and lost the model picker UI but gained model switching by conversation. Net positive after the first week.

Throughput: what actually changed

We re-ran our standard bench — 50 prompts from the LMSYS-Chat-1M subset, 256-token completions, single user — on identical GGUF files. The hardware is the editorial test bench: a single RTX 4090 (24 GB), Ryzen 9 7950X, 64 GB DDR5-6000, Linux 6.12, CUDA 12.6. Numbers are the median of three runs.

Model (Q4_K_M)	Ollama 0.6.x tok/s	llama.cpp b4800 tok/s	Delta	VRAM delta
Llama 3.3 70B (partial offload)	6.2	7.4	+19%	-1.1 GB
Qwen3-Coder 32B	38.1	51.0	+34%	-2.2 GB
GLM-4.6 Air 12B	74.5	88.0	+18%	-0.9 GB
Gemma 3 27B	29.8	36.4	+22%	-1.6 GB
Mistral Small 3.2 24B	44.0	52.7	+20%	-1.3 GB

Two caveats. The 70B number reflects a 28-layer offload; pure CPU and pure GPU runs converge within 4%. And the VRAM gap shrinks if you let Ollama keep its default flash-attention settings — the larger deltas come from running both backends with FA enabled and -ctk q8_0 -ctv q8_0 KV-cache quantization on llama.cpp, which Ollama does not expose. That last point matters: a chunk of the win is configurability, not raw engine speed.

Concurrency: where the gap widens

Single-user numbers undersell the migration. Under concurrent load, Ollama's queueing model serializes requests per model; llama-server's --parallel N flag genuinely batches them. We replicated a similar test to the one described in this April 2026 benchmark and the directional finding holds.

Concurrent users	Ollama aggregate tok/s	llama.cpp --parallel 4 tok/s	p95 latency Ollama	p95 latency llama.cpp
1	51	51	0.4 s	0.4 s
2	54	89	2.1 s	0.9 s
4	58	142	5.8 s	1.6 s
8	61	168	14.2 s	3.4 s

If you are a single developer chatting interactively, you will never see this. If you are running a small team behind a shared endpoint — which the editorial team does for internal eval pipelines — it is the entire reason to switch.

The migration playbook we actually used

This is the path that worked across the bench. It assumes you already have GGUF files, either by re-pulling from Hugging Face or by extracting from Ollama's blob store.

1. Locate or convert your weights. Ollama stores GGUFs in ~/.ollama/models/blobs/ under SHA-256 names. The manifest in ~/.ollama/models/manifests/ maps tag names to blobs. Symlink, don't copy.
2. Build llama.cpp with the right backend. CUDA: cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server. Vulkan and ROCm flags are documented in the official build docs.
3. Smoke-test one model. ./llama-server -m model.gguf -ngl 99 -c 32768 -fa --host 0.0.0.0 --port 8080. Hit http://localhost:8080/v1/models to confirm.
4. Install llama-swap. Single Go binary. Point its YAML config at every GGUF you care about. It handles cold-start, eviction, and OpenAI routing.
5. Rewrite Ollama-specific calls. Anything hitting /api/generate or /api/chat moves to /v1/chat/completions. Anything using Modelfile system prompts moves into the request messages array.
6. Tune per model. Set --ctx-size, --parallel, -ctk q8_0 -ctv q8_0, and the right --chat-template for each model in the llama-swap config. Reasoning models often need --reasoning-budget and an explicit template — the Vibescoder writeup covers the GLM/Qwen flags well.
7. Validate against your eval set. Run a regression pass. The whole point of leaving the wrapper is being able to see and tune what you're running — use that.

What we lost and don't miss

The Ollama UX was genuinely good. ollama pull, ollama list, ollama run — that triplet is hard to beat for first-time users. We lost it. We do not miss it, because we were never the audience for it.

What we do not miss: the hashed blob store, the opaque template resolution, the silent context-window truncation, the lag on new models, the inability to run two models at once on the same GPU with intelligent eviction. What we mildly miss: the one-line install. Building llama.cpp from source is not hard, but it is not curl | sh either, and the prebuilt binaries lag the source by days.

Cost implications

The throughput gains translate to real money once you start amortizing hardware. Using the same assumptions as our cost calculator — $0.14/kWh electricity, 450W sustained draw under inference, 3-year hardware amortization on a $2,300 build — the cost-per-million-tokens shifts as follows for Qwen3-Coder 32B at single-user load.

Backend	Tokens/sec	Energy per 1M tok	Hardware per 1M tok	Total $/1M tok
Ollama	38.1	$1.03	$1.55	$2.58
llama.cpp tuned	51.0	$0.77	$1.16	$1.93
llama.cpp + --parallel 4 (4 users)	142	$0.28	$0.42	$0.70

A 25% reduction at single-user, 73% reduction at four-user concurrent. For comparison, Claude Sonnet 4.6 input tokens are $3.00/M and output $15.00/M — local economics only beat hosted APIs once your volume is high or your privacy requirements are absolute. Our methodology page details the assumptions; raw bench data is available through the BestLLMfor public API (CC BY 4.0) and the open-source MCP server server for anyone replicating these tests.

Who should not switch

This migration is not free. Skip it if:

• You run exactly one model for exactly one user and the model came out more than 60 days ago.
• You need the Ollama ecosystem integrations (Open WebUI's native picker, Continue.dev's auto-discovery, Home Assistant's first-party agent) and the workarounds are blockers.
• You cannot accept any operational burden — llama.cpp asks you to understand context sizes, KV-cache quantization, and chat templates. Ollama hid those choices, sometimes badly, but it hid them.
• Your model lives entirely on CPU. The throughput gap narrows to single digits, and Ollama's UX premium is then worth more than the speed.

For everyone else — multi-model shops, day-one-model testers, anyone tracking VRAM budgets — the migration pays back inside a week.

Verdict

Use case	Recommendation	Why
Single developer, one model, casual use	Stay on Ollama	The UX premium is real and the speed delta is invisible.
Day-one model testing	Switch to llama.cpp	10–21 day lag on the Ollama registry is disqualifying.
Small team, shared endpoint	Switch to llama.cpp + llama-swap	Concurrency gains are 2–3× at 4+ users.
Tight VRAM budget (≤16 GB)	Switch to llama.cpp	1–2 GB of headroom recovered through KV-cache quantization.
Production inference behind an API	Consider vLLM instead	llama.cpp is excellent for ≤16 concurrent users; beyond that, vLLM wins.