Ollama in Docker, Done Right

◆ Local AI — Your private ChatGPT, free, on your own machine, in an hour · $24 · or all kits $49 →

Verdict (September 2026): Running Ollama in Docker is worth it for reproducible, version-pinned deployments and clean teardown — but only if you do two things: pass the GPU through with the NVIDIA Container Toolkit, and mount a persistent volume at /root/.ollama so your models survive restarts. Skip either and you get silent CPU inference or repeated multi-gigabyte downloads. For a single local dev box you touch every day, the bare-metal install is simpler and gives you the same speed with less to wire up.

Why put Ollama in a container at all

The native Ollama install is a one-liner and it works. So the honest question is what a container buys you. The answer is isolation and reproducibility: a pinned image tag, a config that lives in version control, and a teardown that leaves no daemon or systemd unit behind. That matters when you run Ollama alongside a web UI, an API gateway, or a RAG stack that you want to spin up and tear down as a unit.

It buys you nothing on raw inference speed — a correctly configured container runs at the same tokens/sec as bare metal because the model still executes on your GPU. What it costs you is two configuration steps that the native installer handles for you: getting the GPU into the container, and keeping your downloaded weights outside the container's ephemeral filesystem. Both are covered below. If you are still deciding whether Ollama is the right runtime at all, our what is Ollama guide and the Ollama vs llama.cpp comparison lay out the trade-offs.

GPU passthrough: the part everyone gets wrong

Docker does not see your NVIDIA GPU by default. You need the NVIDIA Container Toolkit installed on the host, which registers a runtime that exposes the GPU and driver libraries to containers. Install it, restart the Docker daemon, then confirm the plumbing works before you touch Ollama:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that prints your GPU, you are done with the hard part. If it errors, fix it here — do not move on hoping Ollama will behave differently. When the toolkit is missing, Ollama starts anyway and quietly falls back to CPU, so you get a working endpoint that is 10–40× slower with no obvious error. The tell is inference that pegs your CPU cores while nvidia-smi shows the GPU idle.

For Ollama specifically, pass --gpus all on docker run, or use the deploy.resources.reservations.devices block in Compose (shown below). On Windows, this path runs through WSL2 with the NVIDIA driver installed on the Windows host, not inside WSL. AMD users need the ROCm image tag instead; check the official Ollama Docker docs for the current tag, since it changes.

Persisting models so you don't re-download 20 GB

Ollama stores pulled models under /root/.ollama inside the container. That directory is part of the container's writable layer, which is destroyed on docker rm or docker compose down. Mount something persistent there or you will re-pull every model each time you recreate the container — a real cost when a single 14B model at Q4 is several gigabytes.

You have three approaches. Pick based on how you work, not habit:

For a workstation, a named volume is the right default. Put your fast NVMe behind it if you can — model load time is I/O bound. To size the volume, add up the models you plan to keep: our VRAM calculator estimates the memory footprint, and as a rule of thumb on-disk size tracks it closely — roughly 0.58 GB per billion parameters at Q4_K_M, 1.07 at Q8, and about 2 GB per billion at FP16. Browse tagged options in the best Ollama models roundup.

A compose file that actually works

Here is the minimum that gets you GPU-accelerated, model-persistent Ollama. Create a docker-compose.yml with a single ollama service and these fields:

• image: ollama/ollama — pin a specific tag in production rather than the floating latest.
• ports: ["11434:11434"] — the default API port. Bind to 127.0.0.1:11434:11434 if you don't want it exposed on your LAN.
• volumes: ["ollama:/root/.ollama"] — the persistence line from above.
• A deploy.resources.reservations.devices entry requesting driver: nvidia, count: all, capabilities: [gpu] — this is the Compose equivalent of --gpus all.
• A top-level volumes: block declaring the named ollama volume.

Bring it up with docker compose up -d, then pull a model into the running container: docker compose exec ollama ollama pull <model>. The API answers at http://localhost:11434, and the OpenAI-compatible route lives under /v1 for existing client code. Because the tag names change over time, get the exact model string from the official library rather than copying one from a blog. Verify the GPU is in use with docker compose exec ollama nvidia-smi while a request is running — if the GPU is busy, you're set. Keep the Docker Compose reference docs handy for the current schema, which occasionally shifts between major versions.

When NOT to containerize Ollama

Docker is not free complexity, and for a lot of local setups the native install wins. Skip the container when:

• It's your daily-driver dev box. The bare-metal install auto-detects the GPU, runs as a background service, and updates in place. You gain nothing but two layers of indirection by wrapping it.
• You're on macOS. Docker Desktop on Mac cannot pass through the Apple Silicon GPU, so a containerized Ollama runs on CPU — far slower than the native Metal build. Always run natively on a Mac.
• You have a single tight VRAM budget. The container itself uses negligible VRAM, but the extra abstraction makes it harder to see and tune memory pressure. On a constrained card, run close to the metal and consult our 12GB VRAM benchmarks for what actually fits.

Containerize when you're deploying to a server, running Ollama as one service in a larger Compose stack, need reproducible version pinning, or want clean multi-tenant teardown. That is the line: orchestration and reproducibility justify Docker; a single interactive workstation usually doesn't. Everything in this article was validated against a native and containerized setup on an RTX 5070 Ti — see our methodology for how we test.

Frequently asked questions

By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.