Guide · 2026-07-15

How to Run vLLM in Docker with NVIDIA CUDA — 10-Minute Setup

Last updated 2026-07-15

From a clean machine to a live, OpenAI-compatible inference server in under 10 minutes — GPU passthrough, model sizing, and tuning included.

By Mohamed Meguedmi · 8 min read

Running vLLM inside Docker is the fastest, most reproducible way to stand up a high-throughput LLM server. The container bundles the exact CUDA runtime, PyTorch build, and Python dependencies vLLM needs, so you sidestep the dependency conflicts that plague bare-metal installs. vLLM's throughput advantage comes from PagedAttention, which manages the KV cache like virtual memory — the reason a single consumer GPU can serve thousands of tokens per second. This guide takes you from a clean machine to a live endpoint, and every VRAM and throughput figure below can be cross-checked against the free BestLLMfor cost calculator.

Prerequisites — What You Need Before You Start

You do not need the CUDA Toolkit installed on the host. The Docker image ships its own CUDA 12.x runtime; the host only needs a compatible NVIDIA driver and the container toolkit for GPU passthrough.

Component	Minimum	Recommended	Notes
GPU	NVIDIA Ampere (RTX 30xx)	Ada / Hopper (RTX 40xx, L40S, H100)	Compute capability 7.0+; FP8 needs 8.9 (Ada) or 9.0 (Hopper)
NVIDIA driver	525.60.13	555 or newer	Must satisfy the image's CUDA 12.x runtime
Docker Engine	19.03	24.0+	19.03 introduced the --gpus flag
NVIDIA Container Toolkit	Latest	Latest	Provides GPU passthrough into containers
Host RAM	16 GB	32 GB+	For download staging and KV overflow
Free disk	30 GB	100 GB+	A 7B model is ≈ 15 GB in FP16, ≈ 8 GB in FP8

Verify your driver with nvidia-smi on the host. The driver version must be equal to or newer than the CUDA minor version baked into the image — current vLLM images track CUDA 12.x, which needs driver 525.60.13 or later.

Step-by-Step: vLLM in Docker with CUDA

Four commands stand between you and a running server. On a machine that already has Docker and the driver, the whole thing takes about 10 minutes minus model download time.

1. Install the NVIDIA Container Toolkit (Ubuntu/Debian shown; follow NVIDIA's install guide for other distros):

   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
   sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

2. Confirm GPU passthrough works before touching vLLM. This should print your GPU table:

   docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

   If this fails, fix it now — vLLM cannot succeed until this does.
3. Pull and launch vLLM. This single command pulls the image, exposes the OpenAI-compatible API on port 8000, and loads a model from HuggingFace:

   docker run --runtime nvidia --gpus all \
     -v ~/.cache/huggingface:/root/.cache/huggingface \
     --env "HUGGING_FACE_HUB_TOKEN=YOUR_HF_TOKEN" \
     -p 8000:8000 --ipc=host \
     vllm/vllm-openai:v0.10.2 \
     --model Qwen/Qwen2.5-7B-Instruct --quantization fp8 --max-model-len 8192

   The -v mount persists downloaded weights on the host so restarts are near-instant. The token is only required for gated models.
4. Test the endpoint once the log prints Application startup complete:

   curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Explain PagedAttention in one sentence."}]}'

Because the API mirrors OpenAI's, any existing OpenAI client works by pointing base_url at http://localhost:8000/v1. Model details and cards, such as Qwen2.5-7B-Instruct, live on HuggingFace.

Choosing the Right Model for Your VRAM

VRAM is the single hard constraint. Weights are only half the story — the KV cache grows with context length and concurrency. Budget roughly 20% headroom above the weight size for a 4K-token context, more if you serve many concurrent requests.

Model	Quant	Weights VRAM	+KV cache (4K ctx)	Min GPU
Qwen2.5-7B-Instruct	FP16	~15 GB	~18 GB	RTX 4090 24 GB
Qwen2.5-7B-Instruct	FP8	~8 GB	~11 GB	RTX 4070 Ti 16 GB
Qwen2.5-14B-Instruct	FP8	~15 GB	~19 GB	RTX 4090 24 GB
Qwen2.5-32B-Instruct	FP8	~33 GB	~40 GB	2× 24 GB (TP=2) or L40S 48 GB
Llama-3.3-70B-Instruct	FP8	~70 GB	~80 GB	2× H100 80 GB

Our verdict: for a single 24 GB card, a 14B model in FP8 is the sweet spot — near-7B latency with a meaningful quality bump. Push to 32B only when you have 48 GB or a second GPU. Browse the full model catalog to compare quality tiers before committing VRAM.

Tuning vLLM Flags for Throughput and Memory

The defaults are sane, but four flags do most of the work when you need more concurrency or you are fighting an out-of-memory error:

• --gpu-memory-utilization (default 0.90) — the fraction of VRAM vLLM claims for weights plus KV cache. Drop to 0.80 if the GPU also drives a display.
• --max-model-len — caps context length. Shorter context means a smaller KV cache and more concurrent sequences.
• --tensor-parallel-size N — shards the model across N GPUs. Set it equal to your GPU count for models that overflow one card.
• --quantization fp8 — roughly halves weight memory versus FP16 with negligible quality loss on Ada/Hopper hardware.
• --max-num-seqs — the ceiling on concurrent sequences; raise it for batch workloads, lower it to protect tail latency.

Benchmarks — What to Expect

Numbers below are approximate, measured with short prompts and 256-token generations. Batched throughput assumes a saturated request queue; single-stream is one request at a time. Full methodology and reproducible runs live on the benchmarks hub.

GPU	Model	Config	Batched throughput	Single-stream
RTX 4090	Qwen2.5-7B	FP8	~2,900 tok/s	~95 tok/s
RTX 4090	Qwen2.5-14B	FP8	~1,500 tok/s	~55 tok/s
2× RTX 4090	Qwen2.5-32B	FP8, TP=2	~1,100 tok/s	~40 tok/s
H100 80 GB	Llama-3.3-70B	FP8	~2,400 tok/s	~48 tok/s

The takeaway: batching is where vLLM shines. A 4090 serving 7B FP8 delivers ~30× the throughput of a single stream, which is why one consumer card can back a small team's internal API. Want to script comparisons? Every figure on this site is queryable through the free BestLLMfor public API (CC BY 4.0) and our open-source MCP server, so you can pull VRAM and throughput data straight into your own tooling.

Troubleshooting Common CUDA and Docker Errors

Ninety percent of failures fall into five buckets. Match the error text to the fix:

Error	Cause	Fix
could not select device driver "" with capabilities: [[gpu]]	Container Toolkit not configured	Run nvidia-ctk runtime configure and restart Docker
Failed to infer device type	Host driver too old for the image CUDA	Upgrade the host NVIDIA driver
Bus error (core dumped)	Insufficient shared memory	Add --ipc=host or --shm-size=8g
CUDA out of memory	Weights + KV cache exceed VRAM	Lower --gpu-memory-utilization or --max-model-len, or switch to FP8
401 Client Error on download	Gated model, missing token	Pass HUGGING_FACE_HUB_TOKEN via --env

The Verdict

vLLM in Docker is the right default for anyone running local inference in 2026: reproducible, GPU-native, and OpenAI-compatible out of the box. Pin your tag, mount the HuggingFace cache, and match the model to your VRAM. Here is how we would set it up by scenario:

Scenario	Recommended setup	Why
Solo dev, 24 GB card	vllm/vllm-openai:v0.10.2, 7B–14B FP8	Best balance of speed and simplicity
Small team API	Above + Docker Compose, pinned tag, HF cache volume	Reproducible and fast to restart
70B-class model	Multi-GPU tensor parallelism or a single H100 80 GB	VRAM is the hard limit
Production	Pinned tag + health checks + Prometheus metrics	Stability over the bleeding edge

For deeper dives into deployment patterns and model selection, browse the rest of our guides.