What Is vLLM? The Open-Source LLM Serving Engine, Explained

vLLM in three sentences

vLLM is a Python library and server that loads a model from Hugging Face onto one or more GPUs and exposes it through an OpenAI-compatible HTTP API. It came out of the Sky Computing Lab at UC Berkeley alongside the paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al., SOSP 2023). As of September 20, 2026, it is the default choice when a team or an application, rather than one person, needs to share a self-hosted model.

The problem it solves

While a model generates, it keeps a running record of everything it has read and written so far: the KV cache. That cache grows with every token, and nobody knows in advance whether a reply will be twenty tokens or two thousand. Early serving systems handled the uncertainty by reserving one contiguous slab of GPU memory per request, sized for the worst case.

The vLLM paper measured the consequence: in existing systems, only 20 to 40 percent of KV-cache memory held actual token state. The rest was lost to over-reservation and fragmentation. Wasted memory means fewer requests in flight, which means an expensive GPU idling at a fraction of what it could serve.

PagedAttention: virtual memory for the KV cache

The fix borrows a fifty-year-old idea from operating systems. Instead of one big contiguous block per request, vLLM splits the KV cache into small fixed-size pages, allocates them on demand, and places them anywhere in GPU memory. A block table maps the logical order of tokens to the physical location of pages, exactly the way an OS maps a process's address space onto scattered RAM.

	Contiguous allocation	PagedAttention
Memory reserved per request	Maximum possible length, up front	One page at a time, as tokens arrive
Waste reported in the paper	60–80%	Under 4%
Shared prefixes (same system prompt)	Duplicated for every request	Stored once, pages shared
Effect	Few concurrent requests	Several times more requests on the same GPU

Continuous batching: no request waits for another

A naive server groups requests into a batch and waits for every one of them to finish before starting the next batch. A short answer sits idle behind a long one, and the GPU does padding work. vLLM rebuilds the batch at every decoding step. The moment one sequence finishes, its slot goes to a waiting request. Combined with paged memory, this keeps the GPU saturated with useful work, which is where the large throughput gains over single-user runtimes come from.

The same engine adds the features a production endpoint needs: tensor parallelism across GPUs (--tensor-parallel-size), automatic prefix caching, speculative decoding, structured JSON output, multi-LoRA serving, and quantized formats designed for batched GPU math (AWQ, GPTQ, FP8). GGUF support exists but is experimental.

What vLLM puts in your VRAM

This is the most common surprise for people arriving from Ollama. Ollama pulls a 4-bit GGUF by default. vLLM loads the weights as published, usually BF16, at about 2 GB per billion parameters. Same model, triple the footprint, before any KV cache.

Model	BF16 (vLLM default)	4-bit GGUF (Ollama default)	Smallest setup for BF16
Qwen 3 8B	16 GB	5 GB	One 24 GB card
Gemma 4 12B	24 GB	7 GB	RTX 5090 (32 GB)
Qwen 3 14B	28 GB	9 GB	RTX 5090, short context
Mistral Small 3.2 24B	48 GB	14 GB	2 × RTX 5090, or one 80 GB card
Qwen 3.8 27B	54 GB	16 GB	One 80 GB card
Llama 3.3 70B	140 GB	40 GB	2 × 80 GB cards

Weights only, from the BestLLMfor model catalog, September 20, 2026.

The way out is to serve a GPU-native quantized build. A 4-bit AWQ or GPTQ model occupies roughly what the GGUF does, and FP8 halves BF16 on recent cards. Most popular models have such builds on Hugging Face.

"vLLM is using all my VRAM." By design. At startup vLLM claims 90% of GPU memory (--gpu-memory-utilization, default 0.9): weights first, everything left over becomes KV-cache pages. More pages means more concurrent requests. On a machine that does other things, lower the value.

Where it runs

Platform	Status	In practice
Linux + NVIDIA (CUDA)	Primary target	Best-tested path. Compute capability 7.0 or higher: Volta, Turing (RTX 20) and newer.
Linux + AMD (ROCm)	Supported	Instinct and recent Radeon cards; use the dedicated Docker image.
Windows	No native build	Run it under WSL2 with an NVIDIA GPU.
Apple Silicon	CPU only, experimental	No Metal acceleration in the main project. On a Mac, use MLX, Ollama or llama.cpp.
CPU only	Supported	Fine for testing an integration, not for serving.

Quick start

python -m venv vllm-env && source vllm-env/bin/activate
pip install vllm

vllm serve Qwen/Qwen3-8B --max-model-len 8192

The first launch downloads the model, then serves on port 8000. Any OpenAI client works by changing the base URL:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'

Set --max-model-len from day one. Without it, vLLM sizes the cache for the model's full advertised context, sometimes 128K tokens or more, and refuses to start if memory cannot cover it. For a containerized setup, follow vLLM in Docker with CUDA. Full documentation is at docs.vllm.ai and the source at github.com/vllm-project/vllm.

Should you use vLLM?

Your situation	vLLM?	Why
One person chatting on their own PC	No	No speed gain for a single stream, three times the VRAM at BF16.
You are on a Mac	No	No GPU path. MLX and llama.cpp use unified memory properly.
8–12 GB of VRAM	Rarely	A 4-bit GGUF with partial CPU offload does more for you.
A team of 5 to 50 sharing one model	Yes	Continuous batching serves everyone from one GPU.
An app that calls the model in bursts	Yes	Queueing, high throughput, standard API.
Batch-processing thousands of documents	Yes	The scenario where the throughput gap is widest.

If you are deciding between the two most common options, read vLLM vs Ollama for the short version and Ollama vs vLLM for production self-hosting for costs and migration. For the wider engine landscape, see ExLlama vs vLLM vs llama.cpp. Sizing figures used here are available through the BestLLMfor public API (CC BY 4.0) and our open-source MCP server.