Guide · 2026-07-24

How to Use LiteLLM Router with Ollama — OpenAI-Drop-In

Last updated 2026-07-24

Put a LiteLLM Router in front of Ollama and every OpenAI SDK call — Python, Node, curl — just works against your local models. Here is the exact config.

By Mohamed Meguedmi · 9 min read

Why put LiteLLM Router in front of Ollama?

Ollama already ships an OpenAI-compatible endpoint at /v1, so a fair question is: why add a second process? The answer is everything Ollama's built-in shim deliberately leaves out. Ollama serves one node, one model resolution scheme, no key management, no fallbacks, and no unified logging. The moment you run more than one machine, need per-team API keys, or want a call to silently fail over from a 32B model to an 8B model when the GPU is busy, you need a router.

LiteLLM is that router. It presents a single OpenAI-shaped surface — /v1/chat/completions, /v1/embeddings, /v1/models — and behind it maps friendly aliases to any number of Ollama deployments. Your application keeps calling gpt-4o or my-coder; the proxy decides which physical model on which host actually answers. That indirection is the whole point of a drop-in: the code that talks to the model never changes, even as the model behind it does.

For teams comparing self-hosted options against hosted APIs, this also unlocks apples-to-apples cost math. Route the same alias to a local model or a cloud model and measure the difference — our cost calculator uses exactly this pattern.

Prerequisites and hardware

You need three things: a running Ollama daemon, Python 3.9+, and enough VRAM for the models you intend to alias. LiteLLM itself is CPU-light — it is glue, not inference — so the hardware budget below is entirely about Ollama.

Target model	Quant	VRAM (min)	Suggested GPU	~Tokens/s
Llama 3.1 8B Instruct	Q4_K_M	6 GB	RTX 3060 12GB	55–75
Qwen3-Coder 32B	Q4_K_M	20 GB	RTX 4090 24GB	28–38
Gemma 2 27B	Q4_K_M	18 GB	RTX 4090 24GB	25–33
Mistral Small 3.1 24B	Q5_K_M	18 GB	RTX 4090 24GB	27–35
Nomic-embed-text (embeddings)	F16	1 GB	Any 8GB card	n/a

Figures assume a warm model (already loaded), 4K context, single concurrent request, on Ollama 0.6.x. Cold-start load time for a 32B Q4 model is typically 4–9 seconds off an NVMe SSD — the Router's timeout settings, covered below, exist precisely to tolerate that. Browse the full lineup on the model catalog before committing VRAM.

Install LiteLLM and start the proxy

The following steps get a working OpenAI drop-in in under five minutes on a machine that already runs Ollama.

1. Confirm Ollama is live. Run ollama list and pull at least one model, e.g. ollama pull llama3.1:8b. Verify the API responds: curl http://localhost:11434/api/tags.
2. Install the proxy. pip install "litellm[proxy]" pulls LiteLLM plus the server extras (uvicorn, config parsing). A virtualenv is recommended to avoid dependency clashes.
3. Write a config.yaml. This is the only file that matters — it maps public aliases to Ollama deployments (full example in the next section).
4. Launch it. litellm --config config.yaml --port 4000. The proxy boots on http://0.0.0.0:4000 and prints the registered model list.
5. Smoke-test the OpenAI surface. Fire a standard OpenAI request at the proxy and confirm you get a completion back from your local model:

   curl http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer sk-anything" \
     -d '{"model": "my-coder", "messages": [{"role":"user","content":"ping"}]}'

The Authorization header can be any string in local mode unless you set a master key. In production you should set one — see the hardening section.

The config.yaml — OpenAI drop-in mapping

Everything hinges on this file. Each entry under model_list pairs a model_name (the alias your app sends) with litellm_params (how to reach the real model). The critical detail: prefix Ollama models with ollama_chat/ so LiteLLM applies the model's chat template rather than raw text completion. See the official LiteLLM Ollama provider docs for every supported parameter.

model_list:
  # Alias -> physical Ollama model
  - model_name: my-coder
    litellm_params:
      model: ollama_chat/qwen3-coder:32b
      api_base: http://localhost:11434

  - model_name: fast
    litellm_params:
      model: ollama_chat/llama3.1:8b
      api_base: http://localhost:11434

  # Map the OpenAI name so legacy code "just works"
  - model_name: gpt-4o
    litellm_params:
      model: ollama_chat/qwen3-coder:32b
      api_base: http://localhost:11434

  - model_name: text-embedding-3-small
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://localhost:11434

litellm_settings:
  drop_params: true   # silently ignore OpenAI-only params Ollama can't use
  request_timeout: 600

Note the gpt-4o alias: point it at a local model and any tool hardcoded to that name — an IDE plugin, a CLI, a CI script — now runs offline with zero code changes. That is the literal meaning of "drop-in." The drop_params: true flag is what keeps picky OpenAI clients from erroring on parameters (like logit_bias) that a local model does not implement.

Using it from Python is unremarkable, which is the goal:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-anything")
resp = client.chat.completions.create(
    model="my-coder",
    messages=[{"role": "user", "content": "Refactor this function..."}],
)
print(resp.choices[0].message.content)

Load balancing, fallbacks, and retries

This is where the Router earns its name and pulls ahead of Ollama's bare endpoint. Give two entries the same model_name but different hosts, and LiteLLM round-robins between them. Add a fallbacks rule and a failed or overloaded call automatically retries a cheaper model. Full options live in the LiteLLM routing reference.

model_list:
  - model_name: my-coder
    litellm_params:
      model: ollama_chat/qwen3-coder:32b
      api_base: http://gpu-node-a:11434
  - model_name: my-coder            # same alias, second GPU host
    litellm_params:
      model: ollama_chat/qwen3-coder:32b
      api_base: http://gpu-node-b:11434

router_settings:
  routing_strategy: least-busy      # or simple-shuffle, latency-based
  num_retries: 2
  timeout: 600
  fallbacks:
    - my-coder: ["fast"]            # if 32B fails, answer with 8B

The strategies matter for local fleets. least-busy tracks in-flight requests and steers traffic away from a node already mid-generation — ideal when two GPUs share load. latency-based-routing favors whichever host has been answering fastest, which naturally avoids a node that is cold-loading a model. For a single machine, keep simple-shuffle and lean on fallbacks alone.

Benchmarks — how much does the proxy cost you?

The recurring worry with any proxy is added latency. We measured the LiteLLM hop against direct Ollama calls on the same host, 100 requests each, warm model, 512-token output. The proxy overhead is the added wall-clock time attributable to LiteLLM's routing and translation layer — not inference.

Path	Model	Median TTFT	Proxy overhead	Tokens/s (end-to-end)
Direct Ollama /v1	Llama 3.1 8B Q4	190 ms	—	68
LiteLLM Router	Llama 3.1 8B Q4	195 ms	+5 ms	67
Direct Ollama /v1	Qwen3-Coder 32B Q4	410 ms	—	32
LiteLLM Router	Qwen3-Coder 32B Q4	414 ms	+4 ms	32

The verdict: overhead sits in the 3–6 ms band and never moved tokens/s by more than rounding. Against a 32B model whose first token takes 400+ ms and whose full answer runs several seconds, the Router is free in practice. You are paying single-digit milliseconds for load balancing, fallbacks, key auth, and unified logging — an easy trade. For the raw model numbers, see our benchmarks hub.

Hardening for production

Local-mode defaults are fine on a laptop; a shared server needs three additions. First, set a master key and virtual keys so each team or app gets a revocable credential:

general_settings:
  master_key: sk-1234-change-me

# then mint per-user keys via the /key/generate admin route

Second, enable caching for repeated prompts — LiteLLM supports an in-memory or Redis cache that returns identical completions instantly, which matters for embedding workloads that re-embed the same documents. Third, put the proxy behind a reverse proxy (Caddy or Nginx) for TLS if anything outside localhost talks to it; never expose port 4000 to the open internet unauthenticated.

Ollama itself should be pinned to a known-good build — check the release notes on the official Ollama repository, since chat-template fixes land regularly and directly affect output quality through the ollama_chat/ path.

One more resource worth knowing: BestLLMfor publishes a free public API (CC BY 4.0) and an open-source MCP server that expose our model metadata and benchmark data programmatically — handy if you want your Router config to pull suggested VRAM or quant defaults automatically rather than hardcoding them. Details are on the about page.

Verdict

If you run local models anywhere beyond a single laptop, a LiteLLM Router in front of Ollama is close to mandatory. It costs single-digit milliseconds, requires one YAML file, and converts a bare inference daemon into a production-grade, OpenAI-compatible gateway with auth, load balancing, and fallbacks. The gpt-4o alias trick alone — running unmodified OpenAI-targeted tooling against a local Qwen3-Coder 32B — pays for the setup time immediately.

Scenario	Ollama /v1 only	LiteLLM Router + Ollama	Verdict
Single node, one model, personal use	Sufficient	Overkill but harmless	Either — Ollama alone is fine
Multiple GPUs / hosts	No load balancing	Round-robin + least-busy	Router wins
Team access with keys	None	Virtual keys + master key	Router wins
Reliability under load	Hard fail	Retries + fallbacks	Router wins
Drop-in for OpenAI code	Partial	Full, with aliasing	Router wins

For anything you'd call production, the recommendation is unambiguous: run the Router. Explore matching models and quant levels in our guides hub before you finalize the config.