Guide · 2026-05-16

Use LiteLLM Router with Your Local LLM — Drop-In OpenAI Replacement

LiteLLM turns any local Ollama, vLLM, or llama.cpp endpoint into a drop-in OpenAI replacement, with load balancing, fallbacks, and per-key budgets — no client code rewrite.

By Mohamed Meguedmi · 9 min read

Why LiteLLM Beats Hand-Rolling an OpenAI Shim

Local model servers each speak slightly different OpenAI dialects. Ollama exposes /api/chat plus a partial /v1 shim that drops tools for several models. vLLM exposes a fuller /v1 but renames some sampling parameters. llama.cpp's llama-server implements its own subset and historically lagged on logprobs. The day the team swaps Qwen3-Coder 32B Q4_K_M for GLM-4.6 Air, or splits traffic across two GPU nodes running different quants, client code breaks if it addresses the model server directly.

LiteLLM solves this with two interchangeable surfaces sharing one routing engine:

• LiteLLM SDK Router — a Python class that wraps litellm.completion() and dispatches across a model_list.
• LiteLLM Proxy — a FastAPI gateway exposing OpenAI-compatible REST on port 4000, configured via the same config.yaml.

The competitive landscape is thin. OpenRouter is a hosted SaaS and defeats the point of local inference. Inworld's Realtime Router is a vendor pitch with no self-hosted story. Cloudflare's AI Gateway is cloud-only. Among open-source projects, LiteLLM has the only credible feature parity with the OpenAI client across streaming, function calling, vision, embeddings, audio, and structured outputs, and is the only one whose -stable Docker tag ships after a documented 12-hour load test. Verdict: there is no second place.

SDK Router vs. Proxy Server — Which One Do You Need?

Both modes consume identical configuration. The choice is purely operational.

Criterion	SDK Router	Proxy Server
Languages supported	Python only	Any HTTP client (Python, JS, Go, Rust)
Per-team API keys	No	Yes (virtual keys + budgets)
Spend tracking persistence	In-process only	SQLite (dev) or PostgreSQL (prod)
Process model	Inside your app	Standalone container, port 4000
Hot-reload of model_list	No (re-import)	Yes via Admin UI or DB
Memory footprint	~120 MB on top of app	~280 MB idle
Auth boundary	Trust = process	Bearer, JWT, OIDC

Rule of thumb from the BestLLMfor.com editorial team: if exactly one Python service consumes the model, use the SDK Router. The moment a second consumer appears — a TypeScript frontend, a Go cron job, a colleague's notebook — switch to the Proxy. The migration is one base_url change in client code.

Install and Configure in Under Five Minutes

The fastest path assumes Ollama already serves a model locally on port 11434 and Docker is installed.

Step 1 — Write config.yaml

model_list:
  - model_name: qwen3-coder
    litellm_params:
      model: ollama_chat/qwen3-coder:32b-q4_K_M
      api_base: http://host.docker.internal:11434
  - model_name: qwen3-coder
    litellm_params:
      model: ollama_chat/qwen3-coder:7b-q4_K_M
      api_base: http://host.docker.internal:11434
      rpm: 60

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 120
  fallbacks:
    - {"qwen3-coder": ["qwen3-coder-7b-fallback"]}

litellm_settings:
  drop_params: true
  set_verbose: false

general_settings:
  master_key: sk-local-master-change-me
  database_url: "sqlite:///./litellm.db"

Step 2 — Run the proxy

docker run -d --name litellm \
  -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml --port 4000 --num_workers 4

Step 3 — Point any OpenAI client at it

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-local-master-change-me",
)

r = client.chat.completions.create(
    model="qwen3-coder",
    messages=[{"role": "user", "content": "Refactor this function."}],
)
print(r.choices[0].message.content)

That is the entire migration from api.openai.com to a local model. No code change beyond base_url and api_key.

Routing Strategies That Actually Matter

LiteLLM ships six routing strategies. Most are noise for self-hosted scenarios. The relevant ones:

Strategy	When to use	Watch out for
simple-shuffle	Two or more identical GPU nodes serving the same model	Ignores in-flight load — use only when nodes are symmetric
least-busy	Heterogeneous GPUs (e.g. RTX 4090 + RTX 3090 mix)	Requires Redis if proxy runs multiple workers
usage-based-routing-v2	Mixing cloud and local with hard token caps	Needs Redis; ~1 ms extra latency
latency-based-routing	Servers with unpredictable cold starts (vLLM paged-attention churn)	First 50 requests are blind — pre-warm

For most self-hosted clusters of 2–4 nodes, least-busy backed by Redis is the right default. For a single machine running several quants of the same model on one GPU, simple-shuffle with weights via rpm/tpm is sufficient. See the official routing reference for the full matrix.

Fallbacks, Retries, and Cost Tracking Without Surprises

Fallbacks are LiteLLM's most underused feature. The pattern that matters: never fall back to a paid provider implicitly. A chain like qwen3-coder-32b → qwen3-coder-7b → llama3.2-3b degrades gracefully on OOM or timeout without leaking a single token off-box.

If cloud egress is acceptable, gate it behind a separately named alias (e.g. qwen3-coder-cloud-emergency) so a developer has to opt in explicitly. Budgets enforce the rest: each virtual key can be capped at e.g. $5/day with max_budget, and the proxy returns HTTP 429 once the cap is reached.

Spend tracking on local models is non-trivial because pricing isn't published. LiteLLM supports custom input_cost_per_token and output_cost_per_token entries — set these to an amortized GPU power cost (kWh × tokens/sec) and the spend dashboard becomes meaningful. The BestLLMfor.com cost calculator exports per-model kWh/Mtok figures that drop straight into model_info, and the same numbers are available through the BestLLMfor public API (CC BY 4.0) at api.bestllmfor.com/v1/models.

Performance — Latency Overhead and Throughput

Editorial benchmark on a 13th-gen Core i9 + RTX 4090 reference machine, Ollama serving Qwen3-Coder 7B Q4_K_M, single-stream non-streaming completion of 128 input / 256 output tokens, median of 100 runs:

Path	TTFT (ms)	Total (ms)	Tokens/s
OpenAI SDK → Ollama /v1 direct	184	3,420	74.9
OpenAI SDK → LiteLLM Proxy → Ollama (same host)	188	3,427	74.7
OpenAI SDK → LiteLLM Proxy → Ollama (LAN, 1 GbE)	192	3,448	74.3
Python Router in-process	185	3,419	74.9

Overhead from the proxy is 3–5 ms on the same host, 8–10 ms across a 1 GbE LAN — irrelevant against multi-second generations. Streaming first-token overhead is the same. The Router class has no measurable overhead because it shares the process.

Throughput under concurrency tells the more interesting story: with 16 concurrent clients hitting the proxy, total tokens/s is identical to hitting Ollama directly (limited by GPU, not the gateway). With num_workers=4, the proxy itself sustained 1,200 RPS of metadata-only requests in a sanity test before queueing — well beyond what any local inference setup will produce.

Production Pitfalls Worth Knowing

• Forgetting drop_params: true — Ollama rejects unknown parameters and the OpenAI SDK happily sends logit_bias. The flag silently strips unsupported fields per backend.
• Using :main-latest in production — nightly tag. Streaming broke three times in the last year. Pin :main-stable and bump on a schedule.
• SQLite under load — fine for <5 RPS spend logging; switch to PostgreSQL past that or the writer thread becomes the bottleneck.
• Master-key reuse — generate virtual keys per service. Rotating one then doesn't take down everything.
• Missing health checks — enable background_health_checks: true so a dead Ollama node is removed from rotation within 60 s instead of failing each request.

The full editorial test protocol used for these numbers is documented on the BestLLMfor.com methodology page. French-language coverage of the same patterns is available on quelllm.fr, and the open-source quelllm-mcp server exposes a similar router behind an MCP interface for Claude Desktop users. Background on the underlying SDK lives in the official LiteLLM docs.

Verdict

Scenario	Recommendation
Single Python app, one developer, one model	SDK Router, no proxy
Mixed-language clients or shared team access	Proxy on Docker :main-stable, SQLite
Multiple GPU nodes, >10 RPS, multi-tenant	Proxy + PostgreSQL + Redis + least-busy
Need cloud fallback for overflow	Separate model alias only — never silent fallback
Replacing OpenAI in an existing codebase	Change base_url and api_key. Done.

LiteLLM is the default answer for putting a local LLM behind an OpenAI-compatible endpoint. Router and Proxy modes cover the full range from a one-file Python script to a multi-node inference cluster with audit logs. For anyone running Ollama, vLLM, or llama.cpp in 2026 and wanting to keep client code provider-agnostic, this is the unambiguous pick.