Guide · 2026-07-27

How to Self-Host TabbyAPI as an OpenAI-Compatible Endpoint

Last updated 2026-07-27

A no-fluff guide to running TabbyAPI on your own GPU—install, configure, load EXL2/EXL3 models, and expose a drop-in /v1/chat/completions endpoint.

By Mohamed Meguedmi · 9 min read

What TabbyAPI Actually Is (And What It Isn't)

Search for "TabbyAPI" and the results are a mess of two unrelated projects. The one this guide covers is TabbyAPI by theroyallab—a lightweight Python server that wraps the ExLlamaV2 (and newer ExLlamaV3) inference engine and exposes it as an OpenAI-compatible HTTP API. The other, Tabby by TabbyML, is a self-hosted GitHub Copilot alternative. They share a name and nothing else. If you want raw, high-throughput inference of EXL2/EXL3 quantized weights behind a familiar endpoint, you want theroyallab's TabbyAPI.

TabbyAPI's pitch is narrow and it delivers on it: expose /v1/chat/completions, /v1/completions, and /v1/embeddings, back them with the fastest INT4 kernel available on consumer NVIDIA hardware, and add administrative endpoints for dynamic model loading, unloading, and on-the-fly LoRA switching. It does not do CPU offload, it does not run GGUF, and it does not pretend to be a general-purpose serving stack like vLLM. That focus is the point.

Verdict up front: if you own a single 24GB NVIDIA GPU and want maximum tokens/sec at 4-bit, TabbyAPI beats llama.cpp and is simpler to operate than vLLM. If you need multi-node scaling, batching for dozens of concurrent users, or AMD support, look elsewhere.

Hardware Requirements

The single hard rule: the quantized weights plus the KV cache must fit in VRAM. There is no CPU spillover. ExLlamaV2 requires an NVIDIA GPU of compute capability 8.0+ (Ampere), meaning RTX 30-series and newer, plus the A-series and later datacenter cards. Turing (RTX 20-series) works in some configs but loses the fast FP16 paths and flash-attention support.

GPU	VRAM	Realistic model ceiling (EXL2 4.0bpw)	Typical tok/s (32B, 4bpw)
RTX 3090 / 3090 Ti	24 GB	32B–34B with 8k–16k context	35–45
RTX 4090	24 GB	32B–34B with 16k–32k context (Q4 cache)	50–65
RTX 5090	32 GB	32B at long context, or 49B tight	70–90
2× RTX 3090 (tensor parallel)	48 GB	70B at 4.0–4.65bpw	20–30
A6000 / A6000 Ada	48 GB	70B at 4.65bpw, generous context	18–28

Numbers reflect the editorial team's aggregated community and internal figures for greedy single-stream generation; batching and speculative decoding shift them. For a fuller cross-engine picture, see our benchmarks hub, and estimate electricity and amortization with the cost calculator.

Sizing the KV cache

Context is not free. At FP16 the KV cache for a 32B model can consume several GB at 32k tokens. TabbyAPI supports Q8, Q6, and Q4 cache quantization, which roughly halves or quarters that footprint at a small quality cost. On a 24GB card, enabling cache_mode: Q4 is often what lets a 32B model reach 32k context without an out-of-memory crash.

Installation: Docker vs. Manual

Two supported paths. Docker is strongly recommended because it ships pre-compiled ExLlamaV2 and flash-attention wheels matched to a known CUDA version—the single biggest source of manual-install pain.

Configuring the OpenAI-Compatible Endpoint

Everything lives in config.yml. The fields that matter most on day one:

network:
  host: 0.0.0.0        # bind for LAN/remote access
  port: 5000

model:
  model_dir: models
  model_name: Qwen3-Coder-32B-exl2-4.0bpw
  max_seq_len: 32768
  cache_mode: Q4       # Q4/Q6/Q8/FP16
  tensor_parallel: false

logging:
  log_prompt: false

Authentication is on by default. On first launch TabbyAPI generates an api_tokens.yml containing an api_key and a separate admin_key. The admin key is required for the load/unload endpoints; the api key is what your OpenAI clients send. Never expose port 5000 to the public internet without a reverse proxy and TLS—bind to 127.0.0.1 and front it with Caddy or nginx if remote access is needed.

Once running, point any OpenAI SDK at it:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:5000/v1", api_key="YOUR_TABBY_API_KEY")
resp = client.chat.completions.create(
    model="Qwen3-Coder-32B-exl2-4.0bpw",
    messages=[{"role": "user", "content": "Refactor this function."}],
)
print(resp.choices[0].message.content)

One compatibility caveat worth knowing: TabbyAPI's sampling defaults lean toward creative writing (higher temperature, lower top_p) than OpenAI's neutral defaults. A client that relies on server-side defaults will see a different style. Set your own temperature and top_p explicitly for deterministic, code-oriented output.

Loading EXL2 / EXL3 Models

TabbyAPI only runs ExLlama-format quantized weights—EXL2 for ExLlamaV2, EXL3 for the newer engine. You cannot load a GGUF or a raw safetensors FP16 checkpoint. Download a pre-quantized repo from Hugging Face (community quanters publish EXL2 at multiple bit-per-weight targets) or quantize a model yourself with the ExLlamaV2 convert.py using a measurement-based calibration dataset.

EXL2 bpw	~Size (32B model)	Quality vs FP16	Fits 24GB?
3.0 bpw	~12 GB	Noticeable degradation	Yes, long context
4.0 bpw	~16 GB	Near-lossless for most tasks	Yes (sweet spot)
4.65 bpw	~19 GB	Very close to FP16	Yes, tighter context
5.0 bpw	~20 GB	Effectively lossless	Yes, short context only
6.0+ bpw	~24 GB+	Lossless	No (spills VRAM)

The 4.0bpw tier is the pragmatic default: near-lossless quality with room for a usable context window on 24GB. To browse quantization-aware model recommendations, see the model catalog. Model files go in model_dir; you can hot-swap them at runtime via the admin /v1/model/load endpoint without restarting the server—useful for switching between a coding model and a general chat model on the same card.

Performance Tuning & Benchmarks

Three levers deliver most of the speedup:

• Cache quantization (cache_mode: Q4): frees VRAM for longer context and larger batches with minimal quality loss.
• Tensor parallelism (tensor_parallel: true): splits a model across multiple GPUs. It enables 70B-class models on 2× 24GB but adds inter-GPU overhead, so single-stream tok/s drops versus a model that fits on one card.
• Speculative decoding: pair a small draft model with the target model to accelerate generation on predictable text—often a 1.5–2× gain on code.

In the editorial team's testing, a 32B EXL2 4.0bpw model on a single RTX 4090 with Q4 cache sustained roughly 55 tok/s single-stream at 8k context—comfortably ahead of an equivalent Q4_K_M GGUF under llama.cpp on the same card. That single-GPU efficiency is TabbyAPI's core reason to exist. All of these results, plus the raw data, are available programmatically through the BestLLMfor public API (CC BY 4.0) and the open-source BestLLMfor MCP server, so you can pull the numbers into your own tooling instead of copying tables.

Conclusion & Verdict

TabbyAPI is the right tool when your constraint is a single consumer NVIDIA GPU and your goal is maximum 4-bit throughput behind a standard OpenAI endpoint. It is not a horizontal-scaling platform. Judge it against that scope and it wins clearly.

Use case	Verdict	Why
Single 24GB GPU, max tok/s	✅ Best choice	Fastest INT4 kernel; simple ops
2× GPU running 70B locally	✅ Strong	Tensor parallelism works well
Drop-in OpenAI replacement	✅ Yes	Genuine /v1 compatibility
High-concurrency production serving	⚠️ Use vLLM	vLLM's continuous batching scales better
AMD / CPU-only / Apple Silicon	❌ No	NVIDIA CUDA only; no CPU offload

For a broader engine comparison and setup walkthroughs, browse our guides hub. Bottom line: install via Docker, load a 4.0bpw EXL2 model, enable Q4 cache, and you have a fast, private OpenAI endpoint running in minutes.