Guide · 2026-05-16

TabbyAPI — Self-Host an OpenAI-Compatible API in 10 Minutes

An ExLlamaV2/V3-powered FastAPI server that drops in as a local replacement for OpenAI — faster than Ollama, leaner than vLLM, and ready in ten minutes flat.

By Mohamed Meguedmi · 9 min read

If you have followed local LLM tooling at all in 2025-2026, you already know the standard advice: install Ollama, point your client at http://localhost:11434/v1, done. That advice is fine for hobbyists. It is also leaving 30-60% of your GPU on the table.

TabbyAPI — the FastAPI server built on top of ExLlamaV2 and the newer ExLlamaV3 — is what serious single-user self-hosters run instead. It is OpenAI-compatible, it streams tool calls correctly, and on an RTX 4090 it pushes Llama 3.3 70B at 3.0 bpw to roughly 32-38 tokens/s. That is the headline. The rest of this guide gets you there in ten minutes and tells you when not to bother.

What TabbyAPI actually is (and isn't)

TabbyAPI is maintained by The Royal Lab as the official API frontend for the ExLlama family of inference kernels. It exposes the OpenAI v1 schema — /v1/chat/completions, /v1/completions, /v1/embeddings, plus a non-OAI /v1/token/encode for tokenizer access — and adds quality-of-life features the OpenAI spec never had: hot model swapping, draft-model speculative decoding, tensor parallelism across multiple GPUs, and per-key permissions via a simple api_tokens.yml.

One thing it is not: a GGUF runner. TabbyAPI loads EXL2 and EXL3 quants only. If your collection is GGUF, the same team maintains YALS as a drop-in sister project with the same API surface. Do not confuse this project with Tabby (tabbyml.com), which is an entirely unrelated self-hosted Copilot clone — both projects ranking on the same SERP is a long-running source of pain.

The 10-minute install

The official installer is a shell script that creates a venv, resolves CUDA wheels, and pulls the matching ExLlamaV2/V3 binary for your driver. On a clean Ubuntu 24.04 or Arch box with CUDA 12.4+ and Python 3.11:

git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
./start.sh

On first run, start.sh prompts for CUDA version, AMD ROCm, or CPU. Pick CUDA 12.x for any RTX 20-series or newer. The script installs exllamav2, exllamav3, fastapi, and a pinned torch, then exits asking you to configure a model.

Copy the example config and edit it:

cp config_sample.yml config.yml

The four fields that matter:

• model_dir — path to your models folder, e.g. /srv/models
• model_name — the subdirectory containing the EXL2/EXL3 quant
• max_seq_len — context length; match the model's native window
• cache_mode — Q4 halves KV memory with almost no quality loss

Grab a model. For 24 GB cards, the current sweet spot is Llama-3.3-70B-Instruct-exl2 at 3.0 bpw or, for EXL3, turboderp/Llama-3.3-70B-Instruct-exl3 at 3.5 bpw. Download with huggingface-cli:

huggingface-cli download turboderp/Llama-3.3-70B-Instruct-exl2 \
  --revision 3.0bpw --local-dir /srv/models/llama-3.3-70b-exl2-3.0bpw

Restart ./start.sh and the server binds on 0.0.0.0:5000. Verify:

curl http://localhost:5000/v1/models

That's the ten minutes. Total wall time, assuming the model is already on disk, is closer to four. The bottleneck is the 40 GB HuggingFace download.

Wiring the OpenAI SDK to your local server

The point of all this is that your existing code does not change. Python:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:5000/v1", api_key="sk-local")
resp = client.chat.completions.create(
    model="llama-3.3-70b-exl2-3.0bpw",
    messages=[{"role":"user","content":"Summarize ExLlamaV3 in one sentence."}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

If you set disable_auth: true in config.yml, the api_key field is ignored. For anything exposed beyond localhost, leave auth on and copy the generated admin key from api_tokens.yml. TabbyAPI distinguishes admin keys (can load/unload models, change samplers) from user keys (inference only) — a small thing the OpenAI spec lacks and that matters the moment you share an endpoint with a teammate.

Benchmarks: where TabbyAPI actually wins

Numbers below come from our standard test harness (batch=1, 2k prompt, 512-token completion, prefill excluded from t/s). See /methodology/ for the full procedure.

Model / quant	GPU	TabbyAPI (EXL2/3)	llama.cpp (GGUF Q4_K_M)	vLLM (AWQ)
Llama-3.1-8B Instruct	RTX 4090	148 t/s	112 t/s	134 t/s
Qwen2.5-32B Instruct	RTX 4090	54 t/s	38 t/s	47 t/s
Llama-3.3-70B (3.0 bpw)	RTX 4090	34 t/s	22 t/s (Q3_K_M, offload)	OOM
Llama-3.3-70B (4.5 bpw)	2× RTX 3090 TP	26 t/s	17 t/s	23 t/s
Qwen3-Coder 32B (4.0 bpw)	RTX 4090	48 t/s	34 t/s	42 t/s

Two findings worth pulling out. First, the gap is largest on dense models that fit fully in VRAM — exactly the consumer-card use case. Second, vLLM matches or beats TabbyAPI the moment you cross a batch size of four. If you are building a multi-tenant service, this guide is not for you.

EXL3 vs EXL2: is the new format worth it?

ExLlamaV3, released by turboderp in November 2025, replaces the EXL2 quantization scheme with a new codebook-based approach borrowed from QTIP research. At equal bits-per-weight, EXL3 is measurably closer to FP16 in perplexity — particularly below 4 bpw, where EXL2 historically degraded.

Quant	Format	Wiki2 PPL (Llama-3.3-70B)	VRAM (8k ctx, Q4 cache)
FP16	safetensors	3.84	~140 GB
4.5 bpw	EXL2	3.91	~42 GB
3.5 bpw	EXL3	3.93	~33 GB
3.0 bpw	EXL2	4.18	~28 GB
3.0 bpw	EXL3	3.99	~28 GB

Verdict: on a 24 GB card, use EXL3 at 3.0-3.5 bpw. The 70B class becomes genuinely usable, not a science-fair stunt. On 48 GB and up, EXL2 at 4.5-5.0 bpw is still slightly faster and the quality difference disappears.

Speculative decoding and tensor parallelism

Two features push TabbyAPI's lead further on the right hardware.

Draft models. Set draft_model_name to a small same-family model (e.g. Llama-3.2-1B alongside Llama-3.3-70B) and TabbyAPI runs speculative decoding automatically. Expected uplift on code/structured output: 1.4-1.8×. On pure prose: 1.1-1.2×. The 1B draft costs ~2 GB VRAM.

Tensor parallelism. With tensor_parallel: true across two or more identical GPUs, TabbyAPI splits each layer instead of pipelining. On 2× RTX 3090, this is the difference between 14 t/s and 26 t/s on a 70B at 4.5 bpw. Cards must be identical (or near-identical) and connected via PCIe 4.0 x8 minimum — NVLink helps but is no longer required as of ExLlamaV2 0.2.x.

Production hardening

Out of the box, TabbyAPI is a developer tool. To run it as a real service on a Tailscale network or behind a reverse proxy, change three things:

1. Set disable_auth: false and rotate the admin key. The default api_tokens.yml is generated on first run — back it up, do not commit it.
2. Bind to localhost and put Caddy or nginx in front for TLS. TabbyAPI's built-in HTTPS is functional but offers no cert rotation.
3. Run under systemd with Restart=on-failure. A sample unit file ships in docs/systemd/.

Logs go to stdout in JSON when log_prompts: false and log_generation_params: false — turn both off for any deployment touching real user data.

When NOT to use TabbyAPI

Honest negatives, because the SERP for this query is full of breathless tutorials:

• You want GGUF models. Use YALS, llama.cpp's llama-server, or LM Studio's headless mode. TabbyAPI will not load them.
• You need to serve more than one or two concurrent users. vLLM and SGLang are designed for this. TabbyAPI's batching exists but is not its strength.
• You are on a Mac. ExLlama is CUDA/ROCm only. On Apple Silicon, use MLX-LM's server or llama.cpp Metal.
• You want a one-click experience. Ollama exists for a reason. TabbyAPI rewards readers of YAML files.

For a broader comparison of self-hosted OAI-compatible servers including vLLM, TGI, LocalAI and Ollama, see our cost calculator — it factors in tokens/s per dollar of GPU for the most common consumer and prosumer cards. The full benchmark dataset is also exposed via the BestLLMfor public API (CC BY 4.0) and mirrored on the quelllm-mcp open-source MCP server if you want to query it from an agent.

Verdict

Use case	Pick	Why
Single-user, max tokens/s on one GPU	TabbyAPI	ExLlama kernels are the fastest path through Ampere/Ada/Blackwell.
2-8 concurrent users, same hardware	vLLM	Continuous batching pays off above batch=4.
GGUF library, easy setup	Ollama / YALS	No reformat, no learning curve.
Apple Silicon	MLX-LM server	Native Metal, no CUDA dependency.
Air-gapped enterprise	vLLM or TGI	Battle-tested at scale, audited code paths.

For developers running their own inference on a single consumer or prosumer GPU, TabbyAPI is the right default in 2026. The ten-minute install is real, the OpenAI compatibility is genuine, and ExLlamaV3 finally makes 70B-class quality viable on a 24 GB card. Choose it unless one of the negatives above applies — in which case the verdict table tells you exactly where to go instead. More context on our editorial process at /about/.