Guide · 2026-07-28

How to Install text-generation-webui with CUDA on Windows

Last updated 2026-07-28

A tested, no-nonsense guide to getting oobabooga's text-generation-webui running with full NVIDIA GPU acceleration on Windows 10/11.

By Mohamed Meguedmi · 8 min read

Why text-generation-webui, and why CUDA specifically

text-generation-webui (TGW), universally known as oobabooga after its creator, is the most loader-agnostic local LLM interface available. Unlike Ollama or KoboldCpp, which are effectively single-backend, TGW lets you swap between Transformers, llama.cpp (GGUF), ExLlamaV2/V3 (EXL2/EXL3), and TensorRT-LLM behind one Gradio UI and one OpenAI-compatible API. That flexibility is the reason it remains the editorial team's default recommendation for anyone who wants to test the same prompt across quantization formats.

CUDA is what turns it from a slow CPU toy into a usable tool. On an NVIDIA GPU, 4-bit inference is typically 15–40× faster than CPU-only for the same model. Every benchmark and instruction below assumes a genuine NVIDIA card with a current driver. If you have an AMD or Intel GPU, this guide is not for you — see our guides hub for the ROCm and Vulkan paths.

Prerequisites and hardware

Confirm these before you start. The single most common failure is an outdated driver that does not expose the CUDA version PyTorch was compiled against.

Requirement	Minimum	Recommended	Notes
OS	Windows 10 22H2	Windows 11 23H2+	64-bit only
GPU	NVIDIA, 6 GB VRAM	RTX 3090 / 4090, 24 GB	Pascal (GTX 10xx) works but is slow
NVIDIA driver	552.xx	566.xx or newer	Check with nvidia-smi
Disk	30 GB free	100 GB+ SSD	Env is ~8 GB; models are large
System RAM	16 GB	32 GB+	GGUF offload spills to RAM
CUDA Toolkit	Not required for the one-click install		Runtime is bundled

Open a terminal and run nvidia-smi. The top-right value ("CUDA Version: 12.x") is the maximum your driver supports, not what is installed. As long as it reads 12.1 or higher, the standard installer will work. If the command is not recognized, install or update your driver from NVIDIA's driver page first.

The one-click install (recommended)

This is the path we recommend for 95% of readers. It creates a self-contained Miniconda environment and installs the CUDA-enabled PyTorch wheel automatically — no system-wide Python pollution, no manual toolkit juggling.

1. Install Git. Grab it from git-scm.com and accept the defaults.
2. Clone the repository. Open PowerShell in a short path (avoid spaces and OneDrive folders):

   git clone https://github.com/oobabooga/text-generation-webui
   cd text-generation-webui

3. Run the installer. Double-click start_windows.bat, or run it from the terminal.
4. Choose your GPU vendor. When prompted, type A for NVIDIA. The script downloads Miniconda, creates installer_files\env, and pulls the CUDA build of PyTorch. This takes 5–15 minutes.
5. Wait for the launch. When it prints Running on local URL: http://127.0.0.1:7860, open that address in your browser.

The .bat wrapper always re-activates the correct environment, so from now on you launch TGW simply by double-clicking start_windows.bat.

Verify CUDA is actually being used

Do not skip this. A UI that loads does not prove the GPU is engaged — TGW will silently fall back to CPU. Verify from inside the bundled environment using cmd_windows.bat (this opens a shell with the right Python active):

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

A healthy result looks like 2.4.1+cu124 True NVIDIA GeForce RTX 4090. If it prints False, jump to the troubleshooting section — do not load a model yet. For a live check, load any small model, generate text, and watch nvidia-smi -l 1 in a second terminal: GPU utilization should spike and VRAM should climb.

Choosing the right loader

TGW's superpower is loader choice, but it confuses newcomers. Match the loader to the file you downloaded. Model cards on HuggingFace state the format in the filename — .gguf, -EXL2, or plain safetensors.

Loader	Format	Best for	VRAM behavior
ExLlamaV2 / V3	EXL2, EXL3	Fastest single-GPU chat	Must fit fully in VRAM
llama.cpp	GGUF	Limited VRAM, flexible offload	Splits GPU + CPU/RAM
Transformers	safetensors (fp16/8/4-bit)	Newest models, LoRA, training	High; needs bitsandbytes for 4-bit
TensorRT-LLM	Engine files	Max throughput, batch serving	Fixed at build time

Verdict: if the model fully fits your VRAM, use an EXL2 quant with ExLlamaV2 for the best tokens/sec. If you are VRAM-constrained, use a GGUF with llama.cpp and set the n-gpu-layers slider as high as it goes without an out-of-memory error. In the Model tab, set n-gpu-layers to a high value (e.g. 99 or more) to offload every layer to the GPU when it fits.

Real-world throughput

The numbers below are indicative single-request generation speeds (batch size 1, ~512-token output) for common Windows configurations. Treat them as a planning baseline, not a guarantee — driver, context length, and quant level all move the figure. Our full, reproducible methodology and the raw data behind these ranges are on the benchmarks hub.

GPU (VRAM)	Model & quant	Loader	Approx. tokens/sec	Street price (USD)
RTX 4060 Ti (16 GB)	Llama-3.1-8B Q4_K_M	llama.cpp	55–70	$450
RTX 3090 (24 GB)	Llama-3.1-8B 4.5bpw	ExLlamaV2	95–120	$750 used
RTX 4090 (24 GB)	Qwen3-Coder 32B Q4_K_M	llama.cpp	28–38	$1,900
RTX 4090 (24 GB)	Llama-3.1-8B 4.5bpw	ExLlamaV2	140–170	$1,900

Want to model the electricity and amortization cost of running these locally versus a hosted API? Plug your numbers into our cost calculator. And if you are building tooling on top of these figures, every benchmark on this site is available through the free BestLLMfor public API (CC BY 4.0) and the companion open-source MCP server — no scraping required.

Troubleshooting common CUDA errors

Three failures account for the vast majority of forum threads.

"No CUDA GPUs are available" / torch.cuda.is_available() is False. Almost always a CPU-only PyTorch wheel got installed. Re-run start_windows.bat and be sure you selected A (NVIDIA), not N (None/CPU). If it persists, delete the installer_files folder and reinstall — a stale environment will not self-correct.

"CUDA out of memory." The model plus its KV cache exceed your VRAM. Lower the quant (Q4 instead of Q6), reduce the context length, or — for GGUF — drop n-gpu-layers so some layers stay on the CPU.

bitsandbytes errors on Transformers 4-bit load. On Windows, bitsandbytes historically needed a Windows-specific wheel. Avoid the problem entirely by using a GGUF or EXL2 quant instead of loading raw safetensors in 4-bit.

General hygiene: install into a short path such as C:\text-generation-webui, keep the folder out of OneDrive (sync locks corrupt the environment), and keep your NVIDIA driver current. If PyTorch and driver versions ever mismatch after an update, the safest fix is a clean reinstall of the environment; see the official PyTorch install matrix to confirm which CUDA build your driver supports.

Verdict

For Windows + NVIDIA, the one-click installer with the ExLlamaV2 loader on EXL2 quants is the fastest, least error-prone route to a working local LLM. Reserve llama.cpp/GGUF for when a model does not fit in VRAM, and reach for Transformers only when you need bleeding-edge models or LoRA training.

Your situation	Do this
Model fits in VRAM, want max speed	ExLlamaV2 + EXL2 quant
Limited VRAM (6–12 GB)	llama.cpp + GGUF, tune n-gpu-layers
Newest model / LoRA / training	Transformers loader
Serving concurrent requests	TensorRT-LLM
cuda.is_available() is False	Reinstall env, select NVIDIA (A)

Once you are running, browse the model catalog to pick weights matched to your VRAM.