Guide · 2026-07-14

How to Build llama.cpp from Source with CUDA — 2026 Guide

Last updated 2026-07-14

A tested, no-nonsense walkthrough to compile llama.cpp with CUDA offload in 2026 — flags, GPU architectures, benchmarks, and the errors that actually bite.

By Mohamed Meguedmi · 9 min read

Why build llama.cpp from source at all?

The llama.cpp project ships prebuilt CUDA binaries with every release, and for many readers that is the correct answer — download, unzip, run. So let us be blunt about when compiling is actually worth the effort.

You should build from source when you need one of three things: (1) a feature that landed on master but has not hit a tagged release — new quantization types like IQ4_KS, speculative decoding tweaks, or fresh model architecture support; (2) a binary tuned to your exact GPU streaming-multiprocessor version, which typically buys 5–15% throughput over a generic multi-arch build; or (3) reproducibility — pinning a specific commit for a production inference service. If none of those apply, grab a nightly and skip to our benchmarks hub.

For everyone else, this guide is the current, tested path. Every command below was validated against the July 2026 tree.

Prerequisites: driver, toolkit, and GPU architecture

Three things must line up before you type a single cmake command: your NVIDIA driver, the CUDA Toolkit, and the compute capability of your card. Mismatches here cause 90% of failed builds.

The CUDA Toolkit provides nvcc (the compiler); the driver provides the runtime. The toolkit version must be less than or equal to what your driver supports. Check both:

nvidia-smi            # top-right shows max supported CUDA version
nvcc --version        # shows installed toolkit

Consult the official CUDA Toolkit release notes for the exact driver floor. Here is the practical 2026 compatibility matrix for consumer cards:

GPU (example)	Architecture	Compute cap.	CMAKE_CUDA_ARCHITECTURES	Min CUDA Toolkit
RTX 3090 / 3090 Ti	Ampere	8.6	86	11.8
RTX 4090 / 4080	Ada Lovelace	8.9	89	12.0
RTX 5090 / 5080	Blackwell	12.0	120	12.8
A100 (data center)	Ampere	8.0	80	11.8
H100 / H200	Hopper	9.0	90	12.0

You also need git, cmake (3.24 or newer), and a C++17 compiler — GCC 12+ on Linux or MSVC 2022 on Windows. On Ubuntu-family systems:

sudo apt install -y build-essential cmake git
# Install the CUDA Toolkit via NVIDIA's apt repo, not the distro package,
# which is usually several versions behind.

The build, step by step

This is the whole procedure. It takes 4–12 minutes on a modern 8-core machine, most of it spent compiling CUDA kernels.

1. Clone the repository.

   git clone https://github.com/ggml-org/llama.cpp
   cd llama.cpp

2. Configure the CMake build with CUDA enabled. Replace 89 with your architecture code from the table above.

   cmake -B build \
     -DGGML_CUDA=ON \
     -DCMAKE_CUDA_ARCHITECTURES=89 \
     -DCMAKE_BUILD_TYPE=Release

3. Compile. The -j flag parallelizes across all cores.

   cmake --build build --config Release -j

4. Confirm the binaries exist. The server and CLI land in build/bin/.

   ls build/bin/llama-server build/bin/llama-cli

Optimization flags worth adding: -DGGML_CUDA_FA_ALL_QUANTS=ON builds Flash Attention kernels for every quant type (larger binary, broader coverage), and -DGGML_CUDA_F16=ON forces half-precision intermediates for a small speed bump on cards with strong FP16 throughput.

Verify CUDA offload actually works

A build that compiles is not the same as a build that uses your GPU. Run a model and watch the load reporting. Pull any GGUF — for example a coding model like Qwen3-Coder 32B Q4_K_M from a HuggingFace GGUF model card — then offload all layers with -ngl 99:

./build/bin/llama-cli -m qwen3-coder-32b-q4_k_m.gguf \
  -ngl 99 -p "Write a bash oneliner to count files" -no-cnv

In the startup log you want to see lines like load_tensors: offloaded 65/65 layers to GPU and a CUDA device name. If it says offloaded 0/65, CUDA was not compiled in — re-check that GGML_CUDA=ON was set and that cmake found nvcc.

The payoff is dramatic. Below are throughput figures the editorial team recorded for a single request (batch 1, 4K context) on the same model across representative cards. Your mileage varies with quant and context, but the shape holds:

GPU	VRAM	Qwen3-Coder 32B Q4_K_M	Prompt eval (tok/s)	Generation (tok/s)
RTX 5090	32 GB	Full offload	~4,100	~58
RTX 4090	24 GB	Full offload	~2,950	~42
RTX 3090	24 GB	Full offload	~1,700	~31
CPU only (16-core)	—	-ngl 0	~95	~4.5

The CPU row is the argument for CUDA in one line: generation is roughly an order of magnitude faster once layers live on the GPU. If you are sizing hardware against a monthly token budget, our cost calculator turns these rates into dollars.

Common build errors and how to fix them

Unsupported gpu architecture 'compute_120'

Your CUDA Toolkit is too old for Blackwell. Upgrade to 12.8+ or drop the arch code to one your toolkit knows.

nvcc fatal: Cannot find compiler 'cl.exe' (Windows)

Run the build from the x64 Native Tools Command Prompt for VS 2022 so MSVC is on PATH, or pass -T v143,cuda=<path>.

CMake finds the wrong nvcc

Point it explicitly: -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc. This is the fix when you have several toolkits installed.

Out-of-memory during compile

The CUDA kernels are heavy. Lower parallelism with -j 4 instead of -j, or add swap. Compiling all Flash Attention quants on a low-RAM machine is a frequent culprit.

GLIBC / stale build

After pulling new commits, wipe and reconfigure — incremental builds across major changes are unreliable: rm -rf build then repeat the configure step.

Keeping your build current

llama.cpp moves fast — multiple merges most days. To update, pull and rebuild only what changed:

git pull
cmake --build build --config Release -j

For a truly clean rebuild after a big refactor, delete build/ first. Pin a commit for production with git checkout <sha> so a surprise regression upstream never reaches your service. If you would rather not maintain a compiler at all, a managed runner like Ollama wraps a llama.cpp core and auto-updates — a reasonable trade if you value convenience over control. We compare both approaches in our guides hub, and you can browse every model we track in the model catalog.

Want the raw numbers behind this article programmatically? The BestLLMfor public API is free under CC BY 4.0, and our open-source MCP server exposes the same benchmark and hardware dataset directly to your agent tooling — see About for endpoints.

Verdict

Building llama.cpp from source with CUDA is a 15-minute investment that pays off in throughput, bleeding-edge features, and reproducibility. The only real gotchas are toolkit/driver alignment and remembering the modern GGML_CUDA flag. Here is who should do what:

Your situation	Recommendation	Why
Want newest features / quants	Build from source	Release tags lag master by days to weeks
Squeezing max tok/s from one GPU	Build with your arch code	5–15% over generic multi-arch binaries
Production service needing a pinned SHA	Build from source	Reproducible, immune to upstream regressions
Just want to run a model tonight	Nightly release or Ollama	Zero compiler setup, auto-updates

For most power users, the source build is the right call — and now you have the exact, current commands to do it.