llama.cpp on Blackwell: building for sm_120

Why prebuilt llama.cpp fails on RTX 50-series cards

Blackwell consumer GPUs — the RTX 5070 Ti and the rest of the 50-series — report a new CUDA compute capability of 12.0, which NVIDIA's toolchain names sm_120. That number is the crux of every build problem you will hit. A binary compiled for older architectures (say sm_86 for Ampere or sm_89 for Ada) contains no native machine code your card can run.

Normally CUDA papers over this with PTX just-in-time compilation: if the binary embeds forward-compatible PTX, the driver JIT-compiles it for the new architecture at load time. But many prebuilt llama.cpp releases and older toolkits don't emit PTX new enough for sm_120, so you get one of three outcomes — a hard CUDA error: no kernel image is available for execution on the device, a JIT failure, or the quiet worst case where llama.cpp decides no usable GPU exists and offloads zero layers to the card. The last one is the trap: nothing errors, but your RTX 5070 Ti sits idle while the CPU does the work. If you are still deciding between runtimes before you commit to a source build, our Ollama vs llama.cpp guide covers the trade-offs.

CUDA toolkit, driver, and the flags that matter

Blackwell device code first shipped in CUDA Toolkit 12.8, so that is the practical floor — earlier toolkits simply cannot target sm_120. Pair it with a recent driver and build llama.cpp from source rather than trusting a release binary. The two flags that actually control the outcome are -DGGML_CUDA=ON (enable the CUDA backend) and -DCMAKE_CUDA_ARCHITECTURES=120 (emit native sm_120 code). Leaving the architecture unset makes cmake guess, and on a brand-new card that guess is frequently wrong.

The general shape of the build is unchanged from any other CUDA target: clone the repo, configure with cmake, compile. Because exact package names, cmake option spellings, and tags do drift between releases, pull the current instructions straight from the official llama.cpp repository rather than copying a version-stamped command from a blog. NVIDIA's CUDA GPUs compute-capability list confirms the 12.0 value for your card if you want to double-check before you build.

The build errors you'll actually hit — and the fix

Four failures cover almost every sm_120 report:

• "no kernel image is available for execution on the device" — the binary has no sm_120 code. Rebuild with -DCMAKE_CUDA_ARCHITECTURES=120; do not rely on a prebuilt release.
• "nvcc fatal: Unsupported gpu architecture 'compute_120'" — your CUDA toolkit predates Blackwell. Upgrade to 12.8+; a newer driver alone is not enough, because the compiler, not the driver, generates device code.
• Model loads but runs on CPU — the CUDA backend never initialized. Confirm -DGGML_CUDA=ON was set at configure time (check the cmake output), then verify nvidia-smi shows memory allocated while the model is loaded.
• Compiler / host-compiler mismatch or missing CUDACXX — cmake found the wrong toolkit. Point CUDACXX at the 12.8+ nvcc and clear the stale build directory before reconfiguring, since cmake caches the first compiler it finds.

A blank build directory solves a surprising share of "I changed the flag but nothing happened" cases — the CUDA architecture is cached, so editing it in place is often ignored. When in doubt, delete build/ and configure fresh.

Benchmarking before and after: what to measure

The honest "before" number on a misbuilt setup is whatever your CPU manages — often single-digit tokens/s on a 7–8B model, or a hard crash. The "after" is full GPU offload, where a 16 GB Blackwell card runs the same quant many times faster. Rather than quote fabricated figures, measure your own: llama.cpp's llama-bench reports prompt-processing and token-generation throughput for a fixed model and context, which is the number that matters for chat latency.

Keep the comparison clean. Fix the model file, the quant, the context length, and the number of offloaded layers, then change only the build. Watch nvidia-smi to confirm the GPU is actually loaded — if VRAM usage stays near zero, you are still measuring CPU and the build didn't take. Our own RTX 5070 Ti runs live on the leaderboard, and the test rig and settings are documented under benchmarks so you can reproduce them. Quantization choice moves throughput as much as the build does; if the trade-offs are new to you, start with quantization explained.

Sizing models to 16 GB on a 5070 Ti

A correct build only helps if the model fits. Using the standard rules of thumb — FP16 ≈ 2 GB per billion parameters, Q8 ≈ 1.07 GB/B, Q4_K_M ≈ 0.58 GB/B, plus roughly 20% for KV-cache and overhead at 8K context — here is how common sizes land on a 16 GB card:

So a 5070 Ti comfortably runs 8B–14B models fully on the GPU at Q4_K_M, which is exactly where the sm_120 build pays off. Anything at 32B or above needs partial offload or a smaller quant, and the moment layers spill to CPU your tokens/s collapses — which is why getting the build right matters more than chasing a bigger model. Plug your own target context and quant into the VRAM calculator before you download a 20 GB checkpoint; these figures are approximations, and real footprints shift with context length and cache settings.

Frequently asked questions

What CUDA version do I need for RTX 5070 Ti and other Blackwell GPUs?

You need CUDA Toolkit 12.8 or newer, because that is the first release with native sm_120 (compute capability 12.0) device code. Pair it with a 570-series or later driver. Always confirm current requirements against the official CUDA documentation, since versions change.

What cmake flags build llama.cpp for sm_120?

Configure with -DGGML_CUDA=ON to enable the CUDA backend and -DCMAKE_CUDA_ARCHITECTURES=120 to emit native Blackwell code. If you edit the architecture after a first build, delete the build directory first, because cmake caches the value.

Why does llama.cpp run on CPU instead of my Blackwell GPU?

The CUDA backend never initialized, usually because the binary lacks sm_120 code or GGML_CUDA was off at configure time. Rebuild from source with the correct flags, then check that nvidia-smi shows VRAM in use while the model is loaded.

How do I fix "no kernel image is available for execution on the device"?

That error means the compiled binary contains no code for your card's sm_120 architecture. Rebuild llama.cpp from source with -DCMAKE_CUDA_ARCHITECTURES=120 on CUDA 12.8 or newer instead of using a prebuilt release. Prebuilt binaries rarely target the newest architectures.

---

By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.