What Is CUDA? NVIDIA's GPU Computing Platform, Explained for AI Users

CUDA in plain English

A GPU contains thousands of small processors built to apply the same operation to many pieces of data at once. Originally the only way to use them was through graphics APIs: you had to disguise a calculation as a picture. CUDA, short for Compute Unified Device Architecture, removed the disguise. It lets a programmer write ordinary C, C++ or Python-callable code that runs directly on those processors.

In practice "CUDA" refers to four things at once: the programming model, the nvcc compiler, the runtime built into the driver, and a set of tuned libraries such as cuBLAS for linear algebra and cuDNN for neural-network primitives. Frameworks like PyTorch sit on top of those libraries, and AI applications sit on top of the frameworks. When a tool says it "supports CUDA," it means its heavy math ends up in those NVIDIA libraries. NVIDIA's own overview is at developer.nvidia.com/cuda-toolkit.

What are CUDA cores?

"CUDA cores" is NVIDIA's name for the parallel shader processors in a GPU. An RTX 4060 has 3,072; an RTX 5090 has 21,760. More cores means more arithmetic per second, which matters for image generation, training and the prompt-reading phase of a language model.

It matters much less for the part of LLM use people feel most, token generation. There the limit is how fast weights can be read from memory, not how fast they can be multiplied. A card with fewer cores and more, faster VRAM will often generate text faster than the reverse. That argument is laid out in why VRAM matters more than TFLOPS, and the memory side in what is VRAM.

Driver, runtime, toolkit: what do you actually need?

Component	What it is	Who needs it
NVIDIA driver	Lets the OS talk to the GPU; includes the CUDA driver runtime	Everyone with an NVIDIA card
CUDA runtime libraries	cuBLAS, cuDNN and friends	Bundled inside most apps and inside PyTorch wheels; you rarely install them yourself
CUDA Toolkit	Compiler (nvcc), headers, profilers	Only people compiling CUDA software, e.g. building llama.cpp from source

So for Ollama, LM Studio, ComfyUI Desktop or a pip install torch: update the driver and you are done. Installing the multi-gigabyte Toolkit "just in case" is the most common unnecessary step in local-AI tutorials. If you do need to compile, our guides cover building llama.cpp from source and the Blackwell-specific build.

Checking your versions

nvidia-smi        # driver version, and the MAXIMUM CUDA version it supports
nvcc --version    # the Toolkit version, if one is installed

These two numbers legitimately differ, and the first one confuses almost everyone. nvidia-smi reporting "CUDA Version: 13.0" means your driver can run software built for CUDA up to 13.0. It does not mean the Toolkit is installed, and software built against an older CUDA version still runs fine on a newer driver.

Compute capability by GPU generation

Every NVIDIA GPU has a compute capability number that identifies its architecture. Software is compiled for specific capabilities, and projects periodically drop the oldest ones. This, more than raw speed, is what ends a card's useful life for AI.

Generation	Example cards (BestLLMfor hardware database)	Compute capability	Status for local AI in 2026
Pascal	GTX 1060, 1070, 1080 Ti	6.1	Runs llama.cpp-based tools; no FP16 acceleration; dropped by some newer frameworks
Turing	GTX 1660, RTX 2060–2080 Ti	7.5	Minimum for vLLM (needs 7.0+); broadly supported
Ampere	RTX 3050–3090 Ti	8.6	Fully supported; adds BF16
Ada Lovelace	RTX 4060–4090	8.9	Fully supported; adds FP8
Blackwell	RTX 5050–5090	12.0	Fully supported on recent software; older builds must be updated or recompiled; adds FP4

The Blackwell row explains a wave of "my new RTX 50 card is not detected" reports: binaries compiled before that architecture existed contain no code for capability 12.0. The fix is a newer release of the tool, or a PyTorch build made for a recent CUDA version.

CUDA vs ROCm vs Metal vs Vulkan

Platform	Vendor	Hardware	State for local LLMs
CUDA	NVIDIA	NVIDIA GPUs	The reference target; everything supports it first
ROCm	AMD	Recent Radeon and Instinct	Good on Linux for supported cards; narrower card list; see best LLMs for Radeon with ROCm
Metal	Apple	Apple Silicon	Excellent through llama.cpp and MLX; see MLX on Apple Silicon
Vulkan	Khronos (open standard)	Almost any GPU, including integrated	Works nearly everywhere in llama.cpp-based tools; usually slower than the vendor's native stack
SYCL / oneAPI	Intel	Intel Arc and integrated	Functional, smallest ecosystem

CUDA's lead is not that the alternatives cannot do the math. It is nearly two decades of libraries, tooling and tested code paths. For inference with mainstream tools that gap has narrowed a great deal: llama.cpp runs on all five. For training, fine-tuning and brand-new research code, CUDA remains the only path that works on day one.

What this means when you buy

• If you want everything to work without troubleshooting, buy NVIDIA, and prioritize VRAM capacity over core count.
• If you mainly run GGUF models through Ollama or LM Studio, a Mac or a recent Radeon on Linux is a legitimate choice; you give up little.
• If you plan to fine-tune, run vLLM, or follow new research repositories, the lack of CUDA will cost you time every week.
• Avoid cards below compute capability 7.5 for a new purchase, whatever the price.

To see which models fit a given card, use the VRAM calculator. The hardware list behind this page is available through the BestLLMfor public API (CC BY 4.0) and our open-source MCP server. For background on the platform's history, see the Wikipedia entry on CUDA, and for the per-GPU capability list, NVIDIA's CUDA GPUs page.