Which LLM for 12 GB of VRAM ?

#12 GB of VRAM: the tier where 14 billion parameters become possible

With 12 GB of VRAM, you can run 12- to 14-billion-parameter models in Q4 with plenty of room for context, or a 9B model in Q8. According to the QuelLLM catalog, a Gemma 4 12B in Q4 weighs 7 GB, a Qwen 3 14B in Q4 weighs 9 GB, and a Qwen 3.5 9B in Q8 weighs 10 GB. The 12 GB tier is where you move beyond 8- to 9-billion-parameter models, which are more limited in reasoning and writing quality, without yet reaching the 24-billion-parameter models that require 16 GB. Still out of reach: 27- to 35-billion-parameter models, which require between 16 and 21 GB in Q4.

How to read this table: add the KV cache and about one gigabyte of headroom to the model weights. A Qwen 3 14B in Q4 (9 GB) therefore leaves about 2 GB for context; a Qwen 3.5 9B in Q8 (10 GB) leaves only one, which limits context. Before going further, one correction: an earlier version of this guide described the 9B in Q8 on 12 GB as comfortable; the catalog lists 10 GB of weights, so the headroom is tight.

#12 GB cards: the ones that really have it

Many card families have versions with different capacities; the name alone is not enough. NVIDIA describes, for its RTX 4070 family, the 4070 Ti Super with 16 GB and the 4070 Ti, 4070 Super, and 4070 with 12 GB. The RTX 5070 offers 12 GB of GDDR7 on a 192-bit bus, while the 5070 Ti has 16. The RTX 3060 comes in 12 GB and 8 GB versions, with the 12 GB version being more interesting for local AI. On the AMD side, the RX 7700 XT guide places it in this tier.

For comparison, capacity is the primary criterion and bandwidth the second: at equal capacity, faster memory means smoother generation. Prices change every week; the site’s price tracker shows the cost per gigabyte of VRAM for 12 GB and 16 GB cards, letting you determine whether the next tier is worth the extra cost.

#Which models to choose on 12 GB based on your use case

For coding, the most capable specialized models in the catalog exceed 12 GB: Devstral Small 2 24B weighs 14 GB in Q4 and Qwen3-Coder 30B-A3B 19 GB. On 12 GB, stick with a general-purpose model of 12 to 14 billion parameters or Qwen 2.5 Coder 14B, whose catalog listing gives 9 GB in Q4. These models are comparable to the 8 and 16 GB options in the neighboring guides.

#Speed: the calculation you need to do yourself

Whether you have 12 GB or more, generation speed is capped by the card’s memory bandwidth divided by the size of the weights read for each token. Each manufacturer publishes this figure on the card’s specifications page, in Go/s. Simply divide it by the model size to get a theoretical ceiling, which real-world speed never fully reaches. The table gives this ceiling for 100 GB/s of bandwidth; multiply it by your card’s figure.

This ceiling explains why finer quantization is slower: going from Q4 to Q8 nearly doubles the weights read. It also explains why two cards with the same capacity can differ: the memory bus matters. NVIDIA specifies 192 bits for the RTX 5070 with 12 GB and 256 bits for the 16 GB 5070 Ti. We do not publish tokens per second by card because we lack a comparable source; for your hardware, ollama run with the --verbose option displays the actual speed.

#Verify that a model fits before adopting it

1. 01
   Read the catalog weights

   Open the model page in the QuelLLM catalog and note the size of the target quantization.
2. 02
   Add headroom

   Add about 1 GB for the system and 1 to 3 GB for the KV cache, depending on the target context. The total must stay under 12 GB.
3. 03
   Run and measure

   Load the model, ask a question, then run ollama ps: the PROCESSOR column should show 100% GPU.
4. 04
   Adjust if necessary

   Reduce the context, quantize the KV cache, or drop down one quantization level, in that order.

#Local RAG on 12 GB: how to allocate memory

A local RAG system combines three components: an embedding model that indexes your documents, optionally a reranker that reorders passages, and the LLM that writes the response. The first two are much smaller than the LLM and can run on either the GPU or the processor. Calculate the budget by adding all three, then leave room for the KV cache: the LLM's context grows with the number of passages you send it.

The second strategy is often best: indexing is a background task that doesn't need a GPU, while response generation benefits fully from VRAM. An embedder runs in the background during indexing, then goes idle; on a CPU, indexing a document folder takes longer, but it only happens once. We don't provide sizes for embedders here; consult the guides on embeddings and local RAG for each model's values.

#What 12 GB don't allow

24-billion-parameter models

Mistral Small 24B weighs 14 GB in Q4 according to the catalog: it exceeds the limit. For 24 billion parameters, you need 16 GB.

27- to 32-billion-parameter dense models

Qwen 3.8 27B and similar models weigh 16 GB in Q4; target 24 GB.

30- to 35-billion-parameter MoE models

Qwen 3.6 35B-A3B weighs 21 GB, GLM 4.7 Flash 19 GB, and Qwen3-Coder 30B-A3B 19 GB: out of reach on 12 GB, even though they read only 3 billion active parameters, because the entire model must reside in memory.

Long context on a 14B

A Qwen 3 14B in Q4 leaves about 2 GB: don't expect tens of thousands of tokens without quantizing the KV cache.

Ollama lets you quantize the KV cache to save space: according to its documentation, OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of the default f16, provided Flash Attention is enabled. The KV cache guide details this setting, and Ollama’s default context window, 4 096 tokens, can be changed with OLLAMA_CONTEXT_LENGTH.

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=16384 ollama serve
ollama ps

#12 GB or 16 GB: the right choice for you

• Which LLM for 16 GB of VRAM
• Which LLM for 8 GB of VRAM?
• LLM on RTX 3060 12 GB
• Quantize the KV cache to save VRAM
• Local RAG: introduction
• Local AI graphics card prices

• Source: Ollama FAQ (context, KV cache)
• Source: NVIDIA spec sheet for the RTX 4070 family
• Source: NVIDIA page for the RTX 5070 family
• QuelLLM model catalog (VRAM sizes)

#Frequently asked questions