Which LLM on RTX 3060 12 GB ?

#The RTX 3060 12 GB in figures

The RTX 3060 12 GB loads all models with 7 to 14 billion parameters in Q4 completely: Granite 4.2 8B (5.3 GB in Ollama), Qwen3.5 9B (6.6 GB), Gemma 4 12B (7.6 GB), or Qwen3 14B (9.3 GB). Its compute power is not what determines this; its memory does. The 12 GB of GDDR6 sets the maximum model and context size. The 360 GB/s of bandwidth sets the speed, since the card rereads most of the weights for every token. Third-party measurements report 50 to 65 tokens per second on a model with 8 to 9 billion parameters and 26 to 33 on a 14B, enough for a chat or coding assistant. A dense model with 27 to 32 billion parameters overflows the capacity: you need an MoE, a second card, or more VRAM.

Architecture

Ampere, 3,584 CUDA cores, 3rd-generation Tensor Cores (NVIDIA spec sheet).

Memory

12 GB of GDDR6 on a 192-bit bus. Tom's Hardware puts the peak bandwidth at 360 GB/s.

Power supply

170 W of board power; NVIDIA specifies 550 W for the system's power supply.

Compatibility Ollama

Compute capability 8.6, NVIDIA driver 550 or later on Linux, 551.61 or later on Windows.

Pricing

No prices here; they change every week: see the site's AI GPU Price Tracker.

#1. Which models fit in 12 GB

The rule of thumb: file weights, plus the context cache, plus about 1 GB of headroom for the runtime. The table shows the weights listed by Ollama for models that the card loads entirely and for a few neighboring models that do not fit. The full list, across all cards, is in the guide dedicated to 12 GB of VRAM.

Two pitfalls. The number in the name does not indicate the size: gemma4:e4b weighs 9.6 GB and gemma4:e2b weighs 7.2 GB in Ollama, which is more than the 12B (7.6 GB). Check the size before downloading. Also, Qwen3.5 9B’s Q8_0 technically fits, but leaves only about 1 GB for context and buffers: prefer the 6.6 GB Q4_K_M, which frees up more than 5 GB.

• Which LLM for 12 GB of VRAM: the complete list across all cards

#2. Install Ollama and verify that everything runs on the GPU

1. 01
   Check the NVIDIA driver

   Ollama requires driver version 550 or later on Linux and 551.61 or later on Windows. The nvidia-smi command displays the installed version and the card name.
2. 02
   Install Ollama

   On Windows and macOS, use the installer from ollama.com. On Linux, a single command is enough; the NVIDIA GPU is detected automatically.
3. 03
   Run a first 12 GB model

   ollama run gemma4:12b downloads 7.6 GB and launches a model that also accepts images. For more speed, ollama run qwen3.5:9b weighs 6.6 GB.
4. 04
   Check placement

   ollama ps, in a second terminal, displays the PROCESSOR column. The expected value is 100% GPU.

nvidia-smi
curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4:12b
ollama ps

#3. Published speeds and the theoretical ceiling

For each token, the card rereads the active weights from VRAM. The theoretical ceiling is therefore bandwidth divided by model weight: 360 GB/s divided by 5.3 GB gives about 68 tokens/s for Granite 4.2 8B, 55 for Qwen3.5 9B (6.6 GB), 47 for Gemma 4 12B (7.6 GB), and 39 for Qwen3 14B (9.3 GB). These are ceilings, never measurements: Qwen3.5 9B is published at 49.9 tokens/s (about 90% of the ceiling) and Qwen3 14B at 33.4 (about 86%), confirming that memory limits speed.

Differences on the same model (51.3 versus 64.5 tokens/s for Llama 3.1 8B) come from the method: software, prompt length, and system. Compare only figures from the same source; none were measured by the site.

Generation is only half the experience. LocalScore measures prompt processing at 1,483 tokens/s on Llama 3.1 8B and 759 tokens/s on Qwen2.5 14B. A 10,000-token document therefore waits about 7 seconds before the first response on an 8B model and 13 seconds on a 14B model. This processing depends on compute, not bandwidth.

#4. A 35B model on 12 GB: the MoE case

An MoE (mixture of experts) model activates only a fraction of its parameters for each token. Qwen3.6 35B-A3B activates approximately 3 billion, but its 23 GB file in Ollama does not fit in 12 GB. The solution planned by llama.cpp is the --n-cpu-moe option: it keeps the expert weights from the first N layers in RAM and leaves attention on the GPU.

A user published their measurements on September 22, 2026: RTX 3060 12 GB, Ryzen 5 3600, 32 GB of DDR4-3200, Qwen3.6-35B-A3B in Q4_K_M (about 21 GB). They measured 47 tokens/s on a short question, 45 with 12,000 tokens of context, and 37 with 45,000. With a conventional layer split, the same machine dropped to 7.1 tokens/s on 15,000 tokens (measured with Qwen3-30B-A3B), versus 25.3 after tuning. Only one in four layers of the Qwen3.5 and 3.6 series uses standard attention, so their speed drops less as the context grows.

Three caveats: this is one person’s measurement, using llama.cpp rather than Ollama; it assumes 32 GB of RAM; and the RAM speed matters. Its DDR4-3200 was running at 2,400 by default, and the XMP or DOCP profile added 10 to 17% throughput.

llama-server -m modele-moe-Q4_K_M.gguf -ngl 99 --n-cpu-moe 26 -c 32768 -ctk q8_0 -ctv q8_0

The value 26 is the author's value for their model: the higher it goes, the more VRAM is freed, but the slower generation becomes. Look for the lowest value that leaves about 1 GB of VRAM free.

• Qwen3.6 35B-A3B locally: testing and VRAM requirements

#5. Context and KV cache

#Context, the first limit of 12 GB

Ollama allocates 4,096 context tokens by default under 24 GiB of VRAM, while its documentation recommends at least 64,000 tokens for agents and coding tools. Between the two, each token costs memory. Example with Qwen3 14B, whose architecture is published: 40 layers, 8 KV heads, 128 dimensions per head. Per token, the KV cache in 16-bit occupies 2 (keys and values) × 40 × 8 × 128 × 2 bytes, or 160 KiB: about 2.4 GiB for 16,000 tokens and 3.7 GiB for 24,000.

The 9.3 GB model represents 8.7 GiB, leaving 12 GiB of VRAM. At 16,000 tokens, the total reaches 11.1 GiB: less than 1 GiB remains for runtime buffers, which is too tight. At 24,000 tokens, it reaches 12.3 GiB: loading fails or spills over to the CPU. With an 8-bit cache, 24,000 tokens use only 1.8 GiB, for 10.5 GiB total: it fits.

#8-bit KV cache and Flash Attention

Ollama automatically enables Flash Attention when the software and card support it; OLLAMA_FLASH_ATTENTION=1 forces it. This is a prerequisite for quantized caching: with OLLAMA_KV_CACHE_TYPE=q8_0, the cache uses about half the memory of f16 with very little precision loss, according to the documentation. The setting is global: all served models use the same cache type.

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=16000 ollama serve

• Quantizing the KV cache: save VRAM with long contexts

#6. Two RTX 3060 12 GB: 24 GB of VRAM, not twice as fast

Two cards offer 24 GB of VRAM, enough for a dense 27 to 32B model in Q4: Qwen3.5 27B weighs 17 GB in Ollama. Ollama automatically distributes the model across all cards when it does not fit on one. In llama.cpp, the default split is by layer, in a pipeline: the layers run one after another. The tensor mode, which parallelizes computation, is marked experimental.

The gain is therefore capacity, not speed: each token rereads all the weights at 360 GB/s per card. The estimated ceiling for a 32B model weighing 20 GB is 360 divided by 20, or about 18 tokens/s. A RTX 3090 offers 24 GB on a single card at 936 GB/s according to RunAIHome: the same model then tops out at around 47 tokens/s, or 2.6 times more. Two 170 W cards also mean 340 W for the GPU alone, whereas the 550 W of NVIDIA applies to a single card.

This setup makes sense if you already own a card and want to reach 24 GB. If you are starting from scratch, compare it first with a single 16 or 24 GB card.

• Multi-GPU LLMs with llama.cpp: tensor-split, settings, and pitfalls

#7. RTX 3060 8 GB, 3060 Ti, 4060, 5060: what 12 GB changes

The 3060 8 GB is not simply a 3060 12 GB with less memory: its bus drops from 192 to 128 bits, and Tom's Hardware puts its bandwidth at 240 GB/s versus 360 GB/s, or 33% less. LocalScore records it at 37.0 tokens/s versus 51.3, or 28% less on the same model. These speeds are isolated results, without controlled conditions: the 3060 Ti comes out ahead there, which its 256-bit bus makes plausible, but its 8 GB excludes it from 12- or 14-billion-parameter models. The 5060 offsets its narrow bus with GDDR7, without changing its capacity. Only the 5060 Ti 16 GB exceeds the 3060 in VRAM.

nvidia-smi --query-gpu=name,memory.total --format=csv

#Buying a 3060 12 GB: when it makes sense and when to move on

Buy a 3060 12 GB if

you want a first GPU for Ollama, 7B to 14B models, chat, document RAG, or coding assistance, and capacity matters more than speed.

Aim for 16 GB if

you want Gemma 4 12B with a long context, or more headroom for a 14B. The RTX 5060 Ti 16 GB is the first step; the comparison page quantifies the difference.

Aim for 24 GB if

Dense models from 27B to 32B are your target: a 24 GB card avoids splitting the model across two cards and provides much higher bandwidth.

Avoid or verify if

The announcement does not specify the memory: the 8 GB version offers neither the capacity nor the bandwidth of the 12 GB version.

• Source: NVIDIA spec sheet for RTX 3060 and 3060 Ti
• Source: Ollama documentation, context length
• Source: Ollama documentation, supported GPU
• Source: llama.cpp, server options (--n-cpu-moe, KV cache)
• Source: LocalScore, results from the RTX 3060 12 GB

#Frequently asked questions

Prices change quickly: every Monday and Thursday, our tracker records the lowest price for local AI graphics cards, along with the price per GB of VRAM.

• Local AI graphics card prices (updated twice a week)
• RTX 3060 12 GB vs. RTX 5060 Ti 16 GB comparison