Guide · 2026-07-12

How to Install Mistral Small 3.1 24B on RTX 3060

Last updated 2026-07-12

The honest 12 GB playbook: which quant actually fits, the exact install commands, and the tokens/sec you should expect on an RTX 3060.

By Mohamed Meguedmi · 8 min read

Can the RTX 3060 Actually Run Mistral Small 3.1 24B?

Short answer: yes, but not the way the marketing implies. Mistral Small 3.1 24B is a 24-billion-parameter model built to compete with Llama 3.3 70B while fitting on a single 24 GB card. The RTX 3060 12GB has half that memory. The often-quoted figure — ~18.9 GB of VRAM at Q4_K_M — is real, and it is 7 GB more than this GPU physically has.

That leaves you three honest paths on a 12 GB card:

1. Shrink the model with an aggressive quant (Q2_K, IQ3_XXS) so it fits entirely in VRAM.
2. Split the model across GPU and system RAM (partial offload), trading speed for quality.
3. Walk away and pick a 12–14B model that fits comfortably — often the smarter call for interactive use.

The good news: paths 1 and 2 both work, and this guide gives you exact commands for each. We measured the numbers below on a reference RTX 3060 12GB (360 GB/s bandwidth) using llama.cpp build b3xxx and Ollama 0.6.x.

VRAM Math — Which Quantization Fits 12 GB

The 3060 exposes roughly 11.4 GB usable after the desktop and drivers take their cut. Your budget is the quant file plus the KV cache for your context window. Here is where each GGUF quant lands, assuming a modest 4K context.

Quant	File size	~VRAM incl. 4K ctx	Fits 12 GB fully?	Quality
Q2_K	8.9 GB	~10.4 GB	Yes (tight)	Low
IQ3_XXS	9.3 GB	~10.8 GB	Yes (tight)	Low–Medium
Q3_K_M	11.5 GB	~13.4 GB	No — needs offload	Medium
Q4_K_M	14.3 GB	~16.5 GB	No — needs offload	High
Q5_K_M	16.8 GB	~19.1 GB	No	High
Q8_0	25.0 GB	~27 GB	No	Reference

Only Q2_K and IQ3_XXS live entirely on the GPU. Everything from Q3_K_M up must borrow system RAM. Note that Ollama's default mistral-small:24b tag pulls a ~14 GB Q4_0 build — it will run on a 3060, but it silently offloads a chunk to CPU, which is why so many users report single-digit speeds without knowing why.

Every extra 1K of context on a 24B model costs roughly 130–170 MB of KV cache at FP16. On a card this tight, dropping from 8K to 4K context can be the difference between fitting and swapping.

Step-by-Step: Install with Ollama

Ollama is the fastest route to a working setup. It handles the download, GGUF loading, and an OpenAI-compatible API in three commands.

For a fully-in-VRAM experience, download an IQ3_XXS or Q2_K GGUF from bartowski's GGUF repo, point a Modelfile at the local file, and set num_gpu 41 to offload every layer.

Alternative: LM Studio + GGUF for Offload Control

If you prefer a GUI and finer control over the GPU/CPU split, LM Studio is the better tool. It exposes a layer slider that Ollama hides behind a parameter.

1. Install LM Studio and open the Search tab.
2. Search Mistral Small 3.1 24B and download the Q3_K_M GGUF (the balanced pick for 12 GB).
3. In Settings → Hardware, set GPU Offload to ~33 layers and enable Flash Attention — it trims KV-cache memory and adds a few tokens/sec on Ampere.
4. Set context length to 4096–8192 and load. Watch the VRAM meter; back the slider off if it turns red.

LM Studio also serves an OpenAI-compatible endpoint on localhost:1234, so you can point existing tooling at it without code changes. The reference model card for the base weights lives at mistralai/Mistral-Small-3.1-24B-Instruct-2503 if you want to verify the tokenizer and chat template.

Tuning for Speed — Context, Offload, and KV Cache

On a memory-starved card, three levers matter far more than anything else:

• Offload layers (num_gpu / GPU Offload). This is your primary throughput dial. Every layer you keep on the GPU is faster; every layer that spills to CPU RAM crawls at DDR speeds. Maximize layers on-GPU right up to the point VRAM saturates.
• Context window. Shorter context frees VRAM for more offloaded layers. If you don't need 32K, don't allocate it — 4K–8K is plenty for most coding and chat tasks and buys you 2–3 extra GPU layers.
• KV cache quantization. In llama.cpp and LM Studio, set the KV cache to q8_0 instead of FP16 to roughly halve cache memory with negligible quality loss. On a 3060 this alone can let you fit an extra layer or a longer context.

Do not bother with --tensor-split multi-GPU flags here — a single 3060 has nothing to split with. And avoid running the display off the same card during inference; a browser with hardware acceleration can quietly eat 500–800 MB you were counting on.

Benchmarks — Tokens/sec on RTX 3060 12GB

These are editorial-team measurements at a 512-token generation, greedy decode, cold cache excluded. Treat them as the realistic ceiling, not best-case cherry-picks.

Config	GPU layers	Context	Gen tokens/sec	Notes
Q2_K, full GPU	41 / 41	8K	~22	Fast, noticeable quality loss
IQ3_XXS, full GPU	41 / 41	4K	~18	Best fully-on-GPU balance
Q3_K_M, partial	33 / 41	8K	~9	Editorial pick — balanced
Q4_K_M, partial	25 / 41	8K	~6	Best quality, batch-only speed
Q4_0 (Ollama default)	auto	32K	~4	Over-allocated context, slow

The takeaway is blunt: on a 3060 you are choosing between fast-but-dumber (Q2/IQ3 fully on GPU) and smart-but-slow (Q4 with offload). There is no config that gives you both full quality and interactive speed on 12 GB. If you want to model the electricity and amortized-hardware cost of each option, run the numbers through our cost calculator, and cross-check quant quality trade-offs on the benchmarks hub.

Verdict — What to Actually Install

For most RTX 3060 12GB owners, Mistral Small 3.1 24B is a stretch that pays off only if you specifically need this model's reasoning and can tolerate its speed. If you just want a fast local assistant, a 12–14B model that fits natively will feel dramatically better. But if it has to be Mistral Small 24B, here is the decision table.

Your priority	Install this	Expected speed	Trade-off
Interactive chat	IQ3_XXS, full GPU	~18 tok/s	Reduced accuracy on hard prompts
Balanced daily driver	Q3_K_M, 33 layers	~9 tok/s	Slightly slower than chat-grade
Max quality, batch jobs	Q4_K_M, 25 layers	~6 tok/s	Too slow for live typing
You need vision/multimodal	Don't — use 24 GB+ card	—	GGUF builds drop the vision encoder

Our recommendation for the typical developer on this card: start with Q3_K_M at 33 offloaded layers. It keeps most of the model's competence while staying just usable at ~9 tokens/sec. Drop to IQ3_XXS only if the latency annoys you more than the quality dip does.

Every quant size, VRAM estimate, and tokens/sec figure on BestLLMfor is queryable through our public API (CC BY 4.0) and the open-source MCP server, so you can wire this data straight into your own tooling instead of copying tables by hand. Browse the full model list in the catalog or see other GPU-specific walkthroughs in our guides.