Guide · 2026-07-09

How to Install Llama 3.3 70B on RTX 4090 — Step-by-Step

Last updated 2026-07-09

The honest, data-driven playbook for running Meta's 70B flagship on a single 24 GB RTX 4090 — the two paths that actually work, and the one that doesn't.

By Mohamed Meguedmi · 9 min read

The 24 GB Reality Check

Every guide that promises Llama 3.3 70B "running on your RTX 4090 in 5 minutes" glosses over one number: the model does not fit. Meta's Llama-3.3-70B-Instruct is a 70-billion-parameter model. At the community-standard 4-bit tier (Q4_K_M), the weights occupy roughly 42.5 GB before you add a single token of context. A 4090 has 24 GB. There is no configuration flag that changes this arithmetic.

Llama 3.3 70B is worth the effort — Meta positions it as matching the older 405B model on many instruction-following tasks — but you must choose your compromise up front. Here is the VRAM budget for the default Q4_K_M path:

Component	Approx. size
Q4_K_M weights	~42.5 GB
KV cache @ 8K context (fp16)	~2.5 GB
CUDA + framework overhead	~1.5 GB
Total required	~46.5 GB
RTX 4090 VRAM	24 GB
Deficit	~22.5 GB

That deficit is the whole story. You close it in one of two ways: keep the full Q4 quality and push ~22 GB of layers onto the CPU/RAM (slow), or shrink the model itself with a heavier quantization until it fits in VRAM (fast, lower quality). The table below shows the trade space.

Quantization	File size	Fits in 24 GB?	Single-4090 speed	Quality
Q4_K_M	~42.5 GB	No — offload ~20 GB to RAM	~2–4 tok/s	Reference 4-bit
IQ3_XXS	~27.5 GB	Partial — ~5 GB to RAM	~6–9 tok/s	Good
IQ2_M	~24.1 GB	Tight, short context only	~12–16 tok/s	Noticeable loss
IQ2_XS	~21.1 GB	Yes, at 2–4K context	~15–22 tok/s	Real degradation

Community reports on r/LocalLLaMA occasionally cite 40–60 tok/s for IQ2_XS under ideal batching. Treat those as ceilings, not expectations — sustained interactive speed on a single 4090 lands in the 15–25 tok/s band.

What You Need Before You Start

Regardless of path, provision this first:

• GPU: RTX 4090 24 GB (or 2× for the recommended Q4 path). Driver 550+ with CUDA 12.4 or newer.
• System RAM: 64 GB minimum. The Q4 offload path parks ~20 GB of weights in RAM; 32 GB will thrash.
• Storage: 60–90 GB free on NVMe. GGUF files stream from disk on first load.
• OS: Linux (Ubuntu 22.04/24.04) is fastest; Windows 11 with WSL2 works. Native Windows is fine for Ollama.

Confirm the GPU is visible before installing anything:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

Path A — Fastest Setup with Ollama (Q4, Offloaded)

If you want a working chat in minutes and can tolerate 2–4 tok/s, Ollama is the shortest route. It auto-splits layers between GPU and CPU. It does not warn you that it is doing so.

1. Install Ollama. On Linux: curl -fsSL https://ollama.com/install.sh | sh. On Windows/macOS, download the installer from ollama.com.
2. Pull the model. ollama pull llama3.3:70b fetches the Q4_K_M build (~43 GB). Expect a long download.
3. Run it. ollama run llama3.3:70b. Ollama offloads as many layers as fit into 24 GB and computes the rest on CPU.
4. Verify the split. While generating, run ollama ps — a healthy single-4090 line reads something like 55%/45% CPU/GPU. That CPU fraction is why you are seeing single-digit tok/s.
5. Cap the context to save VRAM. Create a Modelfile: FROM llama3.3:70b then PARAMETER num_ctx 4096, and build it with ollama create llama33-4k -f Modelfile. A smaller KV cache lets one or two more layers stay on-GPU.

Verdict on Path A: correct answer only for offline, batch, or overnight jobs where quality matters more than latency. It is not an interactive experience.

Path B — Maximum GPU Residency with llama.cpp (IQ2_XS)

To get responsive generation from one card, you must make the model small enough to live in VRAM. That means a 2-bit importance-matrix quant and llama.cpp for fine-grained layer control.

1. Build llama.cpp with CUDA. git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp, then cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j.
2. Download an IQ2_XS GGUF. Bartowski's Llama-3.3-70B-Instruct-GGUF repo publishes it: huggingface-cli download bartowski/Llama-3.3-70B-Instruct-GGUF --include "*IQ2_XS*" --local-dir ./models.
3. Load fully onto the GPU. ./build/bin/llama-server -m ./models/Llama-3.3-70B-Instruct-IQ2_XS.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080. The -ngl 99 flag forces all layers to GPU; -c 4096 keeps the KV cache small enough to fit alongside the ~21 GB of weights.
4. Watch VRAM headroom. Run nvidia-smi during load. You want ~1–2 GB free. If it OOMs, drop context to -c 2048 or add --cache-type-k q8_0 --cache-type-v q8_0 to shrink the KV cache.
5. Query it. The server exposes an OpenAI-compatible endpoint at http://localhost:8080/v1/chat/completions.

Verdict on Path B: the best single-card compromise for interactive use. Accept that 2-bit weights measurably degrade reasoning and coding accuracy versus Q4 — for those tasks a 4-bit 32B-class model on the same 4090 often beats a 2-bit 70B.

Performance: What Speed to Actually Expect

Numbers below are single-token generation throughput measured on a 4090 with DDR5-6000 system RAM and a short prompt. Your mileage shifts with RAM speed, context length, and background load.

Configuration	On-GPU weights	Prompt eval	Generation	First-token latency
Q4_K_M, Ollama offload	~55%	~40 tok/s	~2–4 tok/s	1.5–3 s
IQ3_XXS, partial offload	~85%	~120 tok/s	~6–9 tok/s	~1 s
IQ2_XS, all-GPU	100%	~600 tok/s	~15–22 tok/s	<0.5 s
Q4_K_M, dual 4090	100%	~700 tok/s	~16–20 tok/s	<0.5 s

The dual-4090 row is the punchline: two cards holding Q4_K_M entirely in VRAM deliver both full quality and the same interactive speed as a single-card 2-bit build. If you are pricing a purchase rather than working with hardware you already own, model that second card against a used 48 GB workstation GPU using our cost calculator before committing.

Tuning & Common Mistakes

• Don't leave context at 128K. Llama 3.3 supports a long context window, but a large KV cache eats the exact VRAM you need for weights. Start at 4K and raise only if a task demands it.
• Quantize the KV cache with q8_0 in llama.cpp to reclaim ~1–1.5 GB — often the difference between all-GPU and one spilled layer.
• Ignore num_gqa tweaks copied from old blog posts. Modern Ollama and llama.cpp read grouped-query attention from the GGUF metadata; hand-set values silently break inference.
• Close other VRAM consumers. A browser with hardware acceleration or a desktop compositor can hold 1–2 GB — enough to force an OOM on the IQ2_XS path.
• Re-benchmark after driver updates. CUDA 12.x point releases have shifted 70B throughput by 10–15% in both directions.

Once you have a working endpoint, you can plug it into automated comparisons: BestLLMfor publishes an open public API under CC BY 4.0 and an open-source MCP server so you can pull our quantization and throughput data straight into your own tooling. Cross-reference your measured tok/s against our benchmarks to confirm your setup is healthy.

Verdict — Should You Run 70B on One 4090?

Yes, with clear eyes about the trade-off. A single 4090 gives you either full quality at unusable speed, or usable speed at reduced quality — not both. Match the path to the job:

Your situation	Recommended setup	Expected speed	Editorial verdict
Offline/batch, quality is king	Q4_K_M + Ollama offload	2–4 tok/s	Acceptable
Interactive chat on one card	IQ2_XS via llama.cpp	15–22 tok/s	Best single-card option
Coding/reasoning on one card	4-bit 32B-class model instead	30–45 tok/s	Often the smarter buy
You can add a second GPU	Dual 4090, Q4_K_M all-GPU	16–20 tok/s	Recommended

Our position: if 70B-class quality is non-negotiable and you want an interactive experience, a second 24 GB card is the correct investment — a 2-bit squeeze onto one 4090 sacrifices the very reasoning quality that makes the 70B worth running. For a single-card owner who wants speed today, a 4-bit 32B model frequently outperforms a 2-bit 70B on real tasks. Browse the full field in our model catalog and see how the tiers stack up in our local-LLM guides before you buy more hardware.