Guide · 2026-05-16

Why VRAM Matters More Than TFLOPS for LLM Inference

Inference is memory-bound, not compute-bound. Here is why VRAM and bandwidth determine your tokens-per-second far more than peak TFLOPS — and how to budget hardware accordingly.

By Mohamed Meguedmi · 8 min read

Marketing decks for GPUs lead with TFLOPS. Reviewers benchmark them with FP16 GEMMs. Then someone tries to run Qwen3-Coder 32B Q4_K_M locally on an RTX 5070 with 12 GB of VRAM and discovers the model does not even load. A weights file that fits comfortably on a five-year-old RTX 3090 at 24 GB refuses to run on a card with 30% more raw compute. Welcome to the central truth of local LLM inference: VRAM is the gatekeeper, bandwidth is the throttle, and TFLOPS is the tiebreaker.

This guide explains the arithmetic behind that hierarchy, presents measured throughput across consumer and datacenter GPUs, and gives a concrete VRAM-first buying framework. If you want to estimate cost-per-token for a given GPU and model directly, the BestLLMfor cost calculator applies the same math automatically.

The Roofline: Why Decoding Is Memory-Bound

An autoregressive transformer reads every parameter exactly once to produce a single output token. There is no weight reuse within a decoding step — each new token requires the entire model to stream from VRAM to the GPU's compute units. The relevant measure is arithmetic intensity: the ratio of floating-point operations performed to bytes moved from memory.

For batch-size-1 decoding, arithmetic intensity sits around 1–2 ops/byte. Modern GPUs need roughly 100–300 ops/byte to fully utilize their tensor cores. The gap is structural, not a tuning problem. The roofline model — formalized by Williams, Waterman, and Patterson in their 2009 Communications of the ACM paper — predicts that under these conditions, performance is bounded entirely by memory bandwidth.

Concretely: a 70B parameter model in FP16 weighs 140 GB. Generating one token requires moving all 140 GB across the memory bus. An H100 SXM at 3.35 TB/s of HBM3 bandwidth has a hard ceiling of 3350 ÷ 140 ≈ 24 tokens/second, no matter how many TFLOPS sit idle. An RTX 4090 (1008 GB/s) running the same model at Q4_K_M (~40 GB) tops out at roughly 1008 ÷ 40 ≈ 25 t/s — and in practice, kernel overheads, KV-cache reads, and synchronization shave that to 15–18 t/s.

VRAM Is a Hard Wall, Not a Soft Limit

Compute deficits cost throughput. VRAM deficits cost binary access. Try to load a model even 1 GB larger than your card's free VRAM and one of three things happens:

1. The runtime refuses to load (vLLM, TensorRT-LLM, and PyTorch default behavior).
2. Layers are offloaded to system RAM via PCIe — at ~32 GB/s for PCIe 5.0 x16, that is 30× slower than a 1 TB/s GPU bus.
3. The runtime crashes mid-generation with an OOM exception.

Measured impact: llama.cpp running Qwen3-Coder 32B Q4_K_M (19 GB) on a fully-resident 24 GB RTX 3090 produces ~38 tokens/sec. The same model on a 16 GB RTX 4060 Ti, with 3 GB spilled to system RAM via --n-gpu-layers, drops to 3.4 t/s. That is a 91% throughput loss for a 16% VRAM shortfall.

This non-linearity is why VRAM acts as the first hardware filter, not a graded one. See the BestLLMfor benchmark methodology for how we measure these spillover breakpoints across model and context combinations.

Bandwidth Sets the Tokens/Sec Ceiling

Once a model fits in VRAM, throughput tracks memory bandwidth with near-linear fidelity. The table below shows measured single-stream decoding throughput for Llama-3.1 8B Q4_K_M (~4.9 GB) across consumer and prosumer GPUs, alongside their bandwidth and dense FP16 TFLOPS.

GPU	VRAM	Bandwidth	FP16 TFLOPS	Llama-3.1 8B Q4 t/s
RTX 3090	24 GB	936 GB/s	35.6	112
RTX 4080 Super	16 GB	736 GB/s	52.2	89
RTX 4090	24 GB	1008 GB/s	82.6	128
RTX 5090	32 GB	1792 GB/s	104.8	205
RTX 4060 Ti 16 GB	16 GB	288 GB/s	22.1	35
Apple M3 Max (40-core)	128 GB UMA	400 GB/s	~14	48
NVIDIA H100 SXM	80 GB	3350 GB/s	989	352

Note the disconnect. The RTX 4080 Super has 47% more TFLOPS than an RTX 3090 but 21% less bandwidth, and produces 21% fewer tokens/sec. The H100 has 28× the TFLOPS of an RTX 3090 but only 3.6× the bandwidth — and delivers 3.1× the throughput. Bandwidth predicts; TFLOPS does not.

The KV Cache: The Invisible VRAM Consumer

Discussions of "VRAM needed for a 70B model" usually quote the weight footprint and stop. That is wrong for any non-trivial context length. The key-value cache — the running per-token attention state — scales linearly with sequence length and can dwarf the weights themselves.

For Llama-3.1 70B at FP16 KV with grouped-query attention (8 KV heads, 80 layers, 128 head_dim):

• 8K context: ~2.1 GB
• 32K context: ~8.4 GB
• 128K context: ~33.5 GB

At 128K context, the KV cache is nearly as large as a Q4-quantized 70B model. This is why production deployments quantize the KV cache to FP8 or Q4 (vLLM, TensorRT-LLM, and recent llama.cpp builds all support it) and why long-context workloads need either H200-class memory or aggressive cache compression.

VRAM Sizing by Model Class

The table below gives practical VRAM targets including KV-cache headroom at 8K context. For longer contexts, add the KV-cache delta from the previous section.

Model class	FP16 weights	Q4_K_M / AWQ Int4	Recommended VRAM (8K ctx)	Realistic GPU
7B–8B	14–16 GB	4.5–5.5 GB	12 GB	RTX 4070, M2 Pro 16 GB
13B	26 GB	8 GB	16 GB	RTX 4060 Ti 16 GB
32B	64 GB	19 GB	24 GB	RTX 3090 / 4090 / 5090
70B	140 GB	40 GB	48 GB	2× 3090, RTX 6000 Ada, A100 40 GB
120B–180B	240–360 GB	70–110 GB	96–128 GB	H100 80 GB, dual A100, M3 Ultra 192 GB
405B	810 GB	230 GB	256 GB+	4× H100, 2× H200, MI300X cluster

Source data is refreshed quarterly from Meta's official HuggingFace cards and the Ollama model library. The raw measurements are exposed through the BestLLMfor public API under CC BY 4.0, and through the open-source quelllm-mcp server for direct LLM-agent consumption.

Quantization: Trading TFLOPS for VRAM and Winning Anyway

Because inference is memory-bound, quantization is essentially free performance. Cutting weight precision from FP16 to Int4 reduces VRAM by ~3.8× and increases throughput by roughly the same factor — the GPU is simply moving fewer bytes per token. The TFLOPS hit from dequantization kernels is negligible at batch-1 decoding, since the compute units were idle anyway.

Modern quantization formats and their typical perplexity penalty on Llama-class models, per the published llama.cpp quantization tables:

• Q8_0 / FP8: ~0.1% PPL increase, 2× VRAM savings. Use when VRAM is comfortable.
• Q5_K_M: ~0.5% PPL increase, 3.1× savings. Sweet spot for quality-sensitive work.
• Q4_K_M / AWQ Int4: 1–2% PPL increase, 3.8× savings. Default for local inference.
• Q3_K_M / Int3: 3–5% PPL increase, 4.8× savings. Coding and reasoning tasks degrade noticeably.
• Q2_K: 8–15% PPL increase. Only when nothing else fits.

The Buying Framework: VRAM-First Hardware Decisions

Translate the above into a purchase order:

1. Identify the largest model you actually need to run. Be honest. Most production agent workflows run well on 8B–32B class models, and a 24 GB card serves them at high throughput.
2. Add KV-cache headroom for your target context. For 32K+ contexts on a 70B model, plan for 48–64 GB.
3. Pick the highest-bandwidth GPU that meets the VRAM requirement. Prefer GDDR6X or HBM over GDDR6, and never trade VRAM for TFLOPS at the same price point.
4. Only after VRAM and bandwidth are satisfied, optimize for FP8/FP4 tensor-core support (RTX 4090+ for FP8, RTX 5000-series and H100+ for FP4) if you plan to use those formats at scale.

For a deeper look at the editorial framework behind these recommendations, see about BestLLMfor. French-language readers can find the same analysis on our sister site quelllm.fr.

The Verdict

Workload	Minimum VRAM	Bandwidth target	Best value pick (2026)
Single-user 7B–8B chat	12 GB	500 GB/s	RTX 4070 Super 12 GB
Local coding agent, 32B Q4	24 GB	900 GB/s	Used RTX 3090 or new RTX 5080
70B Q4 self-hosted assistant	48 GB	1 TB/s	2× RTX 3090 or RTX 6000 Ada 48 GB
Long-context (128K) 70B	96 GB	3 TB/s	H100 80 GB + KV quantization
Frontier 120B+ MoE	128 GB	3 TB/s+	Apple M3 Ultra 192 GB or H200 141 GB

The bottom line: TFLOPS sell GPUs to gamers. VRAM and bandwidth run LLMs. If a vendor leads their pitch with peak compute numbers and buries the memory specs, assume the card was not designed for inference and price it accordingly.