Guide · 2026-07-31

State of Local LLMs — July 2026 Monthly Report

Last updated 2026-07-31

Our July 2026 read on the local LLM landscape: the models worth running, the benchmarks that moved, and the hardware that earns its price.

By Mohamed Meguedmi · 9 min read

The July 2026 landscape at a glance

The mid-2026 model generation has settled. Three families dominate open-weight local inference this month: Qwen 3.6 (Alibaba), Gemma 4 (Google), and DeepSeek V4. Each targets a different memory budget, and for the first time in a year the "best" model genuinely depends on your VRAM tier rather than a single leaderboard-topping checkpoint.

The headline shift from Q2 is efficiency, not raw capability. A 27B dense model in July 2026 now matches the MMLU and coding scores of 70B-class models from early 2025, and it does so inside 24 GB of memory at a 4-bit quantization. That collapses the hardware bar for "frontier-adjacent" local performance to a single consumer GPU.

For our full ranked list and per-model spec sheets, see the model catalog. Every score below is reproducible against our published methodology.

Model rankings this month

We rank by a blended score across reasoning, coding, multilingual, and instruction-following, weighted for local (quantized) deployment rather than cloud FP16 serving. The table reflects the quantization most users will actually run.

Rank	Model (quant)	Params	MMLU-Pro	HumanEval+	Min VRAM	Best for
1	DeepSeek V4 (Q4_K_M)	236B MoE / 37B active	78.4	84.1	512 GB tier	Absolute quality
2	Qwen3.6 27B (Q4_K_M)	27B dense	72.9	81.7	24 GB	Single-GPU sweet spot
3	DeepSeek V4-Distill 30B (Q4_K_M)	30B dense	71.2	80.3	24 GB	Reasoning on one card
4	Gemma 4 12B (Q4_K_M)	12B dense	66.8	74.5	16 GB	All-round / multilingual
5	Qwen3.6-Coder 14B (Q5_K_M)	14B dense	63.1	82.9	16 GB	Coding-focused
6	Phi-4-mini 4B (Q8_0)	3.8B dense	58.7	67.2	8 GB	Laptops / edge

Scores are indicative July-2026 blended results; the live figures update on our benchmarks hub. All benchmark data is also available through the free BestLLMfor public API (CC BY 4.0) and our open-source MCP server, so you can pull these numbers directly into your own tooling.

Benchmarks — what actually changed

Two things moved this month. First, DeepSeek V4 displaced last quarter's leader on reasoning-heavy suites, posting a 78.4 on MMLU-Pro — the first open-weight model to close the gap with the strongest closed frontier models on that benchmark. The gains come from its refined mixture-of-experts routing; see the official DeepSeek V4 model card for the architecture details.

Second, Qwen3.6 quietly became the most downloaded local model of the month. The 27B checkpoint's HumanEval+ score of 81.7 is remarkable for a dense model that fits a single 24 GB card. Alibaba's release notes on the Qwen3.6 27B card document a training run tuned specifically for quantization robustness — and it shows: the Q4_K_M quality drop versus FP16 is under 1.5 points on our runs, where older models lost 3–4.

Verdict: if you have exactly one 24 GB GPU, Qwen3.6 27B is the model to run this month. It is the best quality-per-gigabyte checkpoint we have measured.

Gemma 4 12B did not top any single category but remains the model we recommend most often. It is the best-behaved instruction follower in the 12–16 GB tier, ships day-one on ollama.com/library/gemma4, and handles non-English workloads better than anything else at its size.

Hardware & VRAM tiers

Match the model to the memory, not the other way around. The table below maps each budget tier to the model we recommend and the realistic quantization for it.

Memory tier	Representative hardware	Recommended model (quant)	Tokens/sec (approx)
8 GB	RTX 4060 / M-series 8 GB	Phi-4-mini 4B Q8_0	45–60
16 GB	RTX 4070 Ti Super / M4 16 GB	Gemma 4 12B Q4_K_M	30–42
24 GB	RTX 4090 / RTX 5080 24 GB	Qwen3.6 27B Q4_K_M	28–38
48 GB	2× 24 GB / RTX 6000-class	DeepSeek V4-Distill 30B Q5_K_M	22–30
512 GB+	Multi-GPU / unified-memory servers	DeepSeek V4 full Q4_K_M	12–20

Throughput figures assume single-stream generation with a warm KV cache and are directional, not guarantees. The 24 GB tier is the clear value inflection point: it unlocks 27–30B dense models that were multi-GPU-only twelve months ago, and used 24 GB cards have fallen roughly 8% in street price since May 2026.

For anything below 16 GB, accept that you are running small models. Phi-4-mini is genuinely useful for autocomplete, summarization, and structured extraction, but it is not a reasoning engine. Don't buy an 8 GB card expecting frontier behavior.

Cost of ownership

Hardware price is only the entry fee. Electricity, and the opportunity cost versus cloud API calls, decide whether local inference actually pays off. The rough five-year math for the 24 GB tier this month:

Line item	Estimate (USD)
Used 24 GB GPU	$900–$1,300
Supporting components (amortized)	$600–$900
Power @ 350W, 4 hrs/day, $0.16/kWh, 5 yrs	~$410
Five-year total	~$1,900–$2,600

Against metered cloud inference, local pays for itself fastest for high-volume, privacy-sensitive, or always-on workloads. For occasional use, an API is cheaper. Run your own numbers in the cost calculator before committing to hardware — the break-even point is heavily sensitive to your daily token volume and local power tariff.

Tooling updates

The runtime landscape consolidated further this month. Three tools cover almost everyone:

• Ollama — still the fastest path from zero to a running model. Day-one support for Gemma 4 and Qwen3.6, clean model management, good default quantizations.
• LM Studio — best GUI for non-terminal users; strong for prompt experimentation and comparing quantizations side by side.
• vLLM — the choice for throughput and concurrent serving. If you're standing up an internal endpoint for a team, this is the engine.

llama.cpp remains the foundation underneath much of this and the right tool if you need maximum control over quantization and offload. Everything else — Jan, GPT4All, and the long tail — is now niche rather than essential.

How to run Qwen3.6 27B this month

The fastest route to our top single-GPU pick, on a 24 GB card:

1. Install Ollama for your OS from the official site.
2. Pull the model: run ollama pull qwen3.6:27b in your terminal.
3. Confirm the quantization fits — the default Q4_K_M lands near 17 GB of VRAM, leaving headroom for context.
4. Start a session with ollama run qwen3.6:27b and verify tokens/sec sit in the 28–38 range.
5. For coding-heavy work, swap to qwen3.6-coder:14b to free memory for a larger context window.

If throughput matters more than convenience, serve the same weights through vLLM instead and expose an OpenAI-compatible endpoint for your applications.

Conclusion — the July 2026 verdict

This month rewards matching the model to your memory rather than chasing a single leaderboard winner. The 24 GB tier is where local inference makes the most sense right now, and Qwen3.6 27B is the model that makes it worthwhile.

Your situation	Run this
One 24 GB GPU	Qwen3.6 27B Q4_K_M
16 GB card or Apple silicon	Gemma 4 12B Q4_K_M
Coding-first, 16 GB	Qwen3.6-Coder 14B Q5_K_M
Maximum quality, big memory	DeepSeek V4 (full)
Laptop / edge, 8 GB	Phi-4-mini 4B Q8_0

We refresh this report monthly. For the full ranked catalog and our top per-use-case picks, see the best-of guides, and check the benchmarks hub for live scores between issues.