Guide · 2026-08-01

State of Local LLMs — August 2026 Monthly Report

Last updated 2026-08-01

Our monthly read on which open-weight models, quants, runtimes and hardware are worth your money right now — with numbers, not vibes.

By Mohamed Meguedmi · 8 min read

What actually changed in August 2026

Three things moved the needle this month. First, the Qwen3.5 refresh shipped improved instruction-following and a longer default context (128K) across the 8B/14B/32B dense line, with day-one GGUF quants on Hugging Face. Second, Gemma 4 quantization-aware training (QAT) weights matured to the point where the Q4_0 build loses almost nothing versus FP16 on everyday tasks — a meaningful win for 16GB owners. Third, Ollama pushed structured-output and tool-calling improvements to stable, closing much of the gap that previously sent power users to vLLM.

None of this is revolutionary on its own, but the cumulative effect is that the "good enough to replace a cloud call" bar is now clearly met on hardware most developers already own. If you benchmarked local models a year ago and walked away unimpressed, August 2026 is a fair time to re-test. Our full model matrix lives in the catalog, and every score below is reproducible from our methodology.

Best open-weight models by VRAM tier

The table below is our current recommendation per tier for a general-purpose local assistant, using the quant we'd actually run. Scores are MMLU-Pro (5-shot) and HumanEval pass@1 from our August test pass; treat them as directional, not absolute.

VRAM	Recommended model + quant	Params (active)	MMLU-Pro	HumanEval	Best for
8 GB	Qwen3.5-8B Q4_K_M	8B	58.1	71.3	Chat, RAG, light coding
12 GB	Gemma 4 12B Q4_K_M	12B	62.4	68.9	Balanced assistant
16 GB	Gemma 4 27B Q4_0 (QAT)	27B	67.0	74.5	General default pick
24 GB	Qwen3.5-32B Q4_K_M	32B	71.8	82.1	Coding + reasoning
24 GB	GLM-5-32B Q4_K_M	32B	71.2	80.4	Agentic / tool use
48 GB+	Qwen3-235B-A22B Q4_K_M	235B (22B)	78.9	86.7	Frontier local reasoning

Two verdicts. For most people the winner is unglamorous: Gemma 4 27B at Q4_0 on a 16GB card is the best effort-to-reward ratio in the ecosystem right now. If you have 24GB, Qwen3.5-32B Q4_K_M is the coding and reasoning champion and the model we'd default to for daily developer work. See the model card on Hugging Face for the full quant list and license terms.

The MoE inflection point

Mixture-of-experts is why a 235B-parameter model is suddenly plausible on a well-specced desktop. Qwen3-235B-A22B activates only ~22B parameters per token, so throughput feels like a 22B dense model even though quality tracks something far larger. The catch is memory footprint: the Q4_K_M weights are ~130GB, so you either need multi-GPU (2× 48GB) or you offload experts to system RAM and accept slower prompt processing.

Our take: MoE is worth chasing only if you have 128GB+ of fast system RAM or dual pro GPUs. For everyone else, a dense 32B remains the pragmatic ceiling. Llama 4 Scout occupies a similar niche — strong, but demanding — and the details are documented in Meta's official model announcements.

Runtimes: which one, and when

The runtime landscape has consolidated into clear lanes. Don't overthink the choice — match it to your workload.

• Ollama — best default for single-user desktop use. One-line pulls, sane defaults, now with solid tool-calling and structured output. Backed by ollama.com/library.
• llama.cpp — the engine underneath most GUIs. Use it directly when you want maximum quant control, CPU/GPU split tuning, or the newest GGUF features first.
• vLLM — the answer for concurrent serving and multi-GPU throughput. If you're serving a team or an app, this is the production pick.
• LM Studio — the friendliest GUI for non-terminal users; great for evaluating models before scripting anything.

Verdict: solo developer on one machine → Ollama or llama.cpp. Serving many users → vLLM. There's no prize for suffering through vLLM's config to chat with a single 8B model.

Hardware: price/performance in August 2026

VRAM is still the binding constraint, and the used market still wins on dollars-per-gigabyte. Approximate US street prices this month:

GPU	VRAM	Approx. price (USD)	$/GB VRAM	Tier it unlocks
RTX 5060 Ti	16 GB	$429	$26.8	Gemma 4 27B Q4_0
RTX 3090 (used)	24 GB	$650	$27.1	Qwen3.5-32B Q4_K_M
RTX 5080	16 GB	$999	$62.4	Fast 27B, tight context
RTX 4090 (used)	24 GB	$1,500	$62.5	32B + long context
RTX 5090	32 GB	$1,999	$62.5	32B headroom / small MoE
Apple M4 Max (128GB)	~110 GB usable	$3,700	$33.6	MoE via unified memory

The standout remains the used RTX 3090: at roughly $650 it delivers a full 24GB and comfortably runs our top 32B pick. For a clean-slate 16GB build, the RTX 5060 Ti reset expectations at the entry level. And for MoE-curious buyers, an Apple M4 Max with 128GB of unified memory is the cheapest way into 200B-class models, at the cost of prompt-processing speed. Model these scenarios against your electricity rate in our cost calculator.

Benchmarks that matter now

Raw MMLU is played out. In August 2026 we weight four things: MMLU-Pro (harder reasoning), HumanEval / LiveCodeBench (real coding), tool-calling accuracy (agentic reliability), and long-context recall (does 128K actually work, or just load?). The gap between advertised and usable context is where many models still disappoint — several 128K-tagged releases degrade sharply past 32K in needle-in-haystack tests. We publish the full battery on the benchmarks page, and every number is queryable through the BestLLMfor public API (CC BY 4.0) or the open-source MCP server, so you can pull live scores straight into your own tooling.

Cost of ownership: local vs cloud

The economic case for local has quietly gotten stronger. A used RTX 3090 running Qwen3.5-32B produces roughly 35–45 tokens/sec on a single stream. At typical US residential power (~$0.17/kWh) and a ~350W draw, marginal cost is a fraction of a cent per thousand tokens — effectively free at the margin once the card is paid off. The breakeven versus a metered API is now measured in weeks of heavy use, not years, for anyone generating serious volume. Privacy, offline capability, and no rate limits are the bonus.

Verdict

Your situation	Buy	Run	Runtime
Entry / first local model	RTX 5060 Ti 16GB	Gemma 4 27B Q4_0	Ollama
Best value all-rounder	Used RTX 3090 24GB	Qwen3.5-32B Q4_K_M	Ollama / llama.cpp
Coding-first developer	Used RTX 4090 24GB	Qwen3.5-32B Q4_K_M	llama.cpp
Agentic / tool use	RTX 5090 32GB	GLM-5-32B Q4_K_M	vLLM
Frontier local reasoning	Apple M4 Max 128GB	Qwen3-235B-A22B Q4_K_M	llama.cpp

Bottom line for August 2026: if you want one recommendation, it's a used 24GB card running Qwen3.5-32B on Ollama. It's the sweet spot of price, quality and simplicity, and nothing this month unseats it. Browse the alternatives in the best-of rankings if your use case is narrower.