State of Local LLMs — June 2026 Monthly Report
Last updated 2026-07-30
MoE-plus-4-bit is now the default, 128GB unified memory is the sweet spot, and Qwen3.6 leads the pack. Here's June 2026, by the numbers.
By Mohamed Meguedmi · 9 min read
Key takeaways
- MoE + 4-bit is the default. Nearly every flagship shipped in Q2 2026 is a mixture-of-experts model quantized to Q4_K_M or better, cutting active parameters (and VRAM) by 60-80% versus dense equivalents.
- Qwen3.6-35B-A3B is the general-purpose winner. With only ~3B active parameters it runs at 40+ tok/s on a 32GB Apple M-series and beats last year's 72B dense models on MMLU-Pro.
- 128GB unified memory is the new reference point. It fits a 100B-class MoE at 4-bit with room for a 256K context, and used prices have fallen below $2,500.
- Speculative decoding is finally practical. Paired draft models now deliver a reliable 1.8-2.4x throughput gain in llama.cpp and vLLM without quality loss.
- The registry war is over: OCI won. Model files now ship like container images, and llms.txt adoption hit 8.7% of the top 1,000 sites.
Welcome to the June 2026 edition of our monthly local-LLM report. The signal this month is consolidation rather than shock: no single release reset the field, but the compounding effect of mixture-of-experts (MoE) architectures, better 4-bit quantization, and mature speculative decoding has quietly moved the frontier of what a single developer can run at home or in a small business. Below we cover the models that matter, the hardware to buy, the inference tricks worth enabling, and a clear verdict on the one setup we recommend today.
What actually changed in June 2026
Three structural shifts defined the month. First, active-parameter counts kept shrinking even as capability rose. The headline example, Qwen3.6-35B-A3B, carries 35B total parameters but activates only ~3B per token, so it behaves like a 3B model on your memory bus and a 30B-class model on the benchmark sheet. Second, quantization stopped being a compromise. Modern Q4_K_M and the newer AWQ/GPTQ variants now land within 1-2 points of full-precision on MMLU-Pro, making FP16 weights a niche concern for most local users. Third, the tooling around packaging and distribution standardized on OCI registries, so pulling a model increasingly feels like pulling a Docker image.
For context on how far the floor has risen: a 7B-class model scored around 64% MMLU in early 2024 (Mistral Small) and now clears 74% (per community trackers such as PromptQuorum's 2026 tracker). The 70B class moved from 75% to the 82-84% range. Those are dense numbers; the MoE models discussed here beat them at a fraction of the active compute. Our own normalized scores are published on the benchmarks hub, and the raw data is available through the free BestLLMfor public API (CC BY 4.0) and our open-source MCP server if you want to pull it into your own pipeline.
The models that matter this month
We track over 4,500 curated and community entries, but only a handful are worth your download bandwidth in June 2026. The table below shows normalized results at the quantization most people will actually run.
| Model (quant) | Total / Active params | MMLU-Pro | Min VRAM/URAM | Best for |
|---|---|---|---|---|
| Qwen3.6-35B-A3B Q4_K_M | 35B / 3B | 71.4 | ~22 GB | General-purpose daily driver |
| Qwen3-Coder-Next 32B Q4_K_M | 32B / 4B | 68.9 | ~20 GB | Coding, agentic tool use |
| Gemma 4 12B Q4_K_M | 12B dense | 63.2 | ~9 GB | 16GB laptops, low latency |
| Nemotron 3 Nano Omni Q4 | 9B dense | 58.1 | ~7 GB | Audio + vision + text edge use |
| Llama 4 Scout (100B-A17B) Q4 | 100B / 17B | 73.8 | ~62 GB | Long-context (10M) analysis |
Qwen3.6-35B-A3B is the clear general-purpose recommendation. The A3B active footprint is what makes it special: you get near-frontier reasoning at a memory and throughput profile that a mid-range machine handles comfortably. For coding, Qwen3-Coder-Next 32B has displaced last cycle's favorites on agentic and fill-in-the-middle tasks; you can see the full model card on Qwen's HuggingFace org. If your workload is document analysis over enormous inputs, Llama 4 Scout's 10M-token context remains unmatched among openly available weights, though the memory cost is real.
Hardware: 128GB unified memory is the reference class
The most important hardware story of 2026 is the normalization of large unified-memory machines. A 128GB unified-memory system fits a 100B-class MoE at 4-bit with a generous context window and no discrete GPU required. The table below compares the realistic options a US/UK/AU buyer faces this month.
| Setup | Memory | Approx. cost (USD) | Qwen3.6-35B tok/s | Largest comfortable model |
|---|---|---|---|---|
| Apple M-series, 32GB URAM | 32 GB | $1,600 | ~42 | 35B MoE @ Q4 |
| Apple M-series, 128GB URAM | 128 GB | ~$2,450 (used) | ~55 | 100B MoE @ Q4 |
| Single RTX 5090, 32GB VRAM | 32 GB | $2,000+ | ~90 | 35B MoE @ Q4 |
| Dual 24GB GPU (48GB pooled) | 48 GB | $1,800 (used) | ~110 | 70B dense @ Q4 |
The verdict on hardware: if raw tokens-per-second is your priority and your target model fits in 32GB, a single modern GPU still wins on throughput. But for the widest range of models per dollar — and near-silent, low-power operation — a 128GB unified-memory machine is the most future-proof choice at under $2,500 on the used market. Run your own numbers with the cost calculator before buying; electricity and resale value shift the answer by region.
Inference tech: quantization and speculative decoding
Two techniques crossed the practicality threshold this year. Quantization to 4-bit is no longer a quality trade you agonize over — Q4_K_M is the sane default, and you only reach for Q5/Q6 or FP16 when a task is provably sensitive. The memory savings compound with MoE: a 35B model at Q4 with a 3B active path is trivially servable on consumer hardware.
Speculative decoding is the second win. By pairing a small, fast draft model with the full target model, llama.cpp and vLLM now deliver a consistent 1.8-2.4x throughput improvement with no measurable quality loss when the draft is well matched. The technique has existed for years, but June's tooling finally makes it a one-flag enable rather than a research project; the underlying method is described in the original speculative decoding paper. Combined with MoE routing, the effective cost per useful token on local hardware has roughly halved year over year.
Coding and multimodal: the specialists
Two categories deserve their own note. On the coding side, Qwen3-Coder-Next 32B and Cohere North Mini Code lead openly available options for agentic workflows; both handle tool-calling and long repository context far better than general chat models. If you pair one with speculative decoding, interactive latency drops to the point where local autocomplete feels competitive with hosted services.
On the multimodal side, the community's own benchmarking (see the r/LocalLLaMA June VLM roundup) puts the Gemma 4 vision variants and Qwen 3.5/3.6 VL models at the front for image understanding, while Nemotron 3 Nano Omni is the pick when you need audio, vision, and text in one small package at the edge. We maintain per-task rankings for these in the model catalog.
How to run Qwen3.6-35B-A3B today
The fastest path from zero to a running model this month is Ollama with an OCI-backed pull. Here is the minimal sequence.
- Install the runtime from ollama.com/download for your OS.
- Pull the quantized model:
ollama pull qwen3.6:35b-a3b-q4_K_M - Start an interactive session:
ollama run qwen3.6:35b-a3b-q4_K_M - For throughput, enable speculative decoding by attaching a small draft model in your server config, then serve over the local API on port 11434.
ollama pull qwen3.6:35b-a3b-q4_K_M
ollama run qwen3.6:35b-a3b-q4_K_M "Summarize this month's local LLM trends."That is genuinely the whole setup on a 32GB machine. For the 100B-class Llama 4 Scout, swap the tag and ensure you have 64GB+ of memory free.
Verdict
June 2026 rewards the buyer who stops chasing dense parameter counts and embraces MoE at 4-bit. Our recommendation is unambiguous.
| Need | Pick | Why |
|---|---|---|
| One model for everything | Qwen3.6-35B-A3B Q4_K_M | Near-frontier quality at a 3B active footprint |
| Coding / agents | Qwen3-Coder-Next 32B Q4_K_M | Best openly available agentic coder |
| Best hardware value | 128GB unified memory (used, <$2,500) | Runs 100B MoE, silent, future-proof |
| Max throughput on a budget | Single 32GB GPU + spec. decoding | 90+ tok/s on 35B MoE |
| Long-context analysis | Llama 4 Scout (100B-A17B) Q4 | 10M-token window, unmatched openly |
Compare these head-to-head on our best-of rankings, and pull the underlying scores yourself via the free public API or MCP server. We'll be back with the July report at the end of next month.
FAQ
What is the single best local LLM in June 2026?
For general-purpose use, Qwen3.6-35B-A3B at Q4_K_M. Its mixture-of-experts design activates only ~3B parameters per token, so it runs on a 32GB machine at 40+ tok/s while scoring like a much larger dense model.
Why does everyone recommend 128GB unified memory now?
Because it fits a 100B-class MoE model at 4-bit with a large context window and no discrete GPU, at a used price now below $2,500. It is the widest model range per dollar and runs near-silent at low power.
Is 4-bit quantization good enough for serious work?
Yes for most tasks. Modern Q4_K_M lands within 1-2 points of full precision on MMLU-Pro. Only reach for Q5/Q6 or FP16 on workloads you have specifically measured as quantization-sensitive.
How much faster is speculative decoding really?
A consistent 1.8-2.4x throughput gain in llama.cpp and vLLM when the draft model is well matched to the target, with no measurable quality loss. In June 2026 it is a one-flag enable rather than a research exercise.
Where can I get the raw benchmark data?
Through the BestLLMfor public API (CC BY 4.0) and our open-source MCP server, or on the benchmarks hub. See our methodology page for how scores are normalized.
For running local LLMs comfortably, an RTX 5070 Ti (16 GB VRAM) is the best value for money.
Amazon Check RTX 5070 Ti price →As an Amazon Associate, BestLLMfor earns from qualifying purchases, at no extra cost to you. It does not influence our independent rankings.