We Tracked Llama vs Qwen vs Mistral for 90 Days — The Data

Methodology: what 90 days of tracking actually measured

Between 14 February and 14 May 2026 the editorial team ran a fixed rotation of three open-weight model families through an identical evaluation harness. Every run logged tokens/sec, time-to-first-token (TTFT), peak VRAM, wall-clock energy from a calibrated Poniie PN2000 meter, and a per-task quality score derived from a held-out rubric of 412 prompts.

The harness covered six workload classes: Python code generation, SQL refactoring, RAG over a 90MB technical corpus, structured JSON extraction, multi-turn chat coherence, and 128K-token needle-in-haystack retrieval. Each model ran every task three times across the tracking window — 47 distinct quantized builds in total, 1.21 million completion tokens generated. Raw logs and the scoring rubric are published under CC BY 4.0 through the BestLLMfor public API, and the orchestration script is the same one shipped with our MCP server.

Reference inference stack: llama.cpp build b4187 with CUDA 12.6, vLLM 0.7.3 for batched runs, and Ollama 0.5.11 for the consumer-tier comparison. Quantization was held at Q4_K_M for all 30B+ checkpoints unless otherwise stated. Full methodology lives at /methodology/.

The cohort: which exact builds were tracked

Family snapshots were locked at the start of the window. Mid-quarter point releases were tested in parallel but did not replace the primary tracked build, to keep the data series comparable.

Cards and weights pulled from meta-llama, Qwen, and mistralai on HuggingFace. Test hardware was an RTX 5090 (32GB) paired with 96GB DDR5; large MoE checkpoints overflowed to CPU via llama.cpp's --n-gpu-layers tuning.

Throughput: tokens per second, day by day

Throughput was the most stable metric across the 90 days — variance under 4% week-to-week once driver versions stabilized in March. Numbers below are medians across 90 daily runs of a 512-token completion at temperature 0.2.

Magistral Small was the throughput champion, which is unsurprising for a 24B dense model on a single high-end GPU. The interesting result is Qwen3-Coder 32B beating Llama 4 Scout by 42% on decode despite being a dense model with no MoE routing — Scout's offload penalty dragged its real-world numbers below what the spec sheet suggests.

Quality: where each family actually wins

Quality is the metric that most public leaderboards bungle, so the rubric was deliberately narrow: each prompt had a deterministic ideal output, and scoring was binary (pass/fail) augmented by a partial-credit ladder for code that compiled but failed tests. Scores below are pass-rates over the 90-day window.

The headline: Qwen wins code, Llama wins long context, Mistral wins everything in the middle. The 128K retrieval gap for Llama 4 Scout is consistent with the architecture work described in the ATOM Report on open-model ecosystem trends — Meta's iRoPE positional scheme materially helps recall past 64K tokens.

Cost of ownership over the full 90 days

Power draw was logged continuously. The cohort drew between 240W (Magistral Small idle-to-idle) and 612W (Llama 4 Maverick under sustained generation) at the wall. Multiply by 90 days of mixed workload — roughly 6 hours/day of active inference — and the numbers stop being theoretical.

For workloads that fit on a single GPU without offload, local inference cleared an 8x cost advantage over hosted equivalents across the quarter. The crossover point is roughly 4 hours/day of active use — anything below that and a managed API endpoint wins on TCO. Run your own numbers through the cost calculator with your local electricity rate before committing to a build.

Stability and drift: the data nobody publishes

Across 90 days, only one family shipped a point release that meaningfully changed our scores: Qwen pushed an updated 32B-Coder checkpoint on 3 April 2026 that improved SQL pass-rate by 2.7 points without regressing other metrics. Mistral's mid-quarter Magistral patch was performance-neutral. Llama 4 had no weight updates in the window, though llama.cpp's MoE kernel improvements between b4090 and b4187 added roughly 11% decode throughput for free.

Model selection is not just a benchmark question — it is a release-cadence question. Apache-2.0 ecosystems iterate faster than restricted-license ones, and that compounds over a quarter.

The verdict: what to actually run

After 90 days the recommendation is workload-specific, not family-specific. Anyone telling you there is one best open-weight model in May 2026 is selling something.

Read the full methodology before generalizing these numbers to a different hardware tier. Our team is also tracking DeepSeek V4 and GLM-4.6 in parallel — those results will land in a follow-up.