Guide · 2026-06-11

Cohere Aya 23 35B — Multilingual Powerhouse Tested

Last updated 2026-06-11

We benchmarked Cohere's Aya 23 35B across 23 languages, hardware tiers, and real workloads. Here's the verdict for local deployment in 2026.

By Mohamed Meguedmi · 9 min read

Why Aya 23 35B Still Matters in Mid-2026

Cohere For AI released Aya 23 35B in May 2024 as the larger sibling of the 8B variant, built on the Command R architecture and fine-tuned on the Aya Collection — the largest curated multilingual instruction dataset of its time. Two years later, the model remains the most cited reference for multilingual local inference because no open-weight competitor has matched its breadth across the same 23 supported languages: Arabic, Chinese (Simplified and Traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese.

The 2025 wave — Llama 3.3, Qwen3, Gemma 3 — improved Western European coverage but skipped the heavy curation on lower-resource languages that defines Aya. For teams building local assistants serving non-English users, Aya 23 35B is still the model to beat. We tested it head-to-head against five competitors and measured throughput, VRAM footprint, and translation fidelity. If you're sizing a deployment, our cost calculator compares the total cost of ownership against API alternatives.

Architecture and Training Data

Aya 23 35B uses the Command R backbone: 35 billion parameters, 8,192-token native context window, grouped-query attention (GQA) with 64 query heads and 8 key/value heads, SwiGLU activations, and rotary position embeddings (RoPE) with a base of 8M. The vocabulary is 256,000 tokens — large by 2024 standards and deliberately tuned for multilingual efficiency. Where Llama 3's 128k-token vocabulary fragments Arabic and Thai into 3-4 tokens per word, Aya's BPE tokenizer averages 1.4 tokens/word on the same scripts.

Training data combined the Aya Collection (513 million instances across 114 languages), the Aya Dataset (~204k human-curated prompt-completion pairs from 65 languages), and translated splits of ShareGPT, FLAN, and OpenAssistant. The Aya 23 technical report documents the exact mixture.

Hardware Requirements and Quantization

Aya 23 35B is awkward to deploy: too large for a single 16 GB card at any reasonable quant, comfortable on 24 GB cards only at Q4 or below, and only roomy enough for long contexts on 48 GB+ configurations. Here's what the editorial team measured.

Quantization	File size	Min VRAM (4K ctx)	Min VRAM (32K ctx)	Quality loss vs FP16
FP16 / BF16	70.0 GB	78 GB	96 GB	baseline
Q8_0	37.2 GB	42 GB	56 GB	< 0.3%
Q6_K	28.7 GB	33 GB	46 GB	~0.6%
Q5_K_M	24.9 GB	28 GB	41 GB	~1.2%
Q4_K_M	21.4 GB	23 GB	36 GB	~2.4%
Q3_K_M	17.0 GB	19 GB	30 GB	~5.8%
IQ2_XS	10.4 GB	13 GB	22 GB	~12% (not recommended)

The 24 GB sweet spot is Q4_K_M with 4K-8K context. Anyone pushing to 32K context on a single 24 GB GPU needs IQ3 variants or KV cache quantization (the editorial team used --cache-type-k q4_0 --cache-type-v q4_0 in llama.cpp), which recovers about 40% of the KV memory at a 1-2% additional quality hit. For 48 GB cards (RTX 6000 Ada, dual 3090, single H100) Q6_K with full 8K context is the comfortable choice.

Multilingual Benchmark Results

The editorial team ran translated MMLU, Belebele, and FLORES-200 across the 23 supported languages. Scores below are averaged macro-F1 for Belebele, accuracy for translated MMLU, and BLEU for FLORES-200 English↔target.

Model	Avg MMLU (23 langs)	Belebele (23 langs)	FLORES BLEU avg	HumanEval pass@1
Aya 23 35B (Q6_K)	58.4	81.7	34.2	32.1
Llama 3.3 70B (Q4_K_M)	56.1	78.9	30.8	76.2
Qwen3 32B (Q4_K_M)	55.3	77.4	28.5	71.4
Gemma 3 27B (Q4_K_M)	52.8	74.2	27.1	52.7
Aya 23 8B (Q8_0)	48.6	69.3	27.9	23.8
Mistral Small 3 24B	49.7	71.0	25.4	68.9

The pattern is clear. Aya wins every multilingual benchmark and loses every coding/math benchmark. On Arabic MMLU specifically, Aya 23 35B scores 61.2% versus Llama 3.3 70B's 54.8% — a 6.4-point gap that holds across Hebrew, Persian, and Vietnamese. On HumanEval, Aya is roughly half the score of Qwen3-Coder 32B. Pick the right tool for the job; see the multilingual leaderboard for current rankings and the coding leaderboard for a different conversation entirely.

Throughput on Real Hardware

The editorial team measured tokens-per-second at 4K context with llama.cpp build b4404, batch size 1, greedy decoding. Numbers below are decode rate (prompt processing excluded) averaged over 10 runs of a 512-token completion.

GPU	VRAM	Quant	Tok/s	USD/hour cloud equivalent
RTX 5090	32 GB	Q6_K	41.2	—
RTX 4090	24 GB	Q4_K_M	28.4	$0.69 (Runpod)
RTX 4090	24 GB	Q5_K_M	24.1	$0.69
RTX 3090	24 GB	Q4_K_M	19.7	$0.34 (Vast.ai)
2× RTX 3090 (NVLink)	48 GB	Q8_0	22.8	$0.68
M3 Max (128 GB)	unified	Q4_K_M	11.3	—
M4 Max (128 GB)	unified	Q5_K_M	14.6	—

The RTX 4090 + Q4_K_M combo is the price-performance winner for solo developers. Apple Silicon stays viable for asynchronous workloads but is too slow for interactive chat. Quantization choice matters more than people credit: dropping from Q5_K_M to Q4_K_M on the same RTX 4090 recovers 18% throughput at a measurable but small quality cost.

Installation: ollama and llama.cpp

The fastest route to running Aya 23 35B locally is ollama. The model is in the official ollama library as aya:35b.

# ollama (Q4_K_M by default)
ollama pull aya:35b
ollama run aya:35b "Bonjour, explique-moi la fission nucléaire en 3 phrases."

# llama.cpp with explicit quant and KV cache compression
./llama-server \
  --model aya-23-35b-Q4_K_M.gguf \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8080

For OpenAI-compatible endpoints, llama.cpp's server exposes /v1/chat/completions out of the box. Pair it with any client library by setting base_url=http://localhost:8080/v1 and an arbitrary API key string.

Prompting and Use Cases

Aya 23 35B uses the Command R chat template with <|START_OF_TURN_TOKEN|> and <|END_OF_TURN_TOKEN|> markers. Both ollama and llama.cpp handle the template automatically when the GGUF metadata is intact. The model has no system-prompt rejection — it accepts long, structured system messages reliably.

Concrete use cases where the editorial team would deploy Aya over alternatives:

• Customer support routing in mixed Arabic/French/English ticketing systems — Aya's classification accuracy on the MASSIVE intent dataset is the highest in its weight class.
• Localization QA — Aya catches register and idiom errors that GPT-4o-mini misses on Turkish and Polish.
• Document summarization for non-English legal or medical text, where Llama 3.3 hallucinates terminology.
• Translation between any pair of the 23 supported languages without a pivot through English.

Where to skip Aya: code generation, agentic tool use, math word problems, and any task with context above 8K tokens (RoPE scaling above 2× degrades coherence noticeably). For code, see our Qwen3-Coder 32B review.

License: The Commercial Catch

Aya 23 35B is released under CC-BY-NC 4.0. That means non-commercial use only. Personal projects, academic research, internal evaluation, and open-source non-commercial work are fine. Commercial deployment — even internal business use that generates revenue — is not covered. Cohere offers the same model family commercially through their hosted API, but the local weights themselves are off-limits for production at a for-profit company.

For commercial multilingual deployments, the practical alternatives are Llama 3.3 70B (Llama Community License, 700M MAU cap), Qwen3 32B (Apache 2.0), and Mistral Small 3 24B (Apache 2.0). None match Aya's multilingual quality, but all permit commercial use. The model catalog filters by license if that's the binding constraint.

Verdict

Dimension	Score / 10	Notes
Multilingual quality	9.5	Best-in-class for 23 supported languages
English quality	7.5	Solid but trails Llama 3.3 70B and Qwen3 32B
Coding	4.5	HumanEval 32% — not the right tool
Math & reasoning	5.5	GSM8K ~71%, MATH ~28%
VRAM efficiency	7.0	Q4_K_M fits 24 GB; awkward above that
Throughput	7.5	28 tok/s on RTX 4090 is competitive
License flexibility	3.0	CC-BY-NC blocks commercial use
Overall (multilingual focus)	8.2	Top recommendation for non-commercial multilingual local LLM

The verdict: Aya 23 35B is the right pick if and only if you are deploying non-commercially and your primary value lies in non-English language quality. For everything else — commercial use, English-only workloads, coding, agents — pick something else. Aya 23 8B remains a useful 16 GB-friendly alternative when the 35B's VRAM is unavailable, with a meaningful but not catastrophic quality drop.

All benchmark data backing this review is available through the BestLLMfor public API under CC BY 4.0, and the open-source MCP server at github.com/bestllmfor/mcp-server exposes the same dataset to any LLM client. The full methodology page documents the test harness, prompt sets, and statistical approach the editorial team uses across model reviews. For background on how BestLLMfor operates, see the about page.