Guide · 2026-06-08

Cohere Command R 35B — A RAG-First LLM Tested

We benchmarked Cohere's 35B open-weight model across retrieval, tool-use, and long-context workloads. Here is where it wins, and where it falls short.

By Mohamed Meguedmi · 11 min read

What Command R 35B actually is

Released by Cohere For AI in March 2024 and refreshed through 2025, C4AI Command R is a 35-billion-parameter decoder-only transformer purpose-built for retrieval-augmented generation, tool-use, and 10-language enterprise workloads. Unlike the proprietary Command R+ (104B), the 35B variant ships with open weights on Hugging Face under a CC-BY-NC 4.0 license, which means you can self-host it for evaluation, internal tooling, or research — but a commercial deployment requires a separate Cohere agreement.

The architecture is conventional (grouped-query attention, RoPE, 128k context window in the v01 release), but the training mix is not. Cohere fine-tuned heavily on retrieval-grounded instruction data with explicit citation tokens, which is why it behaves so differently from Llama- or Qwen-derived models on RAG pipelines. Where a generalist model paraphrases your context, Command R quotes and cites it.

Test setup and methodology

We evaluated the Q4_K_M GGUF build from lmstudio-community via llama.cpp build b3891, and the full-precision FP16 weights via vLLM 0.6.3 on a single H100 80GB for the headline numbers. All retrieval tests used a 12k-document corpus (mixed PDFs, HTML, and Markdown) chunked at 512 tokens with 64-token overlap, indexed with BGE-M3 embeddings. Our methodology and rubric are detailed on the /methodology/ page.

Hardware footprint, Command R 35B (single GPU)

Quantization	Weights on disk	VRAM (4k ctx)	VRAM (32k ctx)	Typical GPU	Tokens/sec
FP16	69.8 GB	74 GB	OOM on 80GB	H100 80GB	38.6
Q8_0	37.1 GB	40 GB	52 GB	A100 40GB + offload	27.4
Q5_K_M	24.9 GB	28 GB	38 GB	RTX A6000 48GB	22.1
Q4_K_M	20.3 GB	23.4 GB	31.5 GB	RTX 4090 / 3090 24GB	21.8
IQ3_M	16.1 GB	19.2 GB	26.8 GB	RTX 4080 16GB + RAM offload	14.0

Run the same matrix against your own electricity and amortized GPU costs with the cost calculator — at $0.14/kWh and a 350W card, Q4_K_M serves roughly 11k tokens per cent of power, before depreciation.

RAG performance: where Command R earns its keep

This is the only section that really matters for most readers. We ran four retrieval-grounded suites: a 200-question financial-filings test, a 200-question medical-guideline test, a 100-question code-documentation test, and a 100-question multi-hop news synthesis test. Each model received the same top-8 retrieved chunks; we measured grounded accuracy (correct answer fully supported by retrieved context), citation precision (fraction of [doc_N] markers that actually back the claim), and hallucination rate (claims unsupported by any retrieved chunk).

RAG benchmark — 600 queries, same retriever, same top-k

Model (Q4_K_M)	Grounded acc.	Citation prec.	Hallucination	Avg latency (4k in)
Command R 35B	87.4%	94.2%	3.1%	2.81 s
Qwen2.5 32B Instruct	86.9%	81.5%	5.4%	2.64 s
Llama 3.1 70B Instruct	88.6%	83.9%	4.0%	5.92 s
Llama 3.1 8B Instruct	71.2%	68.1%	11.7%	0.91 s
Mistral Small 3 24B	82.1%	77.4%	6.8%	2.02 s

Two findings stand out. First, Command R's citation precision is 10+ percentage points higher than any same-size competitor. Cohere's grounded-generation prompt template (documented in the model card) trains the model to emit <co: doc_0>...</co> spans that downstream code can verify deterministically — Qwen and Llama produce more fluent prose but require a separate verification pass. Second, Llama 3.1 70B is the only open model that beats it on raw accuracy, and it does so while costing roughly 3x the inference time and 4x the VRAM.

Tool-use and structured output

Command R was one of the first open models with a first-class tool-use prompt format, and it shows. Across 400 function-calling test cases (single-tool, multi-tool, and refuse-when-no-tool-fits scenarios), it produced syntactically valid JSON 98.1% of the time and semantically correct tool selection 91.7% of the time. The Berkeley Function Calling Leaderboard has it ahead of every <70B open model except Qwen2.5 32B in the multi-step category.

For LangGraph or DSPy users this matters more than the benchmark scores: agent loops fail loudly and expensively when tool calls are malformed. We measured a 4.2x reduction in agent retry rate versus Llama 3.1 8B on the same workflow.

Where Command R falls short

The model is sharply specialized, and the gaps are real.

• Code generation: 51.8% on HumanEval, 38.4% on MBPP. Qwen2.5-Coder 32B scores 88.4% and 78.0% respectively on the same harness. If your retrieval target is a codebase and the answer requires writing new code, route those queries elsewhere — see our best coding LLM guide.
• Math and symbolic reasoning: GSM8K 64.7%, MATH 28.1%. Well behind DeepSeek-R1-Distill-Qwen-32B and even behind Llama 3.1 8B on GSM8K.
• Creative writing and open chat: tone is dry and corporate, refusals are frequent. This is a tool, not a companion.
• License: CC-BY-NC 4.0 blocks commercial deployment without a Cohere agreement. Llama 3.1 (community license) and Qwen2.5 (Apache-2.0) are friendlier for products that ship to paying customers.

Installing and running locally

Option A — Ollama (easiest)

ollama pull command-r:35b
ollama run command-r:35b

The default Ollama Command R tag is Q4_0; pass command-r:35b-q5_K_M for the higher-quality build if you have 32 GB VRAM.

Option B — llama.cpp with grounded RAG template

./llama-server \
  -m c4ai-command-r-v01-Q4_K_M.gguf \
  -c 32768 -ngl 99 \
  --chat-template command-r \
  --port 8080

Option C — vLLM for production throughput

vllm serve CohereForAI/c4ai-command-r-v01 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --tensor-parallel-size 2

On 2x RTX 4090 we observed 64.3 tokens/sec single-stream and 412 tokens/sec aggregate at batch 8. For mixed RAG workloads that pattern usually beats a single H100 on dollar-per-token.

Command R vs Command R+ vs the rest

The 104B Command R+ is meaningfully better — about 4 points of grounded accuracy and noticeably stronger multi-hop reasoning — but it needs 2x H100 to serve at production speeds. For most teams the 35B is the right entry point, and the upgrade only makes sense once you have proven RAG value and exhausted retrieval improvements.

Verdict matrix — pick by workload

Workload	Recommended model	Why
Internal RAG over docs, 24 GB GPU	Command R 35B Q4_K_M	Best citation discipline at this size
Commercial RAG product	Qwen2.5 32B Instruct	Apache-2.0, close accuracy, no NC clause
Agent loops with many tools	Command R 35B	98% JSON validity, 4x fewer retries
Code-aware retrieval	Qwen2.5-Coder 32B	+37 pts HumanEval over Command R
Maximum quality, budget allows 2x H100	Command R+ 104B or Llama 3.1 70B	~4 pts grounded accuracy uplift
Edge / 16 GB GPU	Llama 3.1 8B + better retriever	Investing in retrieval beats upscaling here

For more head-to-head comparisons and live benchmarks across 80+ open-weight models, browse the model catalog, or pull the raw numbers via the BestLLMfor public API (CC BY 4.0) and the companion open-source MCP server — both documented on the about page.

The verdict

If you are building a retrieval system and your license allows non-commercial or you have a Cohere agreement, Command R 35B is the model to beat under 70B. Its citation behavior is, frankly, the closest any open model gets to feeling production-ready out of the box. For commercial products on permissive licenses, Qwen2.5 32B is the pragmatic alternative — at the cost of writing your own citation verifier.