Granite 4.0 H-Small 32B-A9B

Overview

IBM's hybrid Mamba-2 + MoE model with 32B total and 9B active parameters, engineered to slash long-context memory use by roughly 70% versus comparable transformers under Apache 2.0.

When to pick this model

• Long-document RAG pipelines where VRAM is the bottleneck
• Enterprise deployments needing a permissive Apache 2.0 license
• Self-hosted assistants handling 100k+ token transcripts
• Cost-sensitive inference at sustained high concurrency
• Workloads where you want MoE throughput without the H100-class footprint

VRAM requirements by quantization

Quantization	VRAM required
Q4_K_M (recommended)	19 GB
Q5_K_M	23 GB
Q8_0	35 GB
FP16 (no quantization)	64 GB

VRAM figures include model weights plus a typical 8k KV cache and ~600 MB runtime overhead (Ollama / llama.cpp baseline). Add headroom for higher context lengths.

Strengths

• Hybrid Mamba-2 architecture cuts long-context memory by ~70%
• MoE design keeps active params at 9B for fast inference
• Apache 2.0 with no usage restrictions
• Built with enterprise governance and provenance in mind
• Strong throughput on commodity multi-GPU setups

Limitations

• Requires a recent llama.cpp build for hybrid architecture support
• Tooling ecosystem still catching up to dense Llama-class models
• Quality trails frontier 30B+ dense models on hard reasoning

Architecture & training

Architecture: Hybrid Mamba-2/Transformer (9:1) + granular MoE · 32B/9B active

Training: Granite 4.0 family.

Quick start

ollama run granite4:small-h

Or use the open-source MCP server to query this model from Claude Desktop, Cursor, or any MCP-compatible client.