Guide · 2026-07-16

How to Install Ollama with ROCm on AMD GPU

Last updated 2026-07-16

A tested, no-nonsense guide to GPU-accelerating Ollama on AMD Radeon and Instinct hardware with ROCm — including the fixes the official docs skip.

By Mohamed Meguedmi · 9 min read

Why ROCm Is the Only Path Worth Taking on AMD

If you own an AMD GPU, ROCm (Radeon Open Compute) is the runtime that lets Ollama push transformer math onto the GPU instead of grinding it out on the CPU. Without it, Ollama still runs — it just runs slowly, and every token is computed by your processor. The difference is not marginal. Across our test matrix, ROCm acceleration delivered a 3× to 10× throughput increase depending on model size and quantization, and it moved the memory footprint into fast VRAM where it belongs.

The friction has always been support detection. AMD's compute stack targets a specific GPU architecture string (the gfx ID), and Ollama only enables acceleration for architectures its shipped ROCm libraries were compiled against. Match that ID and you are done in five minutes. Miss it and Ollama detects your card, stalls, and quietly falls back to CPU. This guide gets you to the first outcome and shows you how to recover from the second.

Step 1 — Confirm Your GPU Is Supported

Before installing anything, find out which architecture your card reports. On Linux, rocminfo | grep gfx prints it if any ROCm tooling is present; otherwise match your model against the table below. The Official column means Ollama accelerates it with no extra configuration.

AMD GPU family	Example cards	gfx ID	Official Ollama support
RDNA2 (RX 6000)	RX 6800, 6900 XT, 6700 XT	gfx1030	Yes
RDNA3 (RX 7000)	RX 7900 XTX, 7900 XT, 7800 XT	gfx1100 / 1101	Yes
RDNA3 lower tier	RX 7600, 7700 XT	gfx1102	Yes
RDNA4 (RX 9000)	RX 9070 XT, 9070	gfx1201	Needs ROCm 7 build
CDNA (Instinct)	MI210, MI250, MI300X	gfx90a / gfx942	Yes
RDNA2 APU / older	680M, RX 6600, Vega	gfx1031+ / gfx90c	Override required

The authoritative, continuously updated list lives in the Ollama GPU documentation. If your card is in the Yes rows, skip straight to installation. If it is in an override or ROCm 7 row, read those dedicated sections below first.

Step 2 — Install Ollama with ROCm on Linux

This is the fast path for RX 6000, RX 7000, and Instinct cards. It does not require you to install AMD's full ROCm SDK — Ollama's installer pulls a self-contained ROCm library bundle.

That is the entire happy path. On a clean Ubuntu 24.04 or Fedora 42 system with a supported card, expect it to take under five minutes end to end.

Step 3 — Windows Setup

On Windows, download the installer from the official Ollama download page. It ships acceleration libraries compiled against ROCm 6.4, which covers RX 6000 and RX 7000 series natively. Install it, launch Ollama, and run a model exactly as on Linux; verification is identical via ollama ps.

The catch is the RX 9000 (RDNA4) series. Its gfx1201 architecture is only supported by ROCm 7.x, which the stock Windows installer does not yet bundle. On a default install, Ollama discovers the card, hangs for about 30 seconds trying to initialize it, then falls back to CPU. Until an official ROCm 7 build ships, RDNA4 owners on Windows should use a community ROCm 7 rebuild of Ollama or run under WSL2 with a ROCm 7 runtime. This is a compatibility gap, not a hardware limitation — the silicon is more than capable.

Step 4 — Force Support on Unlisted GPUs and APUs

Many perfectly capable chips — RDNA2 APUs like the Radeon 680M, the RX 6600, and various Vega parts — report a gfx ID Ollama does not list, so it skips them. The fix is HSA_OVERRIDE_GFX_VERSION, which tells the ROCm runtime to treat your card as a supported sibling architecture.

Set it as an environment variable for the Ollama service. On a systemd Linux install:

sudo systemctl edit ollama
# add under [Service]:
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Then sudo systemctl daemon-reload && sudo systemctl restart ollama. The value maps to the target architecture: 10.3.0 for RDNA2-class chips (gfx1030), 11.0.0 for RDNA3 (gfx1100). Pick the override that matches the nearest supported architecture to your silicon — forcing a wildly different generation will crash the runtime rather than accelerate it.

An override tells ROCm to pretend. It works because compatible architectures share the same instruction set, but it is unsupported by AMD. Test with small models first and watch journalctl -u ollama -f for HIP errors before trusting it with production workloads.

Step 5 — Benchmarks: What Acceleration Actually Buys You

Numbers cut through the marketing. The table below reports single-request generation throughput from our standardized harness — same prompt, 512-token output, greedy decoding, models pulled directly from the Ollama registry. CPU baseline is a modern 8-core desktop part; every GPU row uses ROCm offload with the full model resident in VRAM.

Hardware	VRAM	Llama 3.1 8B Q4_K_M	Qwen3-Coder 32B Q4_K_M	Approx. street price
CPU only (8-core)	—	~11 tok/s	~3 tok/s	—
RX 7600	8 GB	~42 tok/s	N/A (won't fit)	$270
RX 7800 XT	16 GB	~64 tok/s	~19 tok/s	$490
RX 7900 XTX	24 GB	~85 tok/s	~28 tok/s	$900
Instinct MI210	64 GB	~110 tok/s	~41 tok/s	data-center

Two conclusions stand out. First, VRAM is the gating factor for larger models: a 32B model at Q4_K_M needs roughly 20 GB, so 8 GB cards simply cannot hold it and must offload layers to system RAM, which erases most of the speedup. Second, the RX 7900 XTX is the clear value pick for serious local inference — 24 GB of VRAM keeps 32B-class models fully on-GPU at a fraction of comparable NVIDIA pricing. You can model total cost against cloud alternatives with our cost calculator, and compare quantized memory footprints across our benchmark suite.

Step 6 — Troubleshooting: "GPU Detected, Still 100% CPU"

This is the single most common complaint, and it almost always traces to one of four causes. Work through them in order.

• Permissions. If your user is not in the render and video groups, ROCm cannot open /dev/kfd. Confirm with groups; re-add and re-login if missing.
• Unsupported architecture. Run journalctl -u ollama -e. A line like no compatible amdgpu devices detected means your gfx ID is unlisted — apply the HSA_OVERRIDE_GFX_VERSION fix from Step 4.
• Insufficient VRAM. If the model is larger than free VRAM, Ollama splits layers between GPU and CPU and ollama ps shows a mixed split like 60% GPU / 40% CPU. Choose a smaller quantization (Q4_K_M over Q6_K) or a smaller model.
• Stale service environment. Environment variables set in your shell do not reach the systemd service. Always set them via systemctl edit ollama, not export.

For deeper diagnostics, rocminfo and rocm-smi (from AMD's ROCm tools) report device health and live VRAM usage. AMD's own ROCm system requirements reference is the source of truth for kernel and distribution compatibility if you suspect a driver-level problem.

Build Your Own Comparisons

Every throughput figure on this page is reproducible. Our full model catalog, quantization footprints, and benchmark data are available through the BestLLMfor public API under a CC BY 4.0 license, and the same data is exposed via our open-source MCP server so you can query model recommendations directly from an agent. Browse the interactive versions in our model catalog and the ranked picks in our recommendations.

The Verdict

Installing Ollama with ROCm on a supported AMD GPU is genuinely a five-minute job in 2026 — the one-line installer bundles everything RDNA2 and RDNA3 owners need. The complexity only appears at the edges: brand-new RDNA4 cards waiting on ROCm 7, and older APUs that need a one-line override. For anyone building a local LLM setup on a budget, the RX 7900 XTX is our recommended card: 24 GB of VRAM, native support, and roughly 8× the throughput of CPU-only inference at a price NVIDIA cannot match at the same memory tier.

Scenario	Recommended path	Difficulty
RX 6000 / 7000 on Linux	Official install script — done	Trivial
RX 6000 / 7000 on Windows	Official installer	Trivial
RX 9000 (RDNA4)	ROCm 7 community build or WSL2	Moderate
RDNA2 APU / unlisted card	Install + HSA_OVERRIDE_GFX_VERSION	Moderate
Instinct MI2xx / MI300	Official install, ideal for scale	Trivial

See our testing methodology for how these throughput numbers are produced and standardized across hardware.