What Is an NPU? And Can It Actually Run a Local LLM?

What an NPU is

A neural network is, computationally, a very long sequence of matrix multiplications on low-precision numbers. A CPU can do that, slowly. A GPU does it fast, at tens to hundreds of watts. An NPU is silicon that does nothing else: fixed-function arithmetic units for 8-bit and 4-bit integer math, small on-chip memories placed next to them, and a scheduler that moves data as little as possible. The result is modest peak speed but excellent performance per watt.

The idea is older than the branding (the Wikipedia entry on neural processing units traces the lineage). Apple has shipped a Neural Engine in iPhones since 2017 and in every Apple Silicon Mac; Google's phones have had a TPU block for years. What changed in 2024 is that Intel, AMD and Qualcomm all put sizable NPUs into PC processors, and Microsoft tied a Windows feature tier to them.

NPU vs GPU vs CPU

	CPU	GPU	NPU
Built for	General-purpose logic, low latency	Massively parallel math, graphics	Neural-network inference only
Typical power under AI load	15–125 W	30 W (integrated) to 575 W (RTX 5090)	1–10 W
Memory	System RAM	Own VRAM (discrete) or system RAM (integrated)	System RAM
Software support	Universal	Very broad (CUDA, ROCm, Metal, Vulkan)	Narrow, vendor-specific
Sweet spot	Orchestration, small models in a pinch	LLMs, image generation, training	Sustained light AI on battery

The three are complementary, not competing. On an "AI PC" the NPU handles the webcam effects during a four-hour call while the GPU stays asleep. Ask the same machine to run a 14B-parameter chatbot and the work goes to the GPU, because that is where the memory bandwidth and the software are.

What TOPS means, and what it hides

TOPS counts how many trillion integer operations the NPU can perform per second, almost always at INT8 precision. It is a peak compute figure. It says nothing about how fast data reaches those arithmetic units, and that is precisely what limits a language model.

Chip family	NPU rating	Copilot+ eligible (40+ TOPS)
Intel Core Ultra Series 1 (Meteor Lake)	≈ 11 TOPS	No
Apple M4 Neural Engine	38 TOPS	Not applicable (macOS)
Qualcomm Snapdragon X Elite / X Plus	45 TOPS	Yes
Intel Core Ultra 200V (Lunar Lake)	48 TOPS	Yes
AMD Ryzen AI 300 series	50 TOPS	Yes
AMD Ryzen AI Max+ 395 (Strix Halo)	50 TOPS	Yes

Manufacturer figures. Vendors do not measure TOPS identically, so treat differences of a few points as noise. Microsoft documents the Copilot+ requirement and the supported silicon in its NPU developer guide.

Why an NPU does not make local LLMs fast

To produce one token, a model has to read essentially all of its active weights from memory. A 14B model at 4-bit is about 9 GB; generating 20 tokens per second means streaming roughly 180 GB through the processor every second. The arithmetic itself is cheap by comparison. This is why generation speed tracks memory bandwidth, not compute.

Where the model sits	Bandwidth	Rough ceiling for a 9 GB model
Laptop LPDDR5X (what the NPU and iGPU share)	≈ 70–135 GB/s	≈ 8–15 tok/s
Ryzen AI Max+ 395 unified memory	256 GB/s	≈ 28 tok/s
RTX 4070 (GDDR6X)	504 GB/s	≈ 56 tok/s
RTX 5070 Ti (GDDR7)	896 GB/s	≈ 100 tok/s
RTX 5090 (GDDR7)	1,792 GB/s	≈ 200 tok/s

Ceiling = bandwidth ÷ model size, an upper bound that real systems approach but do not reach. Model size from the BestLLMfor catalog (Qwen 3 14B, 4-bit).

An NPU rated at 50 TOPS sits on the first row. It draws from the same LPDDR5X as the CPU, so for token generation it cannot outrun that memory, however many TOPS it has. A 45 TOPS laptop and a 50 TOPS laptop will generate text at essentially the same speed if their RAM is the same. The full argument, with the roofline model, is in why VRAM matters more than TFLOPS.

Where an NPU can help an LLM is prompt processing, the compute-heavy phase where the model reads your input. Hybrid schemes that run prefill on the NPU and generation on the integrated GPU cut time-to-first-token and power draw on small models. That is a battery-life win, not a way to run bigger models.

The software gap

The second obstacle is support. CUDA, Metal, ROCm and Vulkan are mature targets that every local runtime speaks. NPUs each need their own compiler stack and model format:

• Ollama, llama.cpp, LM Studio: run on GPU and CPU. On a Copilot+ laptop they use the integrated GPU or the CPU and ignore the NPU.
• Windows: Microsoft's on-device models and its Foundry Local runtime target the NPU through ONNX Runtime, with a curated set of small models.
• AMD: Ryzen AI Software and the Lemonade server run selected ONNX models in hybrid NPU + iGPU mode.
• Intel: OpenVINO can target the NPU for supported models.
• Qualcomm: the QNN toolchain and AI Hub provide pre-converted models for the Hexagon NPU.
• Apple: Core ML uses the Neural Engine, but the LLM tools people actually use on Mac (MLX, llama.cpp, Ollama) run on the GPU through Metal.

The common thread: NPU inference means a short list of vendor-converted models, typically 1B to 8B parameters, rather than "pull any GGUF and run it."

What to look at instead when buying for local AI

1. Memory capacity the GPU can use. VRAM on a discrete card, or unified memory on a Mac or Strix Halo machine. This decides which models load at all. See what VRAM is and how much you need.
2. Memory bandwidth. This decides tokens per second.
3. Software path. NVIDIA is frictionless, Apple Silicon is close, AMD is good on Linux.
4. NPU TOPS. Relevant if you want Windows Copilot+ features and long battery life during calls. Irrelevant for running open-weight LLMs today.

Concrete options with numbers: the Ryzen AI Max+ 395 mini PC page covers the one x86 platform where unified memory changes the picture, and the AI hardware hub lists the Macs and GPUs we track. For model sizing by memory, use the VRAM calculator. The underlying data is open through the BestLLMfor public API (CC BY 4.0) and our MCP server.

The verdict

An NPU is a genuinely useful chip for what it was built for: keeping light AI features running all day on a few watts. It is not a hidden LLM accelerator. If your goal is running open-weight models locally, buy memory capacity and bandwidth, treat the NPU as a free extra, and ignore TOPS comparisons between otherwise similar laptops. If vendor toolchains mature and mainstream runtimes gain NPU backends, the prefill and efficiency gains will arrive as a software update. The memory ceiling will not move.