NPU: what is it, and is it useful for a local LLM? ?

#What is an NPU?

From a computational perspective, a neural network boils down to immense series of matrix multiplications on low-precision numbers, repeated billions of times. A conventional processor can do this, slowly, a handful of operations at a time. A graphics card does it quickly, in parallel across thousands of units, consuming tens to hundreds of watts. An NPU is a piece of silicon designed to do only that, and nothing else: fixed-function compute units for 8- and 4-bit integers, small memories placed right next to them to avoid costly round trips, and a scheduler that moves data as little as possible. The result of this extreme specialization is modest peak speed compared with a GPU, but much higher efficiency per watt, which matters enormously on battery power.

The idea is older than the marketing label attached to it today. Apple has included a Neural Engine in its iPhones since 2017 and in all its Apple Silicon Macs since their launch; Google's Pixel chips have included a TPU block for several generations. What really changed in 2024 is that Intel, AMD, and Qualcomm all placed a substantial NPU in their mainstream laptop processors at the same time, and Microsoft immediately tied an entire range of Windows features to it under the commercial name Copilot+ PC. This convergence, more than the technology itself, explains the flood of “AI PC” labels on store shelves over the past two years.

#CPU, GPU, NPU: the differences

The three chips complement each other; they are not competitors: each exists because it makes a different trade-off between speed, power consumption, and versatility. On an “AI PC,” the NPU handles webcam effects for four hours of video conferencing while the GPU remains idle, saving a substantial portion of the battery life that continuous GPU use would consume. Ask the same machine to run an assistant with 14 billion parameters, and the workload moves entirely to the GPU because that is where both the necessary memory bandwidth and mature software support are available: no mainstream runtime can currently run an LLM of this size on an NPU.

#TOPS: what they measure and what they hide

TOPS stands for “trillion operations per second” (Tera Operations Per Second), almost always measured using 8-bit integers to remain comparable across datasheets. It represents peak compute performance, measured in the lab under favorable conditions. It says nothing about how quickly data reaches those compute units from memory, which is precisely what limits a language model in practice: a data-starved NPU spends most of its time waiting, regardless of how high its TOPS are.

This new generation, presented at CES in January 2026 and already available in laptops sold as of September 28, 2026, pushes the displayed figure well beyond the minimum Copilot+ threshold: the Snapdragon X2 Elite nearly doubles its predecessor’s score, reaching 80 TOPS versus 45 previously. None of this changes the conclusion of this page: the point that follows, about shared memory bandwidth, applies identically to these new chips, however impressive their TOPS figure may look on the label.

#Why an NPU Doesn't Accelerate Your LLMs

To produce a single token, a model must reread essentially all of its weights in memory, from the first to the last active parameter. A 14-billion-parameter model in Q4 weighs about 9 GB, so generating 20 tokens per second means moving 180 GB per second between memory and the compute units. The arithmetic itself—the multiplication proper—is cheap compared with this constant transfer. Generation speed therefore follows available memory bandwidth, not the TOPS or TFLOPS compute power advertised on the product sheet, which explains why two chips with very different TOPS ratings can write text at exactly the same speed.

A 50-TOPS NPU, like an 80-TOPS NPU in the very latest chips, sits at the top of the table. It draws on the same shared LPDDR5X as the processor: regardless of its advertised TOPS count, it physically cannot go faster than that memory allows. So a 45-TOPS laptop and an 80-TOPS laptop will write text at exactly the same speed if their memory capacity and bandwidth are identical.

Where an NPU can really help is prompt processing, a phase that is primarily compute-bound rather than bandwidth-bound, because the model processes all the text already present at once instead of generating it word by word. Hybrid configurations that assign this processing to the NPU and generation to the integrated GPU reduce the wait before the first displayed word and overall power consumption on small, well-supported models. This improves battery life and perceived responsiveness, measurable in latency; it is not a way to run larger models or generate continuously faster.

• VRAM and bandwidth: what really determines local AI performance

#The real bottleneck: software

Ollama, llama.cpp, LM Studio

They run on the GPU or CPU. On a Copilot+ laptop, they use the integrated GPU or processor and ignore the NPU.

Windows

Microsoft's embedded models and its Foundry Local environment target the NPU through ONNX Runtime, with a selection of small models.

AMD

Ryzen AI Software and the Lemonade server run some ONNX models in hybrid mode, using the NPU and integrated GPU.

Intel

OpenVINO can target the NPU for supported models.

Qualcomm

The QNN and AI Hub toolchains provide models already converted for the Hexagon NPU.

Apple

Core ML uses the Neural Engine, but the LLM tools actually used on Mac (MLX, llama.cpp, Ollama) run on the GPU through Metal.

The common thread across all these tools is that using the NPU means choosing from a short list of models already converted and validated by the manufacturer, generally with 1 to 8 billion parameters, rather than downloading any GGUF found on Hugging Face and running it directly. This conversion requires a different format and toolchain for each manufacturer, which explains why no community project has yet unified NPU support the way llama.cpp did for GPUs.

• Ryzen AI 9 HX: what the NPU really delivers with a local LLM
• Snapdragon X Elite: local LLM on ARM PC

#What to look at instead before buying

1. 01
   Memory available to the GPU

   The VRAM of a dedicated card, or the unified memory of a Mac or a Strix Halo machine. This is what determines which models can load.
2. 02
   The bandwidth of this memory

   It directly determines the number of tokens generated per second.
3. 03
   The software path

   NVIDIA with no friction, Apple Silicon nearly as good, AMD properly supported on Linux.
4. 04
   NPU TOPS

   Useful if you want Windows Copilot+ features and long battery life during video calls. No effect on open-weight LLMs today.

• Hardware for local AI: our machine guides
• Ryzen AI Max+ 395 mini PC: when unified memory changes the game
• Which GPU for a local LLM?
• Microsoft: Copilot+ devices and their NPU
• Source: Wikipedia, history and architecture of NPUs
• Source: AMD press release, Ryzen AI 400 (CES 2026)

#FAQ