What is Ollama?
TL;DR: Ollama is a free, open-source tool that downloads and runs large language models on your own computer with a single command. It wraps the llama.cpp engine, handles model downloads and GPU acceleration for you, and exposes a local API other apps can talk to. Windows, macOS, Linux — nothing you type leaves your machine.
What Ollama actually does
Type ollama run followed by a model name, and Ollama downloads the model (in the compressed GGUF format), loads as much of it as possible onto your GPU, and drops you into a chat. That’s the whole pitch: it turns “running an LLM locally” from an afternoon of compiling and flag-tuning into one command. It also runs quietly in the background as a service, so any app on your machine can send it requests.
Three things it is not: it is not a model (it’s the runner — the models come from Meta, Google, Alibaba, Mistral and others); it is not a chat interface (the built-in one is a terminal — pair it with a GUI like Open WebUI if you want buttons); and it is not a cloud service — everything runs on your hardware, which is precisely the point.
How it works under the hood
Ollama is built on llama.cpp, the C++ inference engine that made local LLMs practical. Models are stored as GGUF files — quantized, meaning the weights are compressed from 16 bits down to ~4 bits each so a 27B-parameter model fits in ~16 GB instead of 54 GB (our quantization guide unpacks this). At load time, Ollama offloads as many layers as fit onto your GPU and streams the rest from system RAM. It also serves a REST API on localhost:11434 that speaks a de-facto standard many tools integrate with.
The five commands you actually need
| Command | What it does |
|---|---|
ollama run <model> | Download if needed, load, and chat in the terminal |
ollama pull <model> | Download a model without starting a chat |
ollama list | Show every model you have on disk (and their sizes) |
ollama ps | Show what is currently loaded in memory |
ollama serve | Run the API server in the foreground (usually automatic) |
Model names come from the Ollama library (ollama.com/library). A tag after a colon picks the size or quantization — e.g. :9b or :27b.
Which model should you start with?
Depends on your memory, not your ambitions. On an 8GB GPU, start with Qwen 3.5 9B (~6 GB at Q4, Apache 2.0). On a 24GB card (RTX 3090/4090), Qwen 3.5 27B or Gemma 4 31B (~16–18 GB at Q4) are the current sweet spot. Our best Ollama models ranking covers the full picture, the configurator matches models to your exact hardware, and the VRAM calculator tells you what fits before you download 20 GB for nothing.
Where Ollama stops
Ollama optimizes for one user on one machine. If you need to serve many parallel users at high throughput, dedicated serving engines like vLLM take over — different tool, different job. And if you prefer clicking to typing, LM Studio covers the same ground with a desktop interface. Both can coexist on the same machine.
Frequently asked questions
Is Ollama free?
Yes. Ollama is open-source (MIT license) and free to use. The models themselves carry their own licenses — most popular ones (Qwen, Mistral) are Apache 2.0, others (Llama, Gemma) have custom terms worth reading for commercial use.
Does Ollama need a GPU?
No, but you'll want one. On CPU alone a small model works at a few tokens per second; a GPU or an Apple Silicon Mac speeds this up 5-10x. Ollama uses whatever acceleration your machine has automatically.
Is Ollama private? Does my data leave my computer?
Inference runs entirely on your machine — prompts and answers stay local. Ollama contacts the internet to download models and check for updates, nothing more by default.
What is the difference between Ollama and ChatGPT?
ChatGPT is a cloud service running OpenAI's closed models on their servers. Ollama is a tool that runs open-weight models on your own hardware — you trade some raw capability for privacy, zero per-token cost, and full control.
By Mohamed Meguedmi — independent comparator of locally-runnable LLMs, benchmarked on a real RTX 5070 Ti (data CC BY 4.0). See the local LLM leaderboard and the best Ollama models.