Guide · 2026-07-26

How to Use LlamaIndex with a Local Ollama LLM

Last updated 2026-07-26

A verdict-driven guide to wiring LlamaIndex into a local Ollama model for private, offline RAG — with install steps, hardware specs, and model picks.

By Mohamed Meguedmi · 9 min read

Pairing LlamaIndex with a local Ollama LLM is the fastest route to a private retrieval-augmented generation (RAG) pipeline that never sends a byte to a third-party API. This guide takes you from a clean Python environment to a working query engine, then tells you exactly which model to run on the hardware you have. No cloud keys, no per-token billing, no data egress.

Why Pair LlamaIndex with Ollama?

LlamaIndex is an orchestration framework: it chunks your documents, embeds them, stores the vectors, retrieves the relevant context, and stitches it into a prompt. It does not run the model. Ollama is the runtime that actually serves the LLM over a local HTTP endpoint (http://localhost:11434). The division of labor is clean — LlamaIndex handles the RAG logic, Ollama handles inference.

The alternative most tutorials push is a hosted API (OpenAI, Anthropic, or a remote endpoint). That works, but it defeats the two reasons developers reach for this stack in the first place: data privacy and zero marginal cost. If you are indexing legal contracts, internal wikis, patient notes, or proprietary source code, keeping the entire pipeline on local hardware is not a nice-to-have — it is the requirement. Ollama makes local serving trivial, and LlamaIndex ships an official first-party integration for it, so the two are the pragmatic default for offline RAG.

Verdict up front: for a single developer prototyping private RAG on a 16 GB machine, LlamaIndex + Ollama running Llama 3.1 8B is the best cost-to-effort combination available today. Scale the model, not the framework, as your hardware grows.

Prerequisites and Hardware Requirements

Before installing anything, confirm your machine clears the memory bar. Ollama loads the full quantized model into RAM (or VRAM), and LlamaIndex adds a modest overhead for the embedding model and vector store. The table below reflects real-world footprints for Q4_K_M quantizations plus the framework overhead.

Local generator model	Quantization	Min. system RAM / VRAM	On-disk size	Best for
Llama 3.2 3B	Q4_K_M	8 GB	~2.0 GB	Laptops, smoke-testing the pipeline
Llama 3.1 8B	Q4_K_M	16 GB	~4.9 GB	Everyday RAG, the sweet spot
Qwen2.5 14B	Q4_K_M	24 GB	~9.0 GB	Higher reasoning, longer answers
Qwen2.5 32B	Q4_K_M	32 GB+	~20 GB	Near-hosted quality, offline

You also need Python 3.9+ and Ollama installed and running. Grab the installer from the official ollama.com download page. On macOS and Windows it runs as a background service; on Linux it installs a systemd unit. Verify it is live with ollama --version before continuing. If you want to size a model against your exact memory budget, our cost calculator converts parameter count and quantization into a RAM estimate.

Step-by-Step Installation

LlamaIndex is a collection of packages, not a monolith. You install the core plus only the integrations you need — in this case the Ollama LLM integration and a local embedding model.

1. Pull a model into Ollama

# Download the generator (~4.9 GB)
ollama pull llama3.1:8b

# Confirm it is available
ollama list

Model names and tags are documented in the Ollama model library. The tag after the colon selects the quantization/size — llama3.1:8b defaults to Q4_K_M.

2. Install the Python packages

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

pip install llama-index-core \
            llama-index-llms-ollama \
            llama-index-embeddings-huggingface

We deliberately install the HuggingFace embedding integration rather than relying on Ollama for embeddings. The BAAI/bge-small-en-v1.5 model is only ~130 MB, runs on CPU, and produces materially better retrieval than most general-purpose Ollama models pressed into embedding duty. Its model card lives on the HuggingFace hub.

3. Verify the LLM connection

from llama_index.llms.ollama import Ollama

llm = Ollama(model="llama3.1:8b", request_timeout=120.0)
resp = llm.complete("Name the capital of Australia in one word.")
print(resp)   # -> Canberra

If this returns text, LlamaIndex is talking to Ollama and you are ready to build the pipeline. If it hangs and errors, jump to the tuning section below — it is almost always the timeout.

Building Your First Local RAG Pipeline

Create a folder named data/ next to your script and drop in a few .txt, .pdf, or .md files. The following ~15 lines are a complete, fully local RAG engine.

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    Settings,
)
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

# 1. Configure the local models once, globally
Settings.llm = Ollama(model="llama3.1:8b", request_timeout=120.0)
Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

# 2. Load and index your documents
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

# 3. Query — entirely offline
query_engine = index.as_query_engine()
response = query_engine.query(
    "Summarize the key points across these documents."
)
print(response)

What happens under the hood: SimpleDirectoryReader parses your files, the bge embedder converts each chunk to a 384-dimension vector, LlamaIndex stores them in an in-memory vector index, and at query time it retrieves the top matches and hands them to Llama 3.1 through Ollama to compose the answer. Setting the models on the global Settings object means you never pass them again — every index and query engine inherits them.

To persist the index so you do not re-embed on every run, add index.storage_context.persist("storage") and reload with load_index_from_storage. For anything beyond a prototype, swap the in-memory store for a dedicated vector database, but the LLM and embedding wiring stays identical.

Choosing the Right Local Model

The framework code above does not change when you change models — only the Ollama tag does. So the real decision is which generator to serve. We measured single-stream generation throughput on a 32 GB Apple M-series unified-memory machine and a 24 GB consumer GPU, using Q4_K_M weights and a 2K-token context. Numbers are indicative, not lab-grade; treat them as relative ordering.

Model (Q4_K_M)	Params	Approx. tok/s (24 GB GPU)	RAG answer quality	Editorial call
Llama 3.2 3B	3B	~95	Adequate	Prototyping only
Llama 3.1 8B	8B	~62	Strong	Best default
Qwen2.5 14B	14B	~34	Very strong	Best if you have 24 GB
Qwen2.5 32B	32B	~16	Excellent	Quality over speed

The pattern is the usual quality-versus-latency trade. Llama 3.1 8B answers most document-grounded questions faithfully at a responsive ~60 tokens/second and fits comfortably in 16 GB. Qwen2.5 14B is the upgrade that pays off when answers need multi-step reasoning over retrieved context, provided you can spare the memory. The 32B tier approaches hosted-API quality but drops below conversational speed on consumer hardware — worth it for batch summarization, painful for interactive chat. We keep the full quantization-versus-throughput matrix current on our benchmarks hub, and the same figures are queryable through the BestLLMfor public API (CC BY 4.0) or the open-source MCP server if you want to pull them into your own tooling. Browse the full model list on the catalog.

Performance Tuning and Common Pitfalls

Four issues account for the overwhelming majority of support threads on this stack. Handle them up front.

• First-query timeout. The cold start loads gigabytes into memory before generating a single token. If request_timeout is left at its low default, the first query dies. Always pass request_timeout=120.0 (or higher for 32B models). This is the single most-reported failure.
• Wrong embedding model. If you skip setting Settings.embed_model, LlamaIndex tries to reach OpenAI and errors out with a missing-API-key message — surprising people who thought they were fully local. Setting the HuggingFace embedder fixes it and keeps everything offline.
• Context window overflow. Ollama defaults to a modest context length. For long documents, raise it when constructing the LLM: Ollama(model="llama3.1:8b", request_timeout=120.0, context_window=8192), and make sure your machine has the RAM headroom, since larger context inflates memory use.
• Swapping to disk. If tok/s collapses to single digits, the model spilled out of RAM/VRAM into swap. Drop to a smaller model or a lower quantization rather than fighting it — a Q4_K_M 8B that fits will always beat a Q4 14B that swaps.

For document parsing quality, prefer clean source files. Feeding scanned PDFs without OCR produces garbage chunks that no amount of model tuning will rescue. The generation model can only reason over what the retriever surfaces.

The Verdict

LlamaIndex + Ollama is the most direct path to private, offline RAG, and the setup cost is genuinely low — three pip installs and one configuration block. The decision that matters is the generator model, and it maps cleanly to your available memory.

Your situation	Generator	Embedder	Why
16 GB laptop, general RAG	Llama 3.1 8B Q4_K_M	bge-small-en-v1.5	Best balance of speed, quality, footprint
24 GB GPU, reasoning-heavy	Qwen2.5 14B Q4_K_M	bge-small-en-v1.5	Noticeably better multi-step answers
32 GB+, quality first	Qwen2.5 32B Q4_K_M	bge-base-en-v1.5	Near-hosted quality, fully offline
8 GB, prototyping	Llama 3.2 3B Q4_K_M	bge-small-en-v1.5	Validates the pipeline before you scale

Start with Llama 3.1 8B. It is the recommendation for the widest slice of readers, and because the framework code is model-agnostic, upgrading later is a one-line change to the Ollama tag. Compare more guides on the guides hub as your pipeline grows.