What Is RAG? Retrieval-Augmented Generation, Explained Without the Hype

RAG in one paragraph

A language model knows only what was in its training data, frozen at a cutoff date, and it cannot tell you where any particular fact came from. Ask it about your company's leave policy and it will either refuse or invent something plausible. Retrieval-augmented generation fixes this without touching the model: before the model answers, a retrieval step finds the passages in your own documents most relevant to the question and pastes them into the prompt. The model's job shrinks from "know the answer" to "read these excerpts and answer from them." The term comes from a 2020 paper by Lewis et al. at Facebook AI Research, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks".

How a RAG pipeline works

Once, ahead of time: indexing

1. Load the documents: PDFs, web pages, wiki exports, tickets, code.
2. Chunk them into passages, typically 200 to 800 tokens, with a little overlap so ideas are not cut in half.
3. Embed each chunk: an embedding model turns the text into a vector of a few hundred to a few thousand numbers that represents its meaning.
4. Store the vectors, with the original text and metadata, in a vector database.

On every question: retrieval and generation

1. Embed the question with the same embedding model.
2. Retrieve the chunks whose vectors are closest to the question's vector, usually the top 5 to 20.
3. Rerank (optional but valuable): a second, more precise model rescores those candidates and keeps the best few.
4. Generate: build a prompt containing the instructions, the retrieved passages and the question, and send it to the language model.
5. Cite: return the answer with links to the source passages.

Component	What it does	Common local choices
Embedding model	Text → vector	nomic-embed-text, bge-m3, Qwen3-Embedding; see best self-hosted embedding models
Vector store	Finds nearest vectors fast	Chroma, LanceDB, Qdrant, pgvector, FAISS
Reranker	Rescores candidates precisely	bge-reranker and similar cross-encoders
Generator	Writes the answer from the passages	A 7B–14B instruct model; see best LLMs for RAG
Orchestration	Glues the steps together	LlamaIndex, LangChain, Haystack, or a ready-made app such as Open WebUI or AnythingLLM

Why RAG instead of a bigger model or a longer prompt?

Approach	Works when	Breaks when
Ask the model directly	The knowledge is public and stable	It is private, recent or obscure: the model guesses
Paste everything into a long context	The corpus is a few documents	It is thousands of pages: cost, latency and memory explode, and accuracy drops in the middle of long prompts
Fine-tune on your documents	You want a style, format or skill	You want reliable recall of specific facts, or the facts change weekly
RAG	The answer exists somewhere in a large or changing corpus	The question needs reasoning across the whole corpus at once ("summarize every contract")

The long-context row deserves numbers when you run locally. Every token in the prompt occupies KV-cache memory, from about 0.13 MB per token on an 8B model to 0.33 MB on a 70B model. Stuffing 100,000 tokens of documents into an 8B model costs about 13 GB of VRAM for the context alone, on every request. Retrieving 3,000 relevant tokens instead costs 0.4 GB. The arithmetic is laid out in what is a token in AI and context window vs VRAM cost.

RAG vs fine-tuning

	RAG	Fine-tuning
Changes	What the model is shown	The model's weights
Best for	Facts, documents, anything that must be current or cited	Tone, output format, domain jargon, a narrow skill
Updating knowledge	Re-index the changed files: minutes	Retrain: hours to days
Traceability	Can cite the exact passage	None
Access control	Filter retrieval per user	Impossible: knowledge is baked in for everyone
Up-front cost	An indexing pipeline	A training dataset and GPU time

The common mistake is fine-tuning a model on company documents and expecting it to recite them. Fine-tuning teaches patterns, not a reliable lookup table. Mature systems often use both: a model fine-tuned to answer in the house format, fed by retrieval for the facts.

Where RAG goes wrong

• Retrieval misses. If the right passage is not in the top results, the model cannot answer correctly, and a fluent wrong answer is the usual outcome. Always inspect what was retrieved before blaming the model.
• Bad chunking. Splitting a table from its header, or a clause from its exception, produces passages that are individually misleading.
• Pure vector search on exact terms. Embeddings capture meaning, not strings. Part numbers, error codes and names are found better by keyword search. Hybrid search, combining both, is the standard fix.
• Garbage in. Scanned PDFs with broken OCR, duplicated pages and outdated versions poison the index. Document cleaning is most of the work in a real project.
• The model ignores the context. Smaller models sometimes answer from memory anyway. A firm instruction ("answer only from the passages; say so if the answer is not there") and a model tuned for instruction following reduce this.
• Whole-corpus questions. "What are the common themes across all 400 reports" is not a retrieval problem. It needs summarization pipelines or graph-based approaches.

Running RAG locally: what it takes

Piece	Memory	Notes
Embedding model	≈ 0.3–1.5 GB	Runs fine on CPU for small corpora; GPU helps when indexing tens of thousands of pages
Vector store	Disk, plus some RAM	A few hundred thousand chunks fit comfortably on a laptop
Generator, Qwen 3 8B at 4-bit	5 GB + ≈ 1 GB context	Enough for question answering over retrieved passages on an 8 GB GPU
Generator, Qwen 3 14B at 4-bit	9 GB + ≈ 1.5 GB context	Noticeably better at following "answer only from the context"; fits 12 GB

Model sizes from the BestLLMfor catalog, September 20, 2026.

RAG is a good fit for local hardware for a structural reason: the facts arrive in the prompt, so the model does not need encyclopedic knowledge. It needs to read carefully and follow instructions, which small models can do; see small language models. And nothing leaves the machine, which is frequently the whole point: contracts, patient notes and source code are exactly the corpora people want to query and cannot upload. For a complete self-hosted stack, see best self-hosted RAG setups and the Local RAG Kit. Catalog figures here are open through the BestLLMfor public API (CC BY 4.0) and our MCP server.

How to tell whether your RAG works

1. Write 30 to 50 real questions with the document and passage that contains each answer.
2. Measure retrieval alone first. For what share of questions is the right passage in the top 5? Below roughly 80%, no model will save the system.
3. Then measure answers: correct, grounded in the passages, and honest when the corpus has no answer.
4. Change one thing at a time: chunk size, embedding model, hybrid search, reranker. Re-run the same questions.

Frameworks document their evaluation tooling well; LlamaIndex and LangChain's RAG tutorial are reasonable starting points.