Local Agentic RAG: when it's worth the peine

#Agentic RAG in brief

A classic RAG follows a fixed path: the question triggers a search, the retrieved passages go into the prompt, and the model answers. Agentic RAG replaces this path with a loop. The model has search tools and decides for itself whether to call them, with what query, and how many times, until it judges that it can answer or should give up.

Microsoft's AI Agents for Beginners course describes it as a process in which the model plans its next steps while retrieving information from external sources, with a "maker-checker" loop: produce, verify, correct. The same course points out the limit of this autonomy: it operates within the task and the tools you have provided.

The practical takeaway fits in one sentence. If your RAG fails because a single search can't gather the whole answer, a loop can help. If it fails because ingestion or chunking is poor, a loop will only repeat the problem, more slowly.

#Agentic RAG vs. single-pass RAG: the difference

The basic building blocks stay the same: embeddings, a vector database, and a generation model. If you're not familiar with them, start with the introduction to local RAG. What changes is who decides the sequence of steps. In classic RAG, it's your code; in agentic RAG, it's the model, within the limits your code imposes.

Query

Classic RAG: the user's question, sometimes rewritten once. Agentic: the model writes its own queries and can break one question down into several.

Number of searches

Classic RAG: one, known in advance. Agentic: varies from one question to the next, capped by a limit you set.

Sources

Classic RAG: usually a single index. Agentic: several possible tools (vector index, keyword search, SQL database), among which the model chooses.

Checking the result

Classic RAG: none, the model writes with what it received. Agentic: the model can judge the passages insufficient and search again, which isn't a guarantee: it can also get this judgment wrong.

Model requirement

Classic RAG: any model capable of writing from excerpts. Agentic: a model that handles tool calling correctly.

Predictability

Classic RAG: fairly consistent response time. Agentic: variable time and a path to check in the traces.

Three mechanisms make up the loop. Planning: breaking the question down into sub-questions. Tool use: translating each sub-question into a call. Retrying: reading the result and deciding whether to rephrase or conclude. For the overall architecture of an agent (memory, guardrails, monitoring), see the dedicated guide: here, we're only interested in the search loop.

There's also an often-overlooked middle ground: the fixed-step pipeline. Query rewriting, hybrid search, and then a reranker always run in the same order, with no decision by the model. This isn't agentic, and it's often the first thing to try.

#When agentic RAG is worth it

The right criterion is the shape of your questions, not the level of sophistication you want. Look at your current RAG's actual failures, then categorize them.

Multi-hop questions

The answer to a first search determines the second: find a contract's supplier, then the clause that applies to that supplier.

Cross-document comparisons

“Compare the notice periods in contracts A and B” requires two separate searches. A single query can return five passages from the same contract.

Multiple sources of different types

A figure in a SQL database and an explanation in PDFs. The model chooses the tool based on the question, as Microsoft's course describes for complex database interactions.

Tolerance for waiting

Document research, preparing a case file, background processing: a few dozen extra seconds won't bother anyone if the result gets reviewed.

Accuracy over speed

When an incomplete answer is costly, a verification step is defensible, provided you keep human review on high-stakes decisions.

On the other hand, stick with simple RAG in these situations:

FAQs and single-source questions

The answer is in one passage: one search is enough, and a loop only adds waiting.

Conversational assistant on modest hardware

The user expects a fast answer and the model is already running close to its limits.

Search that misses the right passages

Fix chunking, embeddings, and hybrid search first. Rerunning a poorly tuned search just gives you different bad results.

Exact calculation on structured data

A SQL query that has been written and tested is more reliable than an agent improvising one.

Multiple links between entities across the entire corpus

If you need to traverse relationships at scale, a graph approach addresses a different problem: see the GraphRAG guide.

#The cost in latency and model requirements

In the cloud, every loop iteration shows up on your bill. Locally, it shows up on the clock and in memory. Three factors matter.

#Latency multiplies

For N searches, the model generates N + 1 times: once before each search and a final time for the answer. A single-pass RAG generates only once. Because the prompt grows with each turn, each generation rereads more text, except what the runtime's context cache can reuse. Total time depends on the model, the quantization, and the machine; measure it on your own setup rather than relying on a general figure.

#Context fills up quickly

Illustrative calculation: 4 passages of 1,200 characters come to about 1,400 tokens, at 3 to 4 characters per token. Five searches therefore add about 7,000 tokens, not counting the question, the instructions, and the model's reasoning. With num_ctx at 8192, the margin has almost disappeared. Ollama then truncates the context, and the earliest results, sometimes the instructions, can fall out of the window without the application clearly flagging it.

Increasing num_ctx uses more memory and slows down subsequent iterations. It's often better to return fewer passages, or to summarize their content before feeding it back in.

#The model has to sustain the loop

The model has to produce well-formed tool calls, then judge whether what it has read is enough. Check for support with ollama show: the Capabilities section should list tools. This check tells you the format is supported, not that the model makes good decisions.

RTX 3060 12 GB or RTX 4070 12 GB

A 7 to 9B model in Q4_K_M (site reference: 7B ≈ 5 GB) leaves room for context and the embedding model. That's the starting point for the example below, with qwen3.5:9b.

RTX 4080 16 GB

A 14B model in Q4_K_M (≈ 9 GB) fits with a more comfortable margin for an expanded context.

RTX 4090 24 GB or M4 Pro 24 to 48 GB

A larger model becomes an option, such as qwen3.8:27b (about 18 GB). On 24 GB, keep an eye on what remains once the context is expanded.

#Choosing an orchestrator

The orchestrator runs the loop: it passes along tool calls, returns the results to the model, and enforces the limits. There are three approaches, all compatible with a local model.

Hand-written loop

About twenty lines around the Ollama chat API and its tools parameter. Suitable for understanding the mechanism or for a very simple case. You'll write the limits and traces yourself.

LangGraph

The loop is an explicit graph of nodes and edges, with state, a recursion limit, and step streaming. A good choice if you want to control every transition. It's the one used in our example.

LlamaIndex

Worth considering if your ingestion and indexes are already there: its agents reuse your query engines as tools.

Don't choose based on popularity. If your pipeline already runs on a framework, stick with it. In any case, pin your versions: these libraries evolve quickly and their prebuilt functions change. That's why our example writes the graph explicitly, which also makes the loop easy to read.

#Prerequisites

Ollama

Installed and listening on http://localhost:11434.

Python

A recent virtual environment. Check the version required by the packages at install time.

Documents

A few Markdown or text files in a mes_documents folder. For PDFs, reuse the ingestion from the ChromaDB tutorial.

Memory

About 6 GB free for a 9B model and its context, more if you increase num_ctx.

#Agentic RAG example with LangGraph and Ollama

The example builds an agent with a single search tool, on top of a local ChromaDB index. It's deliberately minimal so the loop stays visible.

#Step 1: Install the models and packages

ollama pull qwen3.5:9b
ollama pull nomic-embed-text
ollama show qwen3.5:9b

python -m venv .venv
source .venv/bin/activate
pip install langgraph langchain-ollama chromadb ollama
pip freeze > requirements.txt

#Step 2: Index the documents

Character-based chunking is used here only to keep the example short. On a real corpus, use chunking that respects document structure, and an embedding model validated on your French-language texts.

from pathlib import Path

import chromadb
import ollama

EMBED_MODEL = "nomic-embed-text"

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    "docs", metadata={"hnsw:space": "cosine"}
)


def decouper(texte, taille=1200, recouvrement=150):
    pas = taille - recouvrement
    return [texte[i:i + taille] for i in range(0, len(texte), pas)]


for fichier in Path("mes_documents").glob("*.md"):
    morceaux = decouper(fichier.read_text(encoding="utf-8"))
    if not morceaux:
        continue
    vecteurs = ollama.embed(model=EMBED_MODEL, input=morceaux)["embeddings"]
    collection.upsert(
        ids=[f"{fichier.name}-{i}" for i in range(len(morceaux))],
        documents=morceaux,
        embeddings=vecteurs,
        metadatas=[{"source": fichier.name} for _ in morceaux],
    )

print(collection.count(), "passages indexés")

#Step 3: Write the tool and the loop

The tool description matters: it's what the model reads to decide whether to call the tool. The system prompt requires searching before answering, separating sub-questions, and admitting when information is missing.

import chromadb
import ollama
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.tools import tool
from langchain_ollama import ChatOllama
from langgraph.graph import START, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode, tools_condition

EMBED_MODEL = "nomic-embed-text"
collection = chromadb.PersistentClient(path="./chroma_db").get_collection("docs")


@tool
def chercher_documents(requete: str) -> str:
    """Cherche dans les documents internes. Utiliser une requête courte
    portant sur un seul sujet. Renvoie des passages précédés de leur source."""
    vecteur = ollama.embed(model=EMBED_MODEL, input=requete)["embeddings"][0]
    res = collection.query(query_embeddings=[vecteur], n_results=4)
    passages = zip(res["documents"][0], res["metadatas"][0])
    return "\n\n".join(f"[{m['source']}] {d}" for d, m in passages) or "Aucun passage."


outils = [chercher_documents]
llm = ChatOllama(model="qwen3.5:9b", temperature=0, num_ctx=8192).bind_tools(outils)

CONSIGNE = SystemMessage(content=(
    "Tu réponds uniquement à partir des résultats de chercher_documents. "
    "Si la question porte sur plusieurs sujets, fais une recherche par sujet. "
    "Si les passages ne suffisent pas, reformule une fois puis dis clairement "
    "ce que tu n'as pas trouvé. Cite la source entre crochets."
))


def agent(state: MessagesState):
    return {"messages": [llm.invoke([CONSIGNE] + state["messages"])]}


graphe = StateGraph(MessagesState)
graphe.add_node("agent", agent)
graphe.add_node("tools", ToolNode(outils))
graphe.add_edge(START, "agent")
graphe.add_conditional_edges("agent", tools_condition)
graphe.add_edge("tools", "agent")
app = graphe.compile()

if __name__ == "__main__":
    question = "Compare la durée de préavis du contrat A et du contrat B."
    entree = {"messages": [HumanMessage(content=question)]}
    for etape in app.stream(entree, {"recursion_limit": 12}, stream_mode="values"):
        etape["messages"][-1].pretty_print()

#Step 4: Read the trace

The benefit of stream_mode="values" is that it shows every message. Read the trace before the final answer, using this checklist:

1. 01
   Does the model call the tool?

   If it answers directly, it isn't searching: strengthen the instruction or switch models.
2. 02
   Do the queries make sense?

   For the comparison, you'd ideally expect one query per contract. A single query repeating the entire question indicates that decomposition isn't happening.
3. 03
   Do the passages answer the question?

   If the excerpts are off topic, the problem is the index, not the agent.
4. 04
   Is the answer based on the excerpts?

   Check each bracketed citation against the corresponding passage.

#Step 5: Bound the loop

Each search uses two graph steps (agent, then tool). A limit of 12 therefore allows five searches, then the final answer. Beyond that, LangGraph raises a GraphRecursionError (langgraph.errors module): catch it in your application and return a clear message rather than leaving the user facing a stack trace.

#Pitfalls and troubleshooting

The model never calls the tool

Check tools in ollama show, rephrase the tool description, and try another model. Don't compensate with an ever-longer prompt.

It repeats the same search

The model is going in circles. Lower the limit and, in the tool, return a clear message when the query has already been run.

The answer forgets the first results

Likely a sign of a truncated context. Reduce n_results, split into smaller chunks, or increase num_ctx if memory allows.

Responses become very slow

Check ollama ps: part of the model may be on the CPU, or the models may be swapping in and out of memory. Also count the number of iterations in the trace.

It makes things up when it has found nothing

The loop doesn't prevent fabrication. Check citations programmatically: every cited source must appear in the conversation's tool results.

An overly powerful tool

An agent that reads documents also receives their content, which may be hostile. Give it read-only tools, and require human approval for any action that changes something.

#Decide based on your own questions

Figures published elsewhere cover other corpora, other models, and other settings. Only a comparison on your own data can settle the question, and it doesn't take much hardware.

1. 01
   Building a question set

   Take real questions whose answers and source passages you know, mixing simple questions with multi-hop ones. Keep track of each question's type.
2. 02
   Set a baseline

   Run them through your current RAG, without an agent. Note whether the right passage was retrieved, whether the answer is correct, and the total time.
3. 03
   Change only one thing

   Run the same set with the agent, on the same index and the same model, on the same machine.
4. 04
   Read by question type

   Look at the results separately for simple and complex questions, and record the number of searches per question. An overall average can hide a gain on one group and wasted time on the other.
5. 05
   Keep what's justified

   If only some questions benefit, routing may be enough: direct RAG by default, and the loop for questions detected as complex.

To automate the quality side of this comparison, the guide to Ragas covers suitable metrics. Also record the package versions, the model, the quantization, and num_ctx: without them, your result won't be comparable next month.

#Go further

Agentic RAG builds on a RAG system that already works. These guides cover the foundation to stabilize first, related approaches, and agent architecture.