LLM Council: having multiple models deliberate locaux

#LLM Council: the three-step pattern

The repository describes a fixed three-stage flow. Each stage is an ordinary model call, with no tools or memory: the only trick is how the outputs from the first stage feed the next two.

1. 01
   Initial opinions

   The user's question is sent as-is to each council member separately. Each one responds without seeing the others. In the original interface, these responses appear in tabs, one per model.
2. 02
   Review and ranking

   Each member then receives the other responses, anonymized (labeled “Response A,” “Response B,” ...), and must rank them from best to worst based on accuracy and depth. The anonymization is intended to prevent a model from favoring or disadvantaging a known competitor.
3. 03
   President's synthesis

   A model designated as president receives the question, all the responses, and all the rankings, then writes the single final answer presented to the user.

Nothing in this design requires different models, but that is the whole point: it relies on training diversity so that one model’s errors are spotted by another. The chair can be one of the members or a separate model, generally the most capable one on the list.

#What's in Karpathy's repository

The github.com/karpathy/llm-council repository is a local web application that resembles a traditional chat interface. According to its README and configuration files at the time of writing, it consists of:

A Python backend (FastAPI)

In the backend/ directory: the logic for the three steps, an HTTP client for the OpenRouter API, and conversation storage in JSON files in a data/ directory. Dependencies are managed with uv.

A React frontend (Vite)

In the frontend/ directory: the interface that displays individual responses, rankings, and the summary.

A configuration file

backend/config.py lists the council members (COUNCIL_MODELS) and the chair (CHAIRMAN_MODEL) as OpenRouter identifiers, along with the address of the API being called.

A launch script

start.sh starts the backend and frontend together. An OPENROUTER_API_KEY is expected in a .env file.

The default-configured models are closed cloud models (the GPT, Gemini, Claude, and Grok families in the published version). This is what you need to change for a local version: the rest of the code does not care who responds, as long as the API speaks the “chat completions” format popularized by OpenAI. Review the README and config.py before following this guide; the project may have changed since October 12, 2026.

#Prerequisites for a local version

Ollama installed and working

The server listens on http://localhost:11434 by default. Run the ollama list command to verify that your models appear.

Three to four models from different families

A council of models from the same family loses the benefit of diversity. For example, aim for a Qwen, a Llama, a Mistral, and a Gemma, in similar sizes.

Adequate memory

With Q4_K_M, allow about 5 GB per 7B model and 9 GB per 14B model. The members do not all need to fit in memory at the same time, but your experience depends on it (see the section on costs).

Python 3.10 or later

For this guide’s minimal script, with the requests package. For the original repository, uv and Node.js as well.

Patience

One council round costs at least twice as many generations as the number of members, plus one. This is not an instant chat tool.

#Step 1: choose the council and president

Choosing the members is a balancing act between diversity and memory. The president, meanwhile, mainly needs to read long contexts well: it receives the question, all the responses, and all the rankings, which can easily amount to several thousand tokens. Choose the most capable model your machine can support for this role, even if it is slower, because it is called only once per round.

# Exemple de conseil pour une carte 12 Go (noms à vérifier avec `ollama search` ou sur ollama.com/library)
ollama pull qwen3:8b
ollama pull llama3.1:8b
ollama pull mistral:7b
ollama pull gemma3:12b   # président : plus gros, appelé une seule fois par tour
ollama list

7B to 8B models work well for members on a 12 GB card: one fits in memory with room for the context, and Ollama alternates between them. With 24 GB or a Mac with 48 GB of unified memory, you can move members up to 14B or keep two to three models loaded at the same time, eliminating reloads.

#Step 2: configure Ollama for multiple models

Two Ollama settings matter here. The first is the number of models kept in memory, set by the OLLAMA_MAX_LOADED_MODELS variable: if memory allows, keeping all members loaded avoids reloading them each time you switch between them. The second is the context window, num_ctx, which is short by default on Ollama: the ranking step and the president's synthesis must read several complete responses, and a context that is too small silently truncates the beginning of the prompt.

sudo systemctl edit ollama
# puis dans le bloc [Service] :
# Environment="OLLAMA_MAX_LOADED_MODELS=2"
# Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl restart ollama

On macOS, set the same variables with launchctl setenv before restarting the application. The value of OLLAMA_MAX_LOADED_MODELS must remain consistent with your available memory: two 8B models in Q4 already use about ten GB with their context. The context itself is configured per request (the API's options.num_ctx field) or in a derived Modelfile; the next section passes it with every call.

#Step 3: a minimal council in Python

The script below reproduces the repository’s three steps using the native API of Ollama, without an interface. It shuffles the response order before ranking so the labels do not reveal the author, asks each member for a short ranking, then hands everything to the chair. Adapt the model names to those on your machine.

import random
import requests

OLLAMA = "http://localhost:11434/api/chat"
COUNCIL = ["qwen3:8b", "llama3.1:8b", "mistral:7b"]
CHAIRMAN = "gemma3:12b"
NUM_CTX = 16384


def ask(model, prompt):
    r = requests.post(OLLAMA, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"num_ctx": NUM_CTX, "temperature": 0.3},
    }, timeout=1800)
    r.raise_for_status()
    return r.json()["message"]["content"].strip()


def council(question):
    # Étape 1 : premières opinions, une par membre, sans se voir
    answers = {m: ask(m, question) for m in COUNCIL}

    # Anonymisation : ordre mélangé, étiquettes A, B, C...
    order = list(COUNCIL)
    random.shuffle(order)
    labels = {m: chr(65 + i) for i, m in enumerate(order)}
    packed = "\n\n".join(
        f"### Réponse {labels[m]}\n{answers[m]}" for m in order
    )

    # Étape 2 : chaque membre classe les réponses (les siennes incluses)
    rankings = {}
    for m in COUNCIL:
        rankings[m] = ask(m,
            f"Question : {question}\n\n{packed}\n\n"
            "Classe ces réponses de la meilleure à la moins bonne selon leur "
            "justesse et leur profondeur. Pour chacune, une ligne de justification. "
            "Termine par la ligne 'CLASSEMENT :' suivie des lettres dans l'ordre.")

    # Étape 3 : le président rédige la synthèse finale
    votes = "\n\n".join(
        f"### Avis du juge {labels[m]}\n{rankings[m]}" for m in COUNCIL
    )
    final = ask(CHAIRMAN,
        f"Question : {question}\n\n{packed}\n\n{votes}\n\n"
        "Tu présides ce conseil. En t'appuyant sur les réponses et les classements, "
        "rédige une réponse finale unique, précise et sans redite. Signale les points "
        "sur lesquels les réponses se contredisent.")
    return answers, labels, rankings, final


if __name__ == "__main__":
    import sys
    q = " ".join(sys.argv[1:]) or "Quelle quantification choisir pour un modèle 14B sur 12 Go de VRAM ?"
    answers, labels, rankings, final = council(q)
    for m in COUNCIL:
        print(f"\n===== {m} (Réponse {labels[m]}) =====\n{answers[m]}")
    for m in COUNCIL:
        print(f"\n===== Classement par {m} =====\n{rankings[m]}")
    print(f"\n===== Synthèse ({CHAIRMAN}) =====\n{final}")

python3 -m venv .venv && source .venv/bin/activate
pip install requests
python council.py "Faut-il un reranker dans un pipeline RAG local ?"

The original repository goes a bit further: it extracts each judge's ranking line to calculate an average rank for each response and display it next to the tabs. You can add this in a few lines by reading the “RANKING:” line from each output. Keep in mind that smaller models do not always follow this format: provide a fallback for when the line is missing.

#Step 4: adapt the original repository to Ollama

If you want the web interface, the shortest path is to use the OpenAI-compatible endpoint from Ollama, served at http://localhost:11434/v1/chat/completions. The repository backend already sends requests in this format to OpenRouter: in principle, you only need to replace the API address and model identifiers in backend/config.py, then provide a dummy key, with Ollama ignoring the authorization header.

git clone https://github.com/karpathy/llm-council
cd llm-council
uv sync
cd frontend && npm install && cd ..
echo 'OPENROUTER_API_KEY=ollama' > .env

COUNCIL_MODELS = [
    "qwen3:8b",
    "llama3.1:8b",
    "mistral:7b",
]
CHAIRMAN_MODEL = "gemma3:12b"

# Ollama expose une API compatible OpenAI sur /v1
OPENROUTER_API_URL = "http://localhost:11434/v1/chat/completions"

./start.sh
# backend sur le port 8001, interface Vite sur http://localhost:5173 (valeurs du README à vérifier)

Two points still need to be handled manually. First, context: the OpenAI-compatible endpoint has no num_ctx field, so you must either create model variants with a Modelfile that sets the context or switch the client to the native /api/chat API, which accepts options. Second, the parallelism mentioned above: the first step launches calls at the same time, which means switching to sequential execution if your members do not all fit in memory.

cat > Modelfile.qwen-council <<'EOF'
FROM qwen3:8b
PARAMETER num_ctx 16384
EOF
ollama create qwen3-council -f Modelfile.qwen-council

Several community forks have made these changes for Ollama or for other local servers. Before adopting one, check the date of the latest commit, the number of files changed compared with the original repository, and read the code that calls the model: it’s short, and that’s where the differences lie. This guide does not recommend any particular fork; their status changes too quickly.

#Memory and time cost

The calculation is simple and worth laying out before committing. For a council of N members, a complete round requires N generations in stage 1, N generations in stage 2, and one for the chair, for a total of 2N + 1 calls. The stage 2 calls and the chair’s call take longer to process than the first, because the prompt contains all the responses: prompt reading time is added to generation time.

Three members

Seven generations per question. If each response takes about half a minute on your machine, a round takes more than three minutes, not counting reloads.

Minimum memory

The council’s largest model plus its context. With 8B members in Q4 and a context of 16,000 tokens, a 12 GB card is enough, provided you keep only one model loaded.

Comfortable memory

All members loaded at the same time: the sum of their sizes, plus a context cache for each model. Three 8B models in Q4 require a 24 GB GPU or a Mac with 32 GB of unified memory at minimum.

Rechargements

When the members do not fit together, each model change costs a few seconds to load from disk. Over one round, this can happen up to 2N times. An NVMe SSD makes the cost tolerable; a slow drive makes it painful.

These rough estimates depend on your hardware, quantization, and response length: we did not measure round times for this article, and the figures above are calculation rules, not results. To evaluate your case, time a simple generation with each of your members and apply the formula.

#Council limitations and biases

The pattern is appealing because it resembles peer review. Still, resist the idea that it is necessarily more accurate than a good model on its own: the repository provides no measurement to support that, and several known biases of judge models work against it.

Imperfect anonymization

Labels hide the name, not the style. A model may recognize its own writing style, or that of a family it knows, and rank accordingly. Karpathy himself notes, qualitatively, that models have often preferred another model’s response to their own: this is an observation, not a guarantee.

Position and length bias

Research on judge models, such as Zheng et al.’s MT-Bench paper (2023), documents a preference for the first response read and for longer responses. Shuffling the order reduces the first bias; the second is harder to correct.

Correlated errors

Models trained on similar corpora often make the same mistakes. A council of small models can unanimously vote for a wrong answer, and the president will retain it.

The chair remains alone

The synthesis depends entirely on the chair model: if it hallucinates, the council cannot change that. Read the individual responses when the subject matters; that is, in fact, the purpose of the original interface’s tabs.

Small models, long instructions

The ranking prompt is demanding: read three or four texts, judge them, justify the decision, and follow a format. Below 7B, outputs become inconsistent and the average ranking loses its meaning.

#Real-world use cases for a local LLM Council

The overhead of 2N + 1 generations is justified when the question is rare and important, not for casual chatter. Three situations stand out among people who have adapted the pattern locally.

Reviewing sensitive text or code

A contract letter, a procedure, a critical function: members identify different problems, and the president consolidates them into a list. Here, the ranking matters less than the diversity of the feedback.

Reasoned technical decision

“Should we migrate this database?”, “what format should we use for this export?”: divergent answers are information in themselves. Ask the president to make disagreements visible instead of smoothing them over.

Quality control for a single model

Use the council as a jury to compare two prompt versions or two quantizations of the same model by asking it to rank outputs you produced elsewhere. This is a direct application of stage 2, without stage 1.

In all these cases, privacy is the real argument for the local version: the documents being reviewed never leave the machine, whereas the original version sends them to four different providers through OpenRouter. That is what makes the pattern interesting for a team handling internal data, provided it accepts the minutes of waiting.

#Troubleshooting

The ranking skips a response

The context is too short and the beginning of the prompt was truncated. Increase num_ctx (or the Modelfile) and check the Ollama logs to ensure no truncation warning appears.

Stage 1 calls are very slow

The members are called in parallel and compete for memory. Switch to sequential execution or reduce the council to what fits in VRAM at the same time.

The chair answers beside the point

Its context is the most loaded of the round. Give it a larger model or summarize the rankings before passing them on.

Ranking format missing

Small models forget the “RANKING:” line. Repeat the format at the end of the prompt, lower the temperature, or accept manually reading the justifications.

Empty or English member responses

Check that the model does not return only its reasoning phase, and specify the expected language in the question.

Memory full after a few rounds

OLLAMA_MAX_LOADED_MODELS is too high for the machine. Set it to 1 and let Ollama alternate.

#Go further

LLM Council is a deliberation pattern, not an agent framework: it handles neither tools, roles, nor memory. If your needs go beyond synthesizing opinions, these site guides cover the neighboring building blocks.

CrewAI with Ollama

When models need to divide roles and chain tasks rather than judge the same response. https://quelllm.fr/guide/crewai-ollama-multi-agents

LiteLLM as a local gateway

To expose multiple model servers behind a single OpenAI-compatible API, for example a council distributed across two machines. https://quelllm.fr/guide/litellm-proxy-gateway-local

Architecture of a local AI agent

The components to assemble around a model when you want to give it tools and memory. https://quelllm.fr/guide/agent-ia-local-architecture

Share a Ollama server on the local network

Host the council members on a larger machine and call them from your workstation. https://quelllm.fr/guide/ollama-serveur-reseau-local-partage

Guide written on October 12, 2026, based on Karpathy’s GitHub repository and the documentation for Ollama. No timing or quality measurements were conducted for this article; the memory figures are rough estimates in Q4_K_M. Constant, port, and model names change over time: review the sources below before copying the commands.

• GitHub repository karpathy/llm-council (README, backend/config.py, start.sh)
• Documentation Ollama (native API /api/chat, OpenAI compatibility, environment variables)
• Zheng et al., 2023: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (judge-model bias)