AnythingLLM: Chatting With Your Own Documents, Privately

What it is

Retrieval-augmented generation has four moving parts — ingest, embed, store, retrieve — and assembling them yourself means choosing a parser, an embedding model, a vector database and a retrieval strategy before you can ask a single question. AnythingLLM ships a default for each and hides the wiring. You install it, drop in a folder of PDFs, and ask.

What you keep is the ability to swap most of those defaults later: a different chat model, a different embedding model, an external vector database if the built-in one stops being enough. What you give up is fine-grained control of the retrieval logic itself. For the background on what is happening under that interface, what RAG is is the short version.

Workspaces: the part that matters

A workspace holds documents, a system prompt, a model choice and its own conversation threads. This sounds like organisation and is actually about accuracy: retrieval quality falls as a corpus grows more heterogeneous, because the passages nearest your question start coming from unrelated material. Three focused workspaces beat one workspace with everything in it, every time.

The practical rule is one workspace per question domain, not per file type. "Supplier contracts", "product documentation", "internal HR policies" — each with its own system prompt telling the model what it is looking at and how to answer.

Desktop or Docker

	Desktop app	Docker server
Who it is for	One person, one machine	A team, a shared URL
Accounts and permissions	No	Yes, multi-user
Where documents live	Your machine	A volume on the server
Bundled model runtime	Yes, works with zero setup	You point it at your own
Effort	Install and go	Compose file, volumes, backups

The classic mistake with the Docker deployment is mounting nothing: the container starts, you ingest a hundred documents, and a restart wipes them. Persist the storage volume before the first upload, not after.

Choosing the two models

Every document-chat tool needs two models, and conflating them is the most common cause of bad answers:

• The chat model writes the answer. A 7B–8B model is enough for extraction and summarising from provided text; step up to 14B–27B when the questions require reasoning across passages. Sizing guidance lives in VRAM requirements.
• The embedding model decides what gets retrieved at all. It is small, fast, and it is where quality is won or lost — especially for non-English documents, where a multilingual embedding model is not optional.

Changing the embedding model invalidates every vector already stored. Re-embedding a workspace is the correct move, and doing it deliberately early beats discovering it after ingesting ten thousand pages.

The three settings that decide answer quality

1. Chunk size and overlap. Too small and a chunk loses the context that makes it meaningful; too large and each retrieved chunk wastes context window on irrelevant text. Overlap prevents a sentence that straddles a boundary from disappearing.
2. Number of retrieved chunks. More is not better: every extra chunk is noise the model must ignore, and a long context costs memory.
3. The similarity threshold. The quiet one. Set it too high and genuinely relevant passages are filtered out before the model sees them, producing a confident "the documents do not mention this" about something on page four. When answers claim ignorance about content you know is there, this is the first setting to look at.

Limits worth knowing

• Scanned PDFs index as empty. No text layer, nothing to embed. Run OCR first.
• Tables and layout suffer. Complex documents flatten into linear text; numbers can lose the row they belonged to.
• It does not think across a whole corpus. "Summarise all 200 contracts" is not a retrieval question — top-k passages cannot represent 200 documents.
• Privacy is a configuration, not a property. The app supports cloud providers too. If the requirement is that nothing leaves, set local providers for both models and verify.

AnythingLLM or something else

You want	Better fit
Document chat, minimum setup, one person	AnythingLLM desktop
A general chat front end with many features	Open WebUI or LibreChat
Control over chunking, embeddings and retrieval	Your own pipeline — see the RAG explainer
A platform to build several apps with APIs	Dify

Verdict

AnythingLLM is the shortest honest path from "I have documents" to "I can ask them questions", and the workspace model keeps it usable as the number of corpora grows. Treat the defaults as a starting point: pick a multilingual embedding model if your documents are not in English, keep workspaces narrow, and check the similarity threshold the first time the assistant claims your documents do not mention something they clearly do.