The confidential pro — “My files never leave my workstation.”
Lawyer, clinician, HR, consultant, accountant — your documents are covered by privilege, NDA, or plain common sense. A local LLM paired with retrieval over your own files lets you query, summarize and cross-reference them without uploading anything, anywhere.
By the end, you'll have a private chat interface running locally, a model sized for document analysis, and a folder of your own files indexed and queryable in plain English — with a way to actually verify that nothing leaves your machine.
Who this tour is for
We'd rather tell you upfront than let you lose half an hour for nothing.
- ✓Your documents are covered by professional privilege or an NDA.
- ✓You handle personal data subject to GDPR, HIPAA or similar — avoiding cross-border transfer is a requirement, not a nice-to-have.
- ✓You want a clean graphical interface, not a terminal command.
- ✓You have anywhere from a handful to thousands of PDFs you'd rather not re-upload every session.
- ·Your data is public and you just want a general assistant: see the beginner tour instead.
- ·You manage terabytes of documents for a whole team: that's a server deployment, not a single workstation — this tour covers the individual setup.
- ·You need real-time answers over hundreds of queries a second: a simple local RAG setup won't hold up at that scale.
The path in 6 steps
Each step links to a real, detailed guide. The order is deliberate — don't skip steps the first time through.
-
1Step 01 · 5 min
Why (and how) to replace ChatGPT at work
The case for keeping sensitive prompts off third-party servers, and a realistic picture of what a local replacement can and can't do.
You know what a credible local swap-in for ChatGPT actually looks like.Read the guide → -
2Step 02 · 4 min
Pick a model built for document Q&A
The current ranking of local models for retrieval-augmented generation over your own files, not just open chat.
You've shortlisted a model sized for your document workload.Read the guide → -
3Step 03 · 4 min
Choose your quantization
How much quality survives compression, and how to balance file size against accuracy for long documents.
You know which quantized weights to run.Read the guide → -
4Step 04 · 15 min
Set up a private chat + document interface
A polished, no-terminal interface with a built-in Documents feature for indexing and querying your own PDFs.
You can point it at a folder of documents and query them in natural language, with cited sources.Read the guide → -
5Step 05 · 6 min
Verify you're actually compliant
What “GDPR-compliant local LLM” means in practice, and the checklist to confirm nothing leaves your machine.
You can state precisely why this setup avoids cross-border data transfer.Read the guide → -
+Step bonus · 5 min
Bonus — stricter regimes (HIPAA)
If you handle health data specifically, the additional checklist for HIPAA-grade isolation.
You know the extra steps required for stricter compliance regimes.Read the guide →
The recommended models
Three picks for this tour, lightest to most capable. Click through for the full spec sheet — VRAM, context, benchmarks.
14 GB at Q4. Strong at long-document summarization and structured analysis — the safe default for this tour.
See the page →9 GB at Q4 — comfortable on more modest hardware, still very capable at structured document analysis.
See the page →For a 64 GB Mac or a 48 GB-VRAM workstation. Meaningfully stronger reasoning, at real hardware cost.
See the page →Frequently asked
The real questions readers send in. If yours is missing, reach out via Contribute below.
Is this actually GDPR / HIPAA compliant?
By design, no data is transmitted to a third-party processor — you remain the sole data controller. You still need to handle disk encryption (BitLocker, FileVault, LUKS) and a sensible conversation-retention policy yourself.
How does a local 24B model compare to GPT-4o or Claude for this?
For factual Q&A grounded in your own documents (RAG), a well-set-up 24B model is more than enough for most professional use — summarization, extraction, clause comparison. For open-ended reasoning with no source documents, cloud frontier models still lead.
How many documents can this realistically handle?
A local vector database comfortably handles thousands of documents on a modern SSD. Past tens of thousands of chunks, retrieval quality starts to need a reranker to stay precise — worth knowing before you scale this to a whole archive.
What if several people at the firm need this?
Two realistic options: everyone runs their own workstation setup (simplest, fully segregated data), or one internal server the team connects to (more efficient past 4–5 people, more setup). This tour covers the first.
Does it actually cite its sources?
Yes — a properly configured RAG setup shows the exact document and passage it drew from. That's essential for professional use: you can verify the answer wasn't invented.
What if the model still gets something wrong?
Three levers that help: lower the temperature (more literal, less creative), use a strict system prompt that limits it to the provided excerpts, and add a reranker so only the genuinely relevant passages reach the model.
The beginner — “I just want to try it, no fuss.”
You've heard about local LLMs and you want to see how they run on your own machine. No code, no complicated command line — by the end you'll be chatting with your first m…
A question, a typo, a bug?
This tour evolves with every model release. Your feedback is the raw material.