Guide · 2026-05-16

HIPAA-Compliant Local LLM Stack — What You Actually Need

What HIPAA actually requires from a local LLM deployment in 2026—hardware, models, audit logs, and the controls that survive an OCR audit.

By Mohamed Meguedmi · 9 min read

What HIPAA Actually Requires (And What It Doesn’t)

The HIPAA Security Rule (45 CFR §164.302–318) is a control framework, not a software certification. There is no “HIPAA-certified” LLM, and any vendor claiming otherwise is either confused or selling a Business Associate Agreement (BAA) wrapped around generic cloud infrastructure. The rule defines three control families—Administrative, Physical, and Technical—and the technical safeguards in §164.312 are what intersect with your stack: unique user identification, automatic logoff, encryption in transit and at rest, integrity controls, and audit logs.

Running a local LLM solves one specific problem: it removes the cloud LLM vendor from the BAA chain. That’s the only thing it solves on its own. You still owe every other control. Most public guides—including the otherwise solid HIPAA Vault overview—gloss over §164.312(b), the audit-controls requirement that breaks 80% of LLM proofs-of-concept the moment a privacy officer sees the architecture diagram.

The Five-Layer Reference Architecture

A defensible local LLM deployment has five layers. Skip any one and you fail at audit.

Layer	Component	What it satisfies
1. Hardware + OS	Hardened Linux, LUKS2 full-disk encryption, IOMMU, dm-integrity	§164.312(a)(2)(iv), §164.310
2. Inference engine	vLLM 0.6+ or llama.cpp server, Unix-socket only	§164.312(e)(1)
3. Gateway / proxy	LiteLLM or Envoy with mTLS 1.3, per-team rate limits	§164.312(a)(1), (d)
4. Identity + audit	Keycloak SSO + structured audit sink (Postgres or WORM bucket)	§164.312(a)(2)(i), (b)
5. Application	Open WebUI fork, custom RAG, or n8n flows behind the gateway	§164.308 administrative safeguards

Notice what is not on this list: a dedicated “HIPAA module” or a paid certification add-on. Every control is implemented in open-source components configured correctly.

Hardware: What Runs Clinical Workloads in 2026

Four credible single-node options in May 2026, ordered by price/performance for healthcare text workloads.

Configuration	VRAM	Models supported (Q4_K_M)	Tokens/sec (70B Q4)	Street price (USD)
1× RTX 4090	24 GB	Up to 34B at Q5; 70B with CPU offload	9–14	$1,700–$1,900
1× RTX 5090	32 GB	70B Q4 fully in-VRAM	28–34	$2,400–$2,900
2× RTX 4090 (NVLink)	48 GB	70B Q5/Q6, 120B Q4 with offload	22–26	$3,800–$4,200
1× RTX 6000 Ada	48 GB ECC	70B Q5 fully in-VRAM, ECC memory	20–24	$6,800–$7,400

For clinical documentation, prior-authorization drafting, ICD-10 code suggestion, and structured note summarization, a single RTX 5090 is the sweet spot. ECC memory on the RTX 6000 Ada matters for long-running batch inference where bit-flips in the KV cache could silently corrupt PHI summarization—HIPAA’s integrity requirement (§164.312(c)) doesn’t mandate ECC, but a Security Rule auditor will ask why you didn’t use it.

The often-cited recommendation that a 7B model on a $1,700 GPU is enough is technically true and clinically dangerous. Don’t deploy below 32B parameters for any PHI-touching workload. Our cost calculator breaks down total cost of ownership including power draw (300–600 W sustained) and amortization—a $9,000 single-node stack pays for itself versus a HIPAA-eligible cloud API within ~14 months at 2 million tokens per day.

Model Selection: Llama 3.3, Qwen2.5, and the De-identification Trap

Three open-weight models dominate the May 2026 leaderboards for clinical text:

• Llama 3.3 70B Instruct — Meta’s flagship open-weight generalist. Strong at structured output, average at numeric reasoning. Model card on HuggingFace.
• Qwen2.5-72B Instruct — Alibaba’s model edges Llama on long-context retrieval (128K tokens). Stronger at code, equal at clinical summarization. Model card on HuggingFace.
• MedGemma 27B — Google’s domain-tuned model. Smaller footprint, narrower scope. Useful as a router or pre-filter, not as your primary clinical assistant. MedGemma on ollama.com.

A note on the de-identification myth: running a local model does not exempt you from §164.514. If you process PHI, you process PHI, regardless of where the GPU sits. The Safe Harbor method requires removal of 18 specific identifiers, and no current open-weight model reliably does this end-to-end. Use Microsoft Presidio or a dedicated NER pipeline before any prompt that doesn’t strictly need identifiers, and document the residual-risk analysis required by §164.514(b)(2).

The Audit Trail Problem No One Talks About

§164.312(b) requires the covered entity to “implement hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use ePHI.” For an LLM, “activity” means every prompt, every response, every retrieval hop, every tool call—linked to a unique authenticated user and a tamper-evident timestamp.

Ollama does not provide this. llama.cpp’s built-in server does not provide this. vLLM has request logging but no immutable sink. You need a gateway that sits in front of inference and writes structured logs to an append-only store. LiteLLM with a Postgres backend, configured with pgaudit and row-level immutability, is the minimum. Append a SHA-256 hash chain or push to a WORM-mode object store (MinIO with object lock works fully air-gapped) if your privacy officer wants belt-and-suspenders evidence.

An audit log that an admin can DELETE FROM is not an audit log. Either the storage engine enforces immutability, or you have an undocumented compliance gap.

The BestLLMfor public benchmarks dataset (CC BY 4.0, available via our public API) ships with a reference LiteLLM + Postgres docker-compose used by several pilot deployments. Our methodology page documents how we validate audit-log immutability claims under the same threat model an OCR investigator would apply.

Production Stack: Components and Costs

The stack we recommend for a single-tenant clinical pilot serving 5–50 internal users:

• OS: Ubuntu Server 24.04 LTS or RHEL 9.x with FIPS mode, LUKS2 + dm-integrity on the data volume.
• Inference: vLLM 0.6.x in a hardened container, served over Unix socket only (no public TCP listener).
• Gateway: LiteLLM proxy with OIDC auth, per-team rate limits, Postgres audit sink, request/response hashing.
• Identity: Keycloak 25.x federated against existing AD or Okta. Mandatory 2FA, 15-minute idle timeout, unique account per human (no shared service accounts touching PHI).
• RAG / vector store: Qdrant or Postgres + pgvector. Encrypt collections at rest, never co-locate with anything that lacks the same access controls as the inference node.
• Observability: Loki + Grafana for non-PHI metrics; the Postgres audit log is the only store that sees prompt or response text, behind separate credentials.
• Backup: Restic to MinIO with object lock, AES-256, off-site replication to a second BAA-covered facility.

Approximate first-year cost for a 25-user clinical pilot:

Line item	Cost (USD)
GPU + server (RTX 5090, EPYC, 256 GB RAM, 4 TB NVMe)	$8,500
Networking, UPS, rack space (12 mo)	$1,200
Power (600 W × 24/7 × 12 mo @ $0.14/kWh)	$735
Engineering (40 h compliance + 60 h build @ $180/h)	$18,000
Annual third-party risk analysis	$6,000
Total year one	~$34,400

Compare against a HIPAA-eligible cloud LLM at 2M tokens/day: roughly $4,500/month or $54,000/year for equivalent throughput, with the bonus that your PHI leaves the building under a BAA you cannot fully audit.

Deployment Checklist

The condensed runbook our review team uses when validating a deployment:

1. Sign every upstream BAA. Cloud backup provider, monitoring vendor, on-prem hardware support contract—any third party that could touch PHI metadata.
2. Run a §164.308(a)(1)(ii)(A) risk analysis. Document threats, likelihood, impact. This is the document an OCR investigator reads first.
3. Deploy hardware to a controlled-access room. §164.310(a)(1). Logged badge entry, no shared admin account, no shared keyboard.
4. Encrypt at rest (LUKS2 + dm-integrity) and in transit (mTLS 1.3). Disable TLS 1.2 below the AEAD cipher-suite floor.
5. Configure SSO + 2FA + 15-minute auto-logoff on every entry point. No exceptions for service accounts that handle PHI.
6. Pull model weights once, hash them with SHA-256, store the hash. Re-verify on every container start.
7. Route 100% of inference through the audited gateway. Block direct port access at the host firewall and in nftables, not just at the application layer.
8. Run a tabletop breach exercise before go-live. Document the response runbook, including OCR notification timelines.
9. Schedule quarterly access reviews and an annual external audit. Calendar them now; the calendar invite is the control.

The open-source quelllm-mcp gateway (MIT-licensed, maintained alongside our French sister site quelllm.fr) implements step 7 as a drop-in MCP-compatible audit gateway. It sits in front of vLLM or Ollama without code changes and emits the structured log shape §164.312(b) auditors expect.

Final Verdict

Use case	Recommended stack	Year-one budget
Solo clinician / 1–5 user pilot	RTX 4090 + Llama 3.3 70B Q4 + LiteLLM + Postgres	$12K–$18K
Clinic / 5–50 users	RTX 5090 + Qwen2.5 72B Q4 + vLLM + Keycloak + WORM audit sink	$30K–$40K
Hospital department / 50–500 users	2× RTX 6000 Ada + vLLM cluster + Qdrant + Loki + S3 Object Lock backup	$80K–$150K

A “HIPAA-compliant local LLM stack” is not a product you buy—it is a set of controls you implement around inference. Pick a 32B+ open-weight model, an inference engine with structured logging, a gateway that enforces auth and writes immutable audit trails, and an identity layer that ties every prompt to a named user with 2FA. Skip the audit layer and your deployment is a §164.312(b) violation waiting for its first complaint-driven OCR letter.