Local LLM for Pentesters — Air-Gapped Setup
Last updated 2026-08-18
The models, hardware, and sneakernet workflow to run an offline AI pentest assistant that never leaks a single client byte.
By Mohamed Meguedmi · 9 min read
Key Takeaways
- Air-gap is non-negotiable for engagements. Client scope data, credentials, and exploit output must never touch a hosted API. A local model on isolated hardware is the only defensible architecture.
- A single 24 GB GPU is the practical entry point. An RTX 4090 or RTX 6000 Ada runs a 32B coding model at Q4_K_M with room for a decent context window — enough for real recon and exploit-dev assistance.
- Pick low-refusal, tool-capable models. Qwen3-Coder 32B and security fine-tunes like WhiteRabbitNeo avoid the constant "I can't help with that" walls that break a pentest workflow.
- Transfer via verified sneakernet. Pull models on a staging host, hash them, and carry them across on removable media. Never bridge the two networks, even briefly.
- Ollama for speed of setup, llama.cpp or vLLM for control. All three run fully offline once the weights are local.
Why an air-gapped LLM, and not just "a local one"
"Local" and "air-gapped" are not the same claim. A model running on your own machine can still exfiltrate data through an editor extension, a telemetry callback, or a helpful IDE plugin that syncs to the cloud. For penetration testers, the threat model is stricter: engagement contracts and many compliance regimes (PCI DSS, ISO 27001, government frameworks) treat client credentials, network diagrams, and discovered vulnerabilities as data that must not leave a controlled boundary. A hosted LLM API is a third-party data processor you did not put in scope.
An air gap removes the question entirely. If the inference host has no route to the internet — no Wi-Fi, no bridged NIC, no shared USB with a connected machine — then prompt content physically cannot leak. That is the property you are buying, and it is the reason security teams accept the extra friction of running models offline.
The verdict up front: run a 32B-class coding model at 4-bit quantization on a dedicated 24 GB GPU host with no network interface, load weights through hash-verified removable media, and serve them with Ollama or llama.cpp. Everything below is the detail.
Sizing the hardware
VRAM is the binding constraint. A 4-bit quantized model needs roughly its parameter count in gigabytes plus overhead for the KV cache, which grows with context length. Below is a realistic sizing table for pentest workloads (code analysis, payload drafting, log triage) as of mid-2026. Prices are approximate USD street prices for the compute component only.
| Tier | Hardware | Usable VRAM | Largest comfortable model (Q4_K_M) | ~Cost (USD) |
|---|---|---|---|---|
| Entry | RTX 4090 / 4080 Super | 24 / 16 GB | 32B / 14B | $1,600–$1,900 |
| Pro | RTX 6000 Ada | 48 GB | 70B | $6,800 |
| Dual-GPU | 2× RTX 3090 (NVLink optional) | 48 GB | 70B | $1,600–$2,000 used |
| Unified memory | Mac Studio M3 Ultra 96 GB | ~72 GB usable | 70B–120B | $4,000–$5,500 |
| CPU-only fallback | 64 GB DDR5, no GPU | — | 14B (slow, 3–6 tok/s) | $400 (RAM + board) |
For most solo testers and small teams, the used dual-RTX-3090 configuration is the value leader: 48 GB of VRAM for well under half the price of a single RTX 6000 Ada, at the cost of higher power draw (~700 W under load, roughly 2,400 BTU/h of heat) and a more complex split-tensor setup. If silence and power efficiency matter more than raw throughput, the Mac Studio's unified memory runs 70B models at usable speeds with a ~90 W idle. Model our full parts and running-cost math with the cost calculator.
Which models to run
Pentest work punishes over-aligned models. A general assistant that refuses to write a reverse shell or explain a SQL injection payload is worse than useless during an engagement. You want models that are strong at code, comfortable with offensive-security context, and capable of structured tool use. Our current picks:
- Qwen3-Coder 32B (Q4_K_M). The best all-round choice at 24 GB. Excellent at reading unfamiliar codebases, drafting exploit scaffolding, and explaining decompiler output, with low refusal rates on legitimate security tasks. See the official model card for benchmark detail.
- WhiteRabbitNeo 33B / 70B. A security-focused fine-tune explicitly trained for offensive and defensive cyber use cases. It answers red-team questions directly. Review the license and intended-use notes on its HuggingFace page before deploying.
- Llama 3.3 70B Instruct (Q4_K_M). The strongest reasoning of the group if you have 48 GB. Better for threat-model narratives and report writing than for raw payload generation.
- DeepSeek-Coder-V2 16B. A lighter option for 16 GB cards; fast, code-fluent, and adequate for scripting and log parsing.
| Model | Params | Min VRAM (Q4_K_M) | Best for | Refusal tendency |
|---|---|---|---|---|
| Qwen3-Coder 32B | 32B | ~20 GB | Code review, exploit dev | Low |
| WhiteRabbitNeo 33B | 33B | ~21 GB | Offensive security Q&A | Very low |
| Llama 3.3 70B | 70B | ~40 GB | Reasoning, reporting | Moderate |
| DeepSeek-Coder-V2 16B | 16B | ~11 GB | Scripting, log triage | Low |
Browse the full set of quantizations and measured throughput in the model catalog, and cross-check the numbers against our benchmarks. Every figure on those pages is queryable through the free BestLLMfor public API (CC BY 4.0) or the open-source MCP server, so you can pull model metadata into an offline tooling database before you cut the network cable.
Building the air gap: the transfer workflow
The entire security property depends on never connecting the inference host. That means weights, the runtime binary, and any dependencies arrive by removable media from a separate staging machine. The steps below assume Ollama on Linux, but the pattern is runtime-agnostic.
- On a staging host with internet: pull the model and locate the blob store. For Ollama that is
~/.ollama/models. Download the runtime binary too. - Hash everything. Run
sha256sumover the model blobs and the binary, and record the digests on paper or a separate channel. This is your integrity check against tampering and bit-rot. - Copy to sanitized removable media. Use a dedicated, freshly formatted USB drive or optical disc that is never plugged into a connected machine again after this step.
- On the air-gapped host: mount the media, copy the binary to
/usr/local/bin, restore the model store to~/.ollama/models, and re-runsha256sum. Compare against your recorded digests before doing anything else. - Verify offline operation. Confirm the host has no active network route (
ip routeshould be empty of default gateways), then start the server and run a test prompt.
The critical discipline is one-directional transfer: media flows staging → air-gap, never back, and the two hosts never share a drive in the same session. Ollama's own install docs cover the binary side; the isolation policy is on you.
Serving stack: Ollama vs llama.cpp vs vLLM
All three run fully offline once weights are local. The choice is about control versus convenience.
- Ollama is the fastest path to a working setup. Single binary, GGUF model management, a clean local API on
127.0.0.1:11434. Bind it to localhost only and it exposes nothing. Best for individual testers. - llama.cpp gives you direct control over quantization, context length, GPU layer offload, and sampling. If you need to squeeze a 70B onto tight VRAM or tune KV-cache behavior, this is the tool. The project repository documents every flag.
- vLLM is the pick for a shared team server: paged-attention batching serves multiple concurrent testers with far higher throughput. It wants a beefier GPU and more setup, but for a small pentest team behind one air-gapped box it is the scalable answer.
Whatever you choose, disable any auto-update, telemetry, or model-hub sync feature explicitly. On an air-gapped host these calls simply fail, but a failed outbound call is still a probe you do not want in your logs.
OPSEC and audit trail
Air-gapping the model is necessary but not sufficient. Treat the inference host like any other in-scope asset: full-disk encryption so a stolen drive does not leak engagement data at rest, per-engagement prompt logging you can hand to the client, and a wipe procedure between engagements so one client's data never bleeds into another's session. Keep a written record of every model hash and every media transfer — if a client asks how you guarantee their data never left the boundary, that log is your answer. For the broader trade-offs of self-hosting, our guides hub collects the deployment patterns we recommend.
Verdict
For a solo pentester or a small team, the recommendation is unambiguous: a dedicated 24 GB GPU host running Qwen3-Coder 32B at Q4_K_M under Ollama, weights transferred by hash-verified sneakernet, host physically isolated from any network. It is affordable, fast enough for real engagement work, and defensible under audit. Scale up to 48 GB and a 70B model only when reasoning quality on report-writing becomes the bottleneck.
| Use case | Recommended build | Model | Serving |
|---|---|---|---|
| Solo tester, best value | RTX 4090 24 GB, isolated host | Qwen3-Coder 32B Q4_K_M | Ollama (localhost) |
| Offensive-focused Q&A | 2× RTX 3090 48 GB | WhiteRabbitNeo 33B/70B | llama.cpp |
| Small team, shared box | RTX 6000 Ada 48 GB | Llama 3.3 70B Q4_K_M | vLLM |
| Silent / low-power | Mac Studio M3 Ultra 96 GB | Llama 3.3 70B | Ollama |
| Budget / no GPU | 64 GB DDR5 CPU host | DeepSeek-Coder-V2 16B | llama.cpp |
FAQ
Do I really need an air gap, or is a local firewall enough?
For engagements governed by a contract or compliance regime, a physical air gap is the only architecture that removes the outbound-leak question entirely. A firewall reduces risk but still relies on correct configuration and can be defeated by a misbehaving application. If client data is in scope, air-gap it.
Which single model should I start with?
Qwen3-Coder 32B at Q4_K_M on a 24 GB GPU. It balances code fluency, low refusal rates on legitimate security tasks, and a manageable VRAM footprint. Move to a 70B model only if reasoning quality becomes your limiting factor.
Why do pentesters avoid mainstream aligned models?
Heavily aligned assistants frequently refuse legitimate offensive-security requests — writing payloads, explaining exploits, analyzing malware — which breaks the workflow. Coding-focused and security fine-tuned models like Qwen3-Coder and WhiteRabbitNeo answer these directly within an authorized engagement.
How do I move weights onto the air-gapped host safely?
Download and hash the model on a separate staging machine, copy it to freshly formatted removable media, restore it on the isolated host, and re-verify the SHA-256 digest before use. Media flows one direction only and the two hosts never share a drive in the same session.
Can I run a useful setup without a GPU?
Yes, but expect 3–6 tokens per second on a 14B model with 64 GB of DDR5 RAM. That is workable for occasional scripting and log triage, not for interactive exploit development. A used RTX 3090 transforms the experience for a few hundred dollars.
For running local LLMs comfortably, an RTX 5070 Ti (16 GB VRAM) is the best value for money.
Amazon Check RTX 5070 Ti price →As an Amazon Associate, BestLLMfor earns from qualifying purchases, at no extra cost to you. It does not influence our independent rankings.