FAISS: Vector Search as a Library, Not a Server

Library, not database

Most discussions compare FAISS to vector databases as if they were alternatives. They are not the same category. A vector database is a service with an API, storage, filtering and access control. FAISS is a component, written in C++ with complete Python and NumPy wrappers: you hand it vectors, it builds an index in memory, and it answers "which of these are nearest to that." Several vector databases use it, or something like it, internally. The project is MIT-licensed, had more than 41,000 GitHub stars at the end of September 2026, and its own README states that some of its methods "can scale to billions of vectors in main memory on a single server."

The practical consequence is that choosing FAISS means accepting responsibility for the rest: saving the index to disk, reloading it, keeping it in sync with your documents, mapping vector positions back to the text they came from, and deciding what happens when two processes want to write at the same time. None of that ships by default — it is a deliberate design choice, not an oversight.

Getting running in Python

1. Install it — pip install faiss-cpu pulls the official PyPI package (version 1.15.1 as of late September 2026); the project's own install guide documents a conda path for the GPU build: conda install -c pytorch -c nvidia -c conda-forge faiss-gpu=1.15.1.
2. Build an exact index — index = faiss.IndexFlatL2(d), per the project's own getting-started guide, where d is your embedding dimension.
3. Add vectors — index.add(xb), a NumPy array of shape (n, d) in float32.
4. Search — D, I = index.search(xq, k) returns distances and indices for the k nearest neighbors; mapping I back to your source text is on you.
5. Persist — write_index(index, "large.index") to save, read_index("large.index") to reload, since FAISS does not persist anything on its own.

The index types that matter

Index	How it searches	Use it when
Flat (exact — IndexFlatL2 / IndexFlatIP)	Compares against every vector	Up to tens of thousands of vectors — exact results, no tuning, genuinely fast enough
IVF	Partitions the space, searches a few partitions	Hundreds of thousands and up; needs a training step on representative data
HNSW	Navigates a graph of neighbors	Plenty of RAM or a modest corpus: fast and accurate, but no vector removal
PQ / OPQ (product quantization)	Stores compressed vectors as M-byte codes	The index no longer fits in RAM; accuracy drops, memory drops far more
RaBitQ (maximum compression)	Compresses to roughly 1 bit per dimension plus overhead	The last resort for memory, with a random-rotation preprocessing step for accuracy

The advice that saves the most time: start with exact search. The only indexes that guarantee exact results are IndexFlatL2 and IndexFlatIP, per FAISS's own documentation; they need no tuning, no training step, and no recall to measure. Approximate indexes exist to solve a scale problem, and adopting them before you have that problem buys tuning parameters and a training step in exchange for milliseconds nobody noticed. At local-RAG scale, retrieval is almost never the slow part; generation is.

Choosing an index by dataset size

FAISS's own wiki publishes concrete thresholds, expressed as strings you pass to its index factory. Below one million vectors, IVF_K_ is enough, with K chosen between 4×√N and 16×√N for N vectors, and a training set of 30×K to 256×K vectors. From one to ten million, the recommended combination is IVF65536_HNSW32, which uses HNSW to speed up cluster assignment during search. From ten to a hundred million, IVF262144_HNSW32; beyond that, up to a billion, IVF1048576_HNSW32 — at that scale, training slows down markedly and typically runs on GPU while everything else stays on CPU.

Corpus size	Recommended configuration
Under 1 million	IVF_K (K between 4×√N and 16×√N)
1–10 million	IVF65536_HNSW32
10–100 million	IVF262144_HNSW32
100 million–1 billion	IVF1048576_HNSW32

For a local document corpus — a few thousand to a few hundred thousand text chunks — these thresholds mostly confirm how far you are from needing an approximate index at all: exact search, or at worst a plain IVF_K, covers nearly every real case, and the training sets required at the higher tiers are far beyond typical local document use.

Memory, honestly

A vector of 1,024 dimensions in 32-bit floats occupies about 4 KB. A million of them is roughly 4 GB, before the index structure itself. For an HNSW index specifically, FAISS's own wiki gives the formula (d*4 + M*2*4) bytes per vector, where d is the dimension and M the number of links per vector (4 to 64: more links means more accuracy and more memory). That arithmetic decides most architectures: it is why compression exists, and why a machine that also holds a language model in its GPU memory has less room than people assume.

On compression: product quantization (PQ) encodes each vector into M-byte codes, typically 64 bytes or fewer — beyond that, scalar quantization (SQ) is usually about as accurate and faster. When compression quality really matters, the official guide recommends an OPQ transform first — a linear transformation to a reduced dimension that makes vectors easier to compress — before applying PQ on the result; it costs an extra step at indexing time but reduces accuracy loss compared to PQ alone at the same code size. RaBitQ, the maximum-compression option, brings that down to roughly (d/8 + 8) bytes per vector by keeping just one bit per dimension, at the cost of a random-rotation preprocessing step needed for decent accuracy; multi-bit variants trade some of that storage saving back for more precision.

The missing feature that decides most projects

Real questions carry conditions: only this client's documents, only after that date, only what this user may read. FAISS has no concept of metadata. The usual workaround — retrieve more results than needed, then filter in Python — is wrong in a specific way: if the top 50 all belong to another department, filtering leaves you with nothing, and your assistant reports that it found no information rather than that it found none you can see.

Permission filtering in particular should not be implemented downstream of retrieval. That is the strongest practical argument for a system that filters during the search — a vector database, or vectors inside PostgreSQL, where it is a WHERE clause.

When FAISS is the right call

• A single-process application that loads an index at startup and queries it — a desktop tool, a batch job, a notebook.
• A fixed corpus that is rebuilt on a schedule rather than updated continuously.
• No per-user filtering, or filtering so coarse that one index per category is reasonable.
• Latency-critical embedded search where the cost of a network hop to a database is the thing you are trying to remove.

Outside those cases, the service you avoid installing is usually cheaper, over time, than the persistence, filtering and concurrency code you end up rewriting as the project grows. Worth remembering too: several vector databases you will run into elsewhere do not replace FAISS by magic — they wrap it, or lean on the same index families (IVF, HNSW, product quantization) behind a network API, managed persistence and a filtering layer. Understanding FAISS is, to a large extent, understanding how vector databases work internally, even when the final project reaches for pgvector instead of FAISS directly.

Verdict

FAISS is excellent at exactly what it claims: fast nearest-neighbor search over vectors already in memory. In a local RAG it is the right choice for single-process, fixed-corpus, no-filtering situations — and the wrong one the moment several users, live updates or permissions enter the picture. Start with an exact index, measure before reaching for an approximate one, and move to a database — pgvector for a relational stack — when you catch yourself writing one. Readers building the rest of the stack can start from our local RAG toolkit.

Sources

• FAISS repository on GitHub — license, stars, README
• Guidelines to choose an index — official thresholds and memory formulas
• Getting started — official Python quickstart code