Add a reranker to your pipeline

#Reranker: what it does in a RAG pipeline

A reranker takes the question and a passage, reads them together, and returns a relevance score; you then sort the candidates by descending score and pass only the best ones to the language model. In a local RAG pipeline, it sits between the vector database (ChromaDB, Qdrant, Weaviate) and the LLM: vector search retrieves, for example, 30 passages, and the reranker keeps 5. BAAI's official documentation describes it this way: unlike an embedding model, the reranker receives the question and document as input and directly produces a similarity score instead of a vector. The benefit is greatest when the correct answer is among the candidates but not at the top: passages that discuss the right general subject without answering the precise question occupy the top positions, and the model then generates an off-target or fabricated answer. If the correct answer isn't among the candidates, a reranker can't help: it doesn't search; it sorts.

An embedding produces a vector for each document, independently of the question asked, and the distance measures overall topical similarity. So two passages on the same subject can receive similar scores even though only one contains the answer. The cross-encoder looks at the pair as a whole: it can see whether the requested date, name, or condition appears in the passage. It's more precise for this local judgment and more expensive because it requires one passage through the transformer per pair instead of a single computation per document.

#Bi-encoder and cross-encoder: why combine them

Bi-encoder (embedding)

Encode the question and each document separately; document vectors are computed once during indexing, and search is reduced to a vector comparison. Fast and suitable for millions of passages.

Cross-encoder (reranker)

It encodes the question and passage together and outputs a score. No computation can be reused: it must be redone for every question and candidate. Precise, but impossible to apply to the entire corpus.

The Sentence Transformers documentation explains the reasoning: evaluating thousands or millions of pairs would be rather slow, so the retriever is used to produce a set of candidates, such as a hundred, which the cross-encoder re-ranks. This two-stage design is the same as that used by traditional search engines. It also explains the main tuning parameter: the number of candidates retrieved controls both recall—the wider the selection, the more likely it is to contain the right answer—and latency, since each additional candidate requires a pass through the cross-encoder.

#Which reranking model should you choose for French?

The choice comes down to three criteria: language, license, and memory. The size figures below come from the Hugging Face model cards; note that file size depends on weight precision: F32 for bge-reranker-v2-m3 (about 4 bytes per parameter), and F16 for mxbai.

The bge-reranker-v2-m3 card presents it as a lightweight reranker with strong multilingual capabilities that is easy to deploy and fast at inference; the bge-reranker-v2-gemma card targets multilingual contexts, with good results in both English and multilingual settings. Two useful corrections to what is often reported: the bge-reranker-v2-m3 file weighs more than 2 GB, not 560 MB (568 million parameters in F32), and mxbai-rerank-large-v1 is an English-language model and should not be chosen for French documents. Once loaded in half precision, the m3 model uses approximately 1.1 GB for the weights (568 million parameters × 2 bytes, calculated in this guide), plus the activations for the processed batch.

#The pipeline before and after the reranker

AVANT :
  Question → Embedding → Base vectorielle (top-5) → LLM

APRÈS :
  Question → Embedding → Base vectorielle (top-20 à top-50)
                       → Reranker (top-5) → LLM

On récupère large, puis on ordonne finement.

Two parameters govern the whole system: k_retrieve, the number of candidates returned by the vector database, and k_final, the number of passages sent to the LLM. A k_final of 3 to 5 suits most local models with 7 to 14 billion parameters; beyond that, you fill the context window without a clear benefit, and prompt-processing time increases. k_retrieve depends on corpus difficulty: 20 is a good first try, 50 to 100 if questions are vague or the corpus contains many similar passages. The context-window guide explains the cost of adding more passages to the prompt.

#Implementation: sentence-transformers, FlagEmbedding, llama.cpp

#With sentence-transformers

The CrossEncoder class loads the model and evaluates pairs. Its rank method directly accepts the question and document list and returns the best ones; the top_k parameter limits the number of results (without it, all documents are returned).

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def retrieve_and_rerank(question, k_retrieve=20, k_final=5):
    # 1. Récupération par embedding (ChromaDB, Qdrant, etc.)
    candidats = embedding_search(question, top_k=k_retrieve)  # liste de textes

    # 2. Scoring par le cross-encoder, tri et coupe en une seule étape
    resultats = reranker.rank(question, candidats, top_k=k_final, batch_size=16)
    # resultats = [{'corpus_id': 3, 'score': 0.91}, ...]
    return [candidats[r['corpus_id']] for r in resultats]

#With FlagEmbedding, the model authors' library

The model card uses the FlagEmbedding library. It states that the raw score can be mapped to a value between 0 and 1 using a sigmoid function with normalize=True, and that use_fp16=True speeds up computation at the cost of a slight quality reduction. Remember that the raw score has no absolute scale and is often negative for irrelevant passages; only the ranking matters unless you set a threshold.

from FlagEmbedding import FlagReranker

reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True)
score = reranker.compute_score(['ma question', 'un passage'], normalize=True)  # entre 0 et 1

#With llama.cpp, without Python

The llama.cpp server provides a reranking endpoint, disabled by default. The documentation says it requires a reranking model, cites bge-reranker-v2-m3 as an example, and runs with the --embedding and --pooling rank options. You need a GGUF version of the model. This is the option to choose if your stack is already built around llama.cpp and you want to avoid installing PyTorch. Check the exact option in the installed version: the documentation warns that this endpoint may change.

llama-server -m bge-reranker-v2-m3-Q8_0.gguf --embedding --pooling rank --reranking --port 8081

#With LlamaIndex

from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(model="BAAI/bge-reranker-v2-m3", top_n=5)

query_engine = index.as_query_engine(
    similarity_top_k=20,
    node_postprocessors=[reranker],
)

#Cost: latency, memory, passage length

No latency figure is guaranteed: it depends on the card, precision (FP16 or FP32), number of candidates, and passage length. Focus on the proportions. Reranking time grows linearly with the number of pairs: going from 20 to 100 candidates multiplies the workload by five. It also grows with passage length, since each pair is encoded in full. Measure on your machine with your passages instead of relying on a number from a blog: time one hundred real queries and look at the median and worst-case times.

Number of candidates

First lever. Start at 20, measure recall, and increase to 50 only if good answers are still being left out.

Accuracy

use_fp16=True (FlagEmbedding) or loading in half precision reduces memory usage and speeds up computation, with a slight performance drop depending on the model card.

Batch size

The sentence-transformers rank method processes 32 pairs per batch by default. Reduce this to 8 or 16 if memory is tight; increase it if the GPU is underutilized.

Maximum length

512 tokens per pair (the max_length value in the official examples). A longer passage is truncated: if your chunks exceed this size, the end of the passage isn't read. Shorten the chunks before increasing the limit.

CPU or GPU

On CPU, the reranker works, but each request with 20 to 50 candidates takes seconds; that's acceptable for an internal document assistant, less so for interactive chat.

#Reranking and chunking: the two settings interact

A cross-encoder judges an entire passage. If the passage mixes three topics, its score will be average for the three corresponding questions; if it is too short, it loses the context that would allow it to be recognized as an answer. Splitting text into medium-sized passages with slight overlap gives the reranker enough material without exceeding its maximum length. If you change the chunk size, rerun the recall test: the best k_retrieve and the reranker gain change with it. The guide to chunking strategies explains these choices in detail.

Another interaction: hybrid search. When keyword search (BM25) and vector search are combined, the candidate pool is more diverse, giving the reranker more chances to find a good answer than either method alone would have had. The reranker is the final layer, and hybrid search is the second: they complement rather than replace each other.

#Measure the improvement on your documents

The improvement reported in blogs ranges from twofold to fivefold depending on the corpus, and no general figure applies to yours. On a highly structured corpus (clean technical documentation), vector search alone is already good; on a noisy corpus (emails, notes, poorly extracted PDFs), the gap is more pronounced. The only way to know is to evaluate it.

1. 01
   Prepare 30 to 50 questions

   Use real user questions, each with the passage containing the answer (an ID is enough).
2. 02
   Measure recall at 5 without a reranker

   For each question, check whether the relevant passage appears in the vector database's top 5 results.
3. 03
   Measure recall at 5 with a reranker

   Retrieve 20 candidates, rerank them, and count the relevant passages again among the top 5.
4. 04
   Compare latency as well

   Record the end-to-end median time. A gain of a few recall points does not necessarily justify an additional second of waiting.
5. 05
   Reviewing failures

   For each missed question, check whether the correct answer was among the 20 candidates. If not, the problem is upstream: chunking, embeddings, or text extraction.

def rappel_a_k(pipeline, questions, cibles, k=5):
    ok = 0
    for q, cible in zip(questions, cibles):
        ok += cible in [p.id for p in pipeline(q)[:k]]
    return ok / len(questions)

sans = rappel_a_k(pipeline_sans_reranker, questions, cibles)
avec = rappel_a_k(pipeline_avec_reranker, questions, cibles)
print(f"Sans : {sans:.0%}   Avec : {avec:.0%}")

#Do you need a reranker? Decision guide

#Frequently asked questions about reranking

• Hybrid search: BM25 + vector search
• Chunking strategies
• The best French embedding models
• Understanding the context window
• Local RAG with ChromaDB and Ollama

• Source: BAAI/bge-reranker-v2-m3 Hugging Face page
• Source: BAAI/bge-reranker-v2-gemma specifications
• Source: Sentence Transformers, Retrieve & Re-Rank
• Source: llama.cpp server documentation