Re-Ranking

Vector search and keyword search return a ranked list of documents. But the ranking is approximate: cosine similarity tells you documents are semantically close, not that they answer the question. A document about "JWT security best practices" is semantically close to a query about "JWT expiration handling," but the document that actually explains how to set the exp claim is more relevant. The vector score can't distinguish them.

Re-ranking fixes this. After initial retrieval returns the top 20-50 candidates, a cross-encoder model reads each (query, document) pair and scores actual relevance. The top results get reordered. It costs more per query but the relevance gain is substantial.

What you'll learn

• Bi-encoders (the models that produce embeddings) compare query and document in separate passes; cross-encoders read both together and capture word-level interaction that embeddings miss
• Re-ranking the top 20-50 results with a cross-encoder costs cents per query and reliably pushes the most relevant document into position 1-3
• Re-ranking can't fix bad retrieval. If the top-50 results are all irrelevant, no cross-encoder can find a good answer -- fix retrieval first, then re-rank

The problem: approximate ranking isn't enough

Bi-encoder similarity (the cosine distance between two embedding vectors) captures broad topical overlap. It's fast because you can pre-compute document embeddings and compare them with a single vector operation at query time. But the model never sees the query and document side by side. It cannot notice that the query asks "how to revoke a JWT" while the document explains "how to issue a JWT" -- both live in the same semantic neighborhood.

A cross-encoder takes the full (query, document) pair as input and outputs a single relevance score. It reads the interaction: "the query asks about revocation, but this document is about issuance, so relevance is low." This is more expensive -- you run the model on every candidate pair -- but the ranking is substantially better.

Options & when to use each

Option	Best for	Costs you	When to pick
Cohere Rerank API	Quick integration, strong out-of-box performance	Paid API, ~$1 per 1,000 searches (top-20); latency of ~200ms	You want a managed service and don't want to host a model
mixedbread-ai/mxbai-rerank	Open-source cross-encoder, runs locally	GPU or sufficient CPU RAM; ~200ms per 20 pairs on CPU	You need offline or self-hosted re-ranking
BAAI/bge-reranker-v2-m3	Multilingual re-ranking	Larger model (~568M params); needs GPU for low latency	Your corpus has documents in multiple languages
No re-ranking	Latency-sensitive applications, simple factoid lookup	Lower relevance for nuanced or comparative queries	Vector similarity alone already meets your relevance bar

For this lesson we use the Cohere Rerank API because it requires no model hosting and has a clean Python client. The same pattern works with any cross-encoder.

Build it: re-ranking with Cohere Rerank

Step 1: install and configure

Here is a concrete example demonstrating the implementation details.

pip install cohere

Here is a concrete example demonstrating the implementation details.

import os
import cohere

co = cohere.ClientV2(os.environ["COHERE_API_KEY"])

Step 2: retrieve, then re-rank

Here is a concrete example demonstrating the implementation details.

def retrieve_and_rerank(
    query: str,
    top_k_retrieval: int = 20,
    top_k_final: int = 5,
) -> list[dict]:
    """
    Retrieve top_k_retrieval documents, then re-rank
    to return only the top_k_final most relevant.
    """
    # Step 1: initial retrieval (hybrid search from previous lesson)
    candidates = hybrid_search(query, top_k=top_k_retrieval)

    if not candidates:
        return []

    # Step 2: re-rank with cross-encoder
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[doc["content"] for doc in candidates],
        top_n=top_k_final,
    )

    # Step 3: map scores back to the original documents
    reranked = []
    for result in response.results:
        doc = candidates[result.index]
        reranked.append({
            "id": doc["id"],
            "content": doc["content"],
            "relevance_score": result.relevance_score,
            "original_rrf_score": doc["rrf_score"],
        })

    return reranked

The relevance_score from Cohere is a float between 0 and 1. Scores above 0.5 generally indicate the document addresses the query. Scores below 0.3 are usually off-topic. You can use these thresholds later for failure handling.

Step 3: compare before and after

Here is a concrete example demonstrating the implementation details.

query = "How do I set the expiration time on a JWT token?"

# Before re-ranking
candidates = hybrid_search(query, top_k=5)
print("Before re-ranking:")
for i, doc in enumerate(candidates):
    print(f"  {i+1}. (RRF={doc['rrf_score']:.3f}) {doc['content'][:100]}")

# After re-ranking
reranked = retrieve_and_rerank(query, top_k_retrieval=20, top_k_final=5)
print("\nAfter re-ranking:")
for i, doc in enumerate(reranked):
    print(f"  {i+1}. (score={doc['relevance_score']:.3f}) {doc['content'][:100]}")

You'll typically see the most relevant document move from position 2-5 up to position 1, and at least one document that is semantically close but factually wrong drop out of the top 5 entirely.

The cost-quality tradeoff

Each re-ranking call processes top_k_retrieval document pairs. The Cohere Rerank API charges per search unit (roughly per 100 documents processed). For a typical setup with top_k_retrieval=20:

Searches per day	API cost (approx)	Latency added
100	$0.10/day	~200ms
10,000	$10/day	~200ms per query
100,000	$100/day	~200ms per query (parallelize)

If cost is a concern, apply re-ranking selectively. For a query like "What is JWT?" the top vector result is probably already correct. For "How do I handle JWT expiration when the user's clock is skewed?" the nuance warrants re-ranking.

def should_rerank(query: str) -> bool:
    """Simple heuristic: re-rank longer, more specific queries."""
    return len(query.split()) > 5

results = hybrid_search(query, top_k=5)
if should_rerank(query):
    results = retrieve_and_rerank(query)

What goes wrong

Mistake	How you notice it	The fix
Re-ranking top-50 when retrieval already missed	Reranked scores are all below 0.3, even the "best" document is irrelevant	Fix retrieval first. Check that your chunks actually contain the answer. Use evaluation metrics (next lesson) to measure retrieval quality before adding re-ranking
Re-ranking adds unacceptable latency	Users complain about slow responses; p95 latency jumps by 200-500ms	Re-rank fewer candidates (top 10 instead of top 50). Use async re-ranking so the first few results stream while the rest re-rank. Or skip re-ranking for simple queries
Cross-encoder favors longer documents	Consistently higher relevance scores for long chunks	Cross-encoders can exhibit length bias. Normalize by splitting long chunks or using a model known to be length-calibrated like mxbai-rerank
API key or quota exhausted	CohereError: invalid api token or 429 rate limit	Cache re-ranking results for repeated queries. Batch re-ranking calls to reduce API overhead. Set up usage alerts
Re-ranking the same query repeatedly	You see the same re-ranking API calls in logs	Cache re-ranked results by query hash. A simple functools.lru_cache on retrieve_and_rerank eliminates duplicate API calls

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Test with a query that requires fine-grained relevance discrimination
query = "How do I revoke a JWT before it expires?"

# Get re-ranked results
results = retrieve_and_rerank(query, top_k_retrieval=20, top_k_final=5)

# Verify the top result is actually about revocation, not issuance or validation
assert len(results) > 0, "No results returned"
top_content = results[0]["content"].lower()
assert any(term in top_content for term in ["revok", "revoc", "blacklist", "invalid"]), \
    f"Top result does not appear to be about revocation: {top_content[:200]}"

print("Top result relevance score:", results[0]["relevance_score"])
print("Top result snippet:", top_content[:200])
# Expected: relevance_score > 0.5, content discusses revocation or blacklisting

Next: Evaluating Retrieval