Evaluating Retrieval

You built a retrieval pipeline. Hybrid search. Maybe re-ranking. You ran a few queries and the results looked right. But "looked right" doesn't scale past a dozen queries, and it can't tell you whether last week's chunking change made things better or worse. You need numbers.

Retrieval evaluation measures how often your pipeline finds the right documents, how high they rank, and whether the ordering makes sense. This lesson covers the three metrics that matter, how to build a test set, and how to automate evaluation so you can iterate on retrieval with confidence.

What you'll learn

• recall@k tells you whether the relevant document is in the top k results at all; it's the most forgiving metric and the right one when you re-rank afterward
• MRR tells you how high the first relevant document ranks; it's the metric for question-answering systems where the user reads one answer
• NDCG rewards relevant documents ranked higher and graded relevance; use it when documents have different degrees of usefulness, not just binary relevant/not

The problem: you don't know if retrieval works

A developer adds semantic chunking. The pipeline feels better. Another developer tunes the hybrid search weights. Did recall go up or down? Without metrics, every change is a guess. With metrics, you run the evaluation suite and get a number: recall@5 went from 0.72 to 0.81. That's progress. Without it, you're navigating blind.

Retrieval evaluation is also the only way to catch regressions. You add a filter to exclude short chunks and suddenly recall drops because a one-sentence answer got filtered out. The evaluation catches that before it reaches users.

Options & when to use each

Metric	What it measures	When to use
recall@k	Fraction of queries where the relevant document appears in the top k results	Your pipeline re-ranks afterward; you care about getting the right document into the candidate set, not exact position
MRR (Mean Reciprocal Rank)	Average of 1/rank for the first relevant document across queries	Question answering, chatbots, any system where the user reads the first answer
NDCG (Normalized Discounted Cumulative Gain)	Position-weighted relevance with graded scores (0-3)	Documents have degrees of relevance (somewhat useful vs exactly right); you care about the full ordering
Precision@k	Fraction of top k results that are relevant	You want to minimize noise in the results shown to the user

For most RAG systems, start with recall@5 and MRR. If your documents have graded relevance (e.g., "exact answer," "related but not answering," "off-topic"), add NDCG.

Build it: a retrieval evaluation pipeline

Step 1: build a test set

A test set is a collection of (query, relevant_document_ids, [optional: relevance_grades]) tuples. You need at least 30-50 queries for a meaningful evaluation. Fewer than 20 and the variance is too high to trust.

test_queries = [
    {
        "query": "How do I set JWT expiration?",
        "relevant_ids": [42, 87],           # doc 42 is the exact answer, doc 87 is also correct
        "grades": [3, 2],                    # 3=perfect, 2=good, 1=related, 0=irrelevant
    },
    {
        "query": "What is the maximum JWT token size?",
        "relevant_ids": [15],
        "grades": [3],
    },
    {
        "query": "How to handle JWT in microservices?",
        "relevant_ids": [103, 56, 201],
        "grades": [3, 2, 1],
    },
    # ... at least 30 queries total
]

Building this test set is the hardest part of retrieval evaluation. Strategies:

• Start with known answers: If you have an FAQ or documentation with known question-answer pairs, use those. The document containing the answer is your relevant document.
• Use an LLM to judge: For each query, have an LLM read the top 20 retrieved documents and mark which ones are relevant. This gets you a test set quickly but requires verification.
• Hand-label the hard cases: Spend your labeling time on the queries where the LLM is uncertain, not the obvious ones.
• Log production queries: Record real user queries and have a human label the top results. This gives you a test set that matches actual usage.

Step 2: compute the metrics

Here is a concrete example demonstrating the implementation details.

from typing import Sequence

def recall_at_k(
    retrieved_ids: Sequence[int],
    relevant_ids: set[int],
    k: int,
) -> float:
    """Fraction of relevant documents found in top k results."""
    if not relevant_ids:
        return 1.0  # vacuously true
    top_k = set(retrieved_ids[:k])
    return len(top_k & relevant_ids) / len(relevant_ids)


def mrr(retrieved_ids: Sequence[int], relevant_ids: set[int]) -> float:
    """Mean reciprocal rank of the first relevant document."""
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / rank
    return 0.0


def ndcg_at_k(
    retrieved_ids: Sequence[int],
    relevant_ids: set[int],
    grades: dict[int, int],
    k: int,
) -> float:
    """
    Normalized discounted cumulative gain at k.
    grades maps doc_id -> relevance grade (0-3).
    """
    def dcg(ids):
        gain = 0.0
        for i, doc_id in enumerate(ids[:k]):
            rel = grades.get(doc_id, 0)
            gain += rel / (__import__("math").log2(i + 2))  # i+2 because i is 0-indexed
        return gain

    # Ideal ordering: all relevant docs sorted by grade descending, padded to k
    ideal_ids = sorted(grades.keys(), key=lambda did: grades[did], reverse=True)
    ideal = dcg(ideal_ids)

    if ideal == 0:
        return 0.0
    return dcg(retrieved_ids) / ideal

Step 3: run the evaluation

Here is a concrete example demonstrating the implementation details.

def evaluate_retrieval(
    search_fn: callable,
    test_queries: list[dict],
    k_values: list[int] = [1, 3, 5, 10],
) -> dict:
    """
    Run full evaluation suite against a retrieval function.
    search_fn(query, top_k) should return list of dicts with an 'id' key.
    """
    max_k = max(k_values)
    recall_sums = {k: 0.0 for k in k_values}
    mrr_sum = 0.0
    ndcg_sums = {k: 0.0 for k in k_values}
    n = len(test_queries)

    for item in test_queries:
        query = item["query"]
        relevant = set(item["relevant_ids"])
        grades = dict(zip(item["relevant_ids"], item.get("grades", [1] * len(relevant))))

        results = search_fn(query, top_k=max_k)
        retrieved_ids = [doc["id"] for doc in results]

        for k in k_values:
            recall_sums[k] += recall_at_k(retrieved_ids, relevant, k)
            ndcg_sums[k] += ndcg_at_k(retrieved_ids, relevant, grades, k)

        mrr_sum += mrr(retrieved_ids, relevant)

    return {
        **{f"recall@{k}": recall_sums[k] / n for k in k_values},
        "mrr": mrr_sum / n,
        **{f"ndcg@{k}": ndcg_sums[k] / n for k in k_values},
    }

Here is a concrete example demonstrating the implementation details.

# Run it
scores = evaluate_retrieval(hybrid_search, test_queries)

for metric, value in scores.items():
    print(f"{metric}: {value:.3f}")
# Example output:
# recall@1: 0.433
# recall@3: 0.689
# recall@5: 0.811
# recall@10: 0.892
# mrr: 0.612
# ndcg@5: 0.734

Step 4: automate the evaluation

Here is a concrete example demonstrating the implementation details.

# Save this as evaluate.py and run it on every retrieval change
import json
from datetime import datetime

def run_and_log(search_fn, test_queries, label: str = ""):
    scores = evaluate_retrieval(search_fn, test_queries)
    scores["label"] = label
    scores["timestamp"] = datetime.now().isoformat()
    scores["num_queries"] = len(test_queries)

    # Append to a log file
    with open("retrieval_metrics.jsonl", "a") as f:
        f.write(json.dumps(scores) + "\n")

    return scores


# After the baseline
run_and_log(hybrid_search, test_queries, label="baseline-hybrid-k60")

# After tuning weights
run_and_log(
    lambda q, k: hybrid_search(q, top_k=k, vector_weight=0.7, text_weight=0.3),
    test_queries,
    label="tuned-vector-0.7",
)

Now git diff on retrieval_metrics.jsonl shows exactly how each change affected every metric.

What goes wrong

Mistake	How you notice it	The fix
Test queries overlap with training data	Evaluation scores are suspiciously high (recall@5 > 0.95)	Hold out documents from the test set. If you used an LLM to generate queries, verify it didn't copy chunks verbatim
Too few test queries	Metrics swing wildly between runs (recall@5 jumps from 0.6 to 0.8 with a one-line change)	Minimum 30 queries. 50+ for production. Compute confidence intervals: 1.96 * sqrt(p*(1-p)/n)
Binary relevance when graded is needed	All relevant docs treated equally, even though one is the exact answer and another is tangentially related	Add grades to your test set and use NDCG. A document that mentions the topic but doesn't answer gets a 1, not a 3
ID mismatches between test set and retrieval	All metrics at 0.0 because none of the relevant_ids appear in results	Verify that the IDs in your test set match the IDs returned by your retrieval function. Use a join query to check: SELECT id FROM documents WHERE id = ANY(%s)
Evaluating only the happy path	Metrics look great but production users report failures	Add adversarial queries: misspellings, very short queries, queries about topics not in your corpus. These expose robustness gaps

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Sanity check: the evaluation should return sensible numbers
# Create a trivial search function that always returns the same docs
def dummy_search(query, top_k):
    return [{"id": i} for i in range(1, top_k + 1)]

test = [{"query": "test", "relevant_ids": [1, 3, 5], "grades": [3, 2, 1]}]
scores = evaluate_retrieval(dummy_search, test, k_values=[1, 3, 5])

assert scores["recall@1"] == 1/3, "recall@1 should be 1/3"
assert scores["recall@3"] == 2/3, "recall@3 should be 2/3"
assert scores["recall@5"] == 1.0, "recall@5 should be 1.0"
assert scores["mrr"] == 1.0, "MRR should be 1.0 (first doc is relevant)"

print("Sanity checks passed")
print(f"MRR: {scores['mrr']:.3f}")
print(f"NDCG@5: {scores['ndcg@5']:.3f}")

Next: Query Rewriting