Similarity Search

You've built the pipeline: generate embeddings, store them in pgvector, query by semantic similarity. Now you need to understand what happens when the search returns bad results, why it happens, and how to fix it. Similarity search is not magic. It's k-NN with distance thresholds, and it has predictable failure modes that you can diagnose and correct. This lesson covers the mechanics of nearest-neighbor search, when to filter by distance, how metadata changes the result set, and what those distance scores actually mean.

What you'll learn

• k-NN search finds the k closest vectors to a query vector; with an HNSW index it's approximate (a-kNN) -- slightly different results each time, but orders of magnitude faster
• Distance thresholds filter out low-quality matches: if the best result has a cosine distance of 0.8, it's probably irrelevant regardless of its rank
• Metadata filtering narrows the vector search to a subset of documents, but it's a pre-filter: documents outside the metadata filter are excluded before the vector comparison runs
• Distance scores are relative to a specific query, not absolute quality measures -- a distance of 0.3 might be excellent for one query and terrible for another

The problem: when the top result is wrong

Embedding search always returns something. Sort by cosine distance and the first row is the closest match in the database. That does not mean it's a good match. If none of your documents are relevant to the query, the "best" result is still wrong. If your corpus is small or your domain is niche, the closest embedding might be a document about an unrelated topic that happened to use a few similar words during training.

You need two things to make similarity search reliable: distance thresholds that discard irrelevant results, and an understanding of what the distance numbers mean so you can set those thresholds correctly.

How k-NN (and approximate k-NN) works

At its core, nearest-neighbor search is simple: given a query vector Q, find the k vectors in the database with the smallest distance to Q. An exact k-NN search computes the distance from Q to every vector in the table and returns the closest k. That's O(n) per query.

HNSW (Hierarchical Navigable Small World) makes this approximate. Instead of computing every distance, it traverses a graph where each node (a vector) is connected to its nearest neighbors. The search jumps between layers of the graph, following ever-closer neighbors until it reaches a local minimum. The result is approximate -- it might miss the true nearest neighbor -- but the speedup is dramatic: sub-millisecond searches on millions of vectors instead of seconds.

-- Exact search (full table scan): accurate but slow
SELECT content, embedding <=> query_vec AS distance
FROM documents
ORDER BY embedding <=> query_vec
LIMIT 10;

-- Approximate search (HNSW index): fast but slightly different results
-- The HNSW index is used automatically when available and LIMIT is small
-- compared to the table size

The tradeoff is recall vs speed. With default HNSW parameters, you get ~99% recall (the true nearest neighbor is in the top 10) at a tiny fraction of the cost of exact search. You can tune ef_search at query time to increase recall at the cost of speed:

-- Increase ef_search for higher recall (default is 40)
SET hnsw.ef_search = 100;

-- Your query here
SELECT content, embedding <=> query_vec AS distance
FROM documents
ORDER BY embedding <=> query_vec
LIMIT 10;

Distance thresholds: rejecting irrelevant results

Without a threshold, every query returns k results even if the closest match is 0.9 cosine distance away (nearly orthogonal, i.e., unrelated). Adding a WHERE clause on distance means the query returns fewer than k results when nothing is relevant:

def search_with_threshold(
    cur,
    query: str,
    limit: int = 10,
    max_distance: float = 0.5,
    metadata_filter: dict | None = None
) -> list[dict]:
    """
    Semantic search that discards results above a distance threshold.
    Returns empty list if nothing is close enough.
    """
    query_embedding = embed([query])[0]

    conditions = ["embedding <=> %(query_vec)s < %(max_dist)s"]
    params = {
        "query_vec": query_embedding,
        "max_dist": max_distance,
        "limit": limit
    }

    if metadata_filter:
        for k in metadata_filter:
            conditions.append(f"metadata->>'{k}' = %({k})s")
            params[k] = metadata_filter[k]

    where_clause = " AND ".join(conditions)

    sql = f"""
        SELECT content, metadata, embedding <=> %(query_vec)s AS distance
        FROM documents
        WHERE {where_clause}
        ORDER BY embedding <=> %(query_vec)s
        LIMIT %(limit)s
    """

    cur.execute(sql, params)
    return [
        {"content": row[0], "metadata": row[1], "distance": float(row[2])}
        for row in cur.fetchall()
    ]

How to pick a threshold

There is no universal "good" threshold. It depends on your embedding model and your corpus. The empirical approach:

def estimate_threshold(cur, queries: list[str], labels: list[bool]) -> dict:
    """
    Given queries with known relevance labels (1=relevant, 0=irrelevant),
    compute precision/recall at different distance thresholds.
    """
    thresholds = [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
    results = {}

    for t in thresholds:
        tp = fp = fn = 0
        for query, is_relevant in zip(queries, labels):
            hits = search_with_threshold(cur, query, limit=10, max_distance=t)
            found = any(h for h in hits)  # At least one result returned
            if is_relevant and found:
                tp += 1
            elif is_relevant and not found:
                fn += 1
            elif not is_relevant and found:
                fp += 1
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        results[t] = {"precision": precision, "recall": recall}

    return results

# Example: you know 3 queries are relevant and 2 are not
test_queries = [
    "database vector search",    # relevant
    "python web framework",       # relevant
    "machine learning basics",    # relevant
    "recipe for lasagna",         # irrelevant
    "weather in Tokyo",           # irrelevant
]
test_labels = [True, True, True, False, False]

thresholds = estimate_threshold(cur, test_queries, test_labels)
for t, metrics in thresholds.items():
    print(f"Threshold {t}: P={metrics['precision']:.2f} R={metrics['recall']:.2f}")

For OpenAI text-embedding-3-small, a threshold of 0.4-0.5 typically works well for English RAG. Start at 0.5 and tune based on your data.

Metadata filtering: narrowing the search space

Metadata filtering happens before the vector comparison. PostgreSQL applies the WHERE clause first, then runs the vector search on the filtered subset. This means:

-- This: filter first, then search
SELECT * FROM documents
WHERE metadata->>'topic' = 'databases'
ORDER BY embedding <=> query_vec
LIMIT 10;

-- Is NOT the same as: search first, then filter
-- The first approach is usually faster because it reduces
-- the number of vectors the HNSW index must compare.

But metadata filtering has a subtle failure mode: if the filter is too restrictive, it might eliminate the best matches. Consider a document about "PostgreSQL" tagged as topic: databases. A query for "open source relational storage" might semantically match it, but if you filter for topic: storage, you'll miss it because the tag is databases, not storage.

Strategy: start without metadata filters. Add them only when you observe specific categories consistently returning irrelevant results.

def search_ranked_hybrid(
    cur,
    query: str,
    limit: int = 10,
    max_distance: float = 0.5,
    boost_metadata: dict | None = None
) -> list[dict]:
    """
    Search with optional metadata boosting instead of hard filtering.
    Documents matching metadata come first; unmatched documents still
    appear if they're semantically close.
    """
    query_embedding = embed([query])[0]

    if boost_metadata:
        boost_clause = " AND ".join(
            f"metadata->>'{k}' = %({k}_val)s" for k in boost_metadata
        )
        params = {
            "query_vec": query_embedding,
            "max_dist": max_distance,
            "limit": limit
        }
        for k, v in boost_metadata.items():
            params[f"{k}_val"] = v

        sql = f"""
            SELECT content, metadata, embedding <=> %(query_vec)s AS distance,
                   CASE WHEN {boost_clause} THEN 0 ELSE 1 END AS metadata_rank
            FROM documents
            WHERE embedding <=> %(query_vec)s < %(max_dist)s
            ORDER BY metadata_rank, embedding <=> %(query_vec)s
            LIMIT %(limit)s
        """
    else:
        sql = """
            SELECT content, metadata, embedding <=> %(query_vec)s AS distance
            FROM documents
            WHERE embedding <=> %(query_vec)s < %(max_dist)s
            ORDER BY embedding <=> %(query_vec)s
            LIMIT %(limit)s
        """
        params = {"query_vec": query_embedding, "max_dist": max_distance, "limit": limit}

    cur.execute(sql, params)
    return [
        {"content": row[0], "metadata": row[1], "distance": float(row[2])}
        for row in cur.fetchall()
    ]

Understanding distance scores

Cosine distance values are not universal quality scores. A distance of 0.25 is "close" for any embedding model, but whether it means "good enough to show the user" depends on your application.

Distance range	Typical meaning	Action
0.0 - 0.2	Nearly identical text or very close paraphrase	Confidently use as context
0.2 - 0.4	Semantically related	Usually good enough for RAG; include with medium confidence
0.4 - 0.6	Loosely related, shared topic or domain	Borderline; consider showing to users with lower prominence
0.6 - 0.8	Same broad domain, different specific topic	Likely irrelevant; discard for RAG
0.8 - 2.0	Unrelated	Definitely discard

These ranges are approximate and vary by model and corpus. A specialized medical corpus might have tighter clusters (everything is about medicine) where 0.5 means "entirely different disease." A general web corpus might have looser clusters where 0.5 means "same general topic area."

Build it: a unified search function

The individual pieces explored above -- threshold filtering, metadata filtering, metadata boosting -- compose into a single search() function you can drop into any RAG pipeline:

from typing import Optional

def search(
    cur,
    query: str,
    limit: int = 10,
    max_distance: float = 0.5,
    metadata_filter: Optional[dict] = None,
    metadata_boost: Optional[dict] = None
) -> list[dict]:
    """
    Unified semantic search with optional threshold, metadata filter,
    and metadata boosting. Returns results sorted by relevance.
    """
    query_embedding = embed([query])[0]
    conditions = ["embedding <=> %(query_vec)s < %(max_dist)s"]
    params = {
        "query_vec": query_embedding,
        "max_dist": max_distance,
        "limit": limit
    }

    # Hard metadata filter
    if metadata_filter:
        for k, v in metadata_filter.items():
            conditions.append(f"metadata->>'{k}' = %({k}_val)s")
            params[f"{k}_val"] = v

    where_clause = " AND ".join(conditions)

    # Soft metadata boost as ordering tier
    if metadata_boost:
        boost_clause = " AND ".join(
            f"metadata->>'{k}' = %({k}_boost)s" for k in metadata_boost
        )
        for k, v in metadata_boost.items():
            params[f"{k}_boost"] = v
        order_clause = f"""
            CASE WHEN {boost_clause} THEN 0 ELSE 1 END,
            embedding <=> %(query_vec)s
        """
    else:
        order_clause = "embedding <=> %(query_vec)s"

    sql = f"""
        SELECT content, metadata, embedding <=> %(query_vec)s AS distance
        FROM documents
        WHERE {where_clause}
        ORDER BY {order_clause}
        LIMIT %(limit)s
    """

    cur.execute(sql, params)
    return [
        {"content": row[0], "metadata": row[1], "distance": float(row[2])}
        for row in cur.fetchall()
    ]

This function is the interface every other part of your RAG system calls. It embeds the query, runs cosine similarity with the HNSW index, applies thresholds and metadata controls, and returns ranked results. The embed() function comes from lesson 2; the database connection and register_vector() come from lesson 3.

Options & when to use each

Option	What it's good for	What it costs you	When to pick it
k-NN without threshold	Maximum recall; you always get k results	Returns irrelevant results when nothing matches; wastes LLM context on noise	Prototyping or when you're certain your corpus covers every possible query
k-NN with threshold	Prevents hallucination fuel from entering RAG context	Requires tuning the threshold for your model and corpus	Production RAG; the default choice
Metadata pre-filtering	Tightens results to a known category	Can exclude relevant documents with misaligned tags	When you have high-quality, consistently applied metadata
Metadata boosting	Biases results without hard exclusion	Slightly more complex query; need to tune the boost weight	When metadata is useful but not reliable enough for hard filtering

What goes wrong

Mistake	How you notice it	The fix
No distance threshold in production	Irrelevant documents appear in RAG context; LLM generates hallucinated answers or "I don't know"	Add a WHERE clause with embedding <=> query_vec < threshold. Start at 0.5, tune with real queries
Threshold too strict	Too few results; some queries return empty when there are relevant documents in the corpus	Raise the threshold incrementally. Plot precision-recall at different thresholds (code above)
Threshold too loose	Every query returns k results regardless of relevance; RAG context is noisy	Lower the threshold. Better to return 3 good results than 10 where 7 are noise
Comparing distance scores across different queries	"Why is the best match for 'Python' 0.12 but the best match for 'database' is 0.35?"	Don't compare distances across queries. A distance of 0.12 and 0.35 are both "good" -- the absolute value depends on how common the query's semantic neighborhood is in your corpus
Expecting the HNSW index to guarantee exact results	The same query returns slightly different top-10 results on different runs	Set hnsw.ef_search higher for more consistent results, or accept that approximate search trades a tiny recall hit for massive speed gains

Confirm it worked

Here is a concrete example demonstrating the implementation details.

with conn.cursor() as cur:
    # 1. Relevant query: should return results below threshold
    relevant = search_with_threshold(cur, "vector database for PostgreSQL", max_distance=0.5)
    print(f"Relevant query returned {len(relevant)} results")
    assert len(relevant) > 0, "Expected results for a relevant query"

    # 2. Irrelevant query: should return nothing
    irrelevant = search_with_threshold(cur, "recipe for chocolate cake", max_distance=0.5)
    print(f"Irrelevant query returned {len(irrelevant)} results")
    assert len(irrelevant) == 0, (
        "Expected no results for an irrelevant query. "
        "If this fails, your threshold is too loose for this corpus."
    )

    # 3. Metadata filtering: should only return matching topic
    filtered = search_with_threshold(
        cur, "programming tools", max_distance=0.5,
        metadata_filter={"topic": "programming"}
    )
    for r in filtered:
        assert r["metadata"]["topic"] == "programming", \
            f"Metadata filter failed: got topic={r['metadata']['topic']}"
    print(f"Metadata filter passed: {len(filtered)} results, all topic='programming'")

    # 4. Metadata boosting: programming should come first, but databases can appear
    boosted = search_ranked_hybrid(
        cur, "data storage and retrieval", limit=5,
        boost_metadata={"topic": "databases"}
    )
    topics = [r["metadata"].get("topic", "") for r in boosted]
    print(f"Boosted results topics: {topics}")
    assert topics[0] == "databases", \
        f"Boosting should put databases first, got: {topics}"

print("Similarity search is working with thresholds, filters, and boosts.")

Next: Day 2 -- Chunking & Ingestion