Metadata & Filtering

A vector search returns chunks sorted by similarity and nothing else. For a demo, that is enough. For a system people rely on, it is not. Users need to filter by source, by date, by document type, by section -- and they need those filters to work before the vector search runs, so the model never sees chunks from the wrong department or the wrong year. Metadata is what turns a vector store from a search engine into an information retrieval system.

What you'll learn

• Metadata attaches structured attributes (source, page, date, section, document type) to every chunk, enabling filtering that vector similarity alone cannot provide
• pgvector supports WHERE clauses on metadata columns, and when combined with an index on the filter column, PostgreSQL can filter before the vector distance computation runs
• Pre-filtering (metadata filter before vector search) is usually faster and safer than post-filtering, but it requires an index on the filter column and awareness of how PostgreSQL's query planner makes the decision

The problem

A user asks "What was our Q3 revenue?" Your vector search returns chunks about revenue -- from Q1, Q2, Q4, and three different departments. The model now has to sort through six chunks to find the one about Q3. Worse, it might stitch together numbers from different quarters and produce a believable but wrong answer.

The vector embedding captures semantic meaning. It does not capture structured facts like "this chunk is from page 42 of the 2024 annual report." Metadata handles those facts. And in pgvector, metadata is not a separate system -- it is just columns in the same table, filterable with the same WHERE clauses you already know.

Options & when to use each

Strategy	What it is good for	What it costs you	When to pick it
Pre-filtering (WHERE before vector search)	Fast, safe, deterministic; the vector search only runs on relevant rows	Requires an index on the filter column; if the planner gets it wrong, performance degrades	Most cases. Filter restricts the candidate set, vector search ranks within it
Post-filtering (vector search then WHERE)	Works when you have no filter index; simple to reason about	Wastes vector computation on rows you will discard; can miss results if top-k is fully consumed by filtered-out rows	Small datasets where performance does not matter, or when you cannot add an index
Hybrid: pre-filter + post-boost	Combines structured filters with a metadata relevance boost; good for "prefer Q3 but show Q2 if it is much better"	More complex query; harder to tune the boost weight	When user intent is fuzzy ("results from this year, but older results are OK if highly relevant")

Build it

Schema with metadata columns

Start with a chunks table that carries metadata alongside the vector embedding:

CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(1536),

    -- Metadata columns
    source TEXT NOT NULL,          -- e.g., 'annual_report_2024.pdf'
    page_number INTEGER,           -- nullable: not all sources have pages
    section_title TEXT,            -- e.g., 'Q3 Financial Results'
    document_date DATE,            -- when the document was published
    document_type TEXT,            -- 'report', 'email', 'contract', 'transcript'
    author TEXT,                   -- who wrote it
    token_count INTEGER,

    -- Chunking metadata
    chunking_strategy TEXT,
    level TEXT DEFAULT 'child',    -- 'child' or 'parent' for recursive chunking
    parent_chunk_id INTEGER REFERENCES chunks(id)
);

-- Indexes for fast filtering and vector search
CREATE INDEX idx_chunks_source ON chunks(source);
CREATE INDEX idx_chunks_document_date ON chunks(document_date);
CREATE INDEX idx_chunks_document_type ON chunks(document_type);
CREATE INDEX idx_chunks_section ON chunks(section_title);

Attaching metadata during ingestion

Here is a concrete example demonstrating the implementation details.

from datetime import date

def ingest_with_metadata(
    conn,
    document_id: str,
    text: str,
    chunks: list[str],
    metadata: dict,
):
    """Ingest chunks with structured metadata."""
    cur = conn.cursor()

    for i, chunk in enumerate(chunks):
        embedding = openai.embeddings.create(
            model="text-embedding-3-small", input=chunk
        ).data[0].embedding

        cur.execute("""
            INSERT INTO chunks (
                document_id, chunk_index, content, embedding,
                source, page_number, section_title,
                document_date, document_type, author,
                token_count, chunking_strategy
            ) VALUES (
                %s, %s, %s, %s,
                %s, %s, %s,
                %s, %s, %s,
                %s, %s
            )
        """, (
            document_id, i, chunk, embedding,
            metadata["source"],
            metadata.get("page_number"),
            metadata.get("section_title"),
            metadata.get("document_date"),
            metadata.get("document_type"),
            metadata.get("author"),
            len(enc.encode(chunk)),
            metadata.get("chunking_strategy", "fixed"),
        ))

    conn.commit()

# Example: ingesting a specific section of an annual report
report_text = "Q3 revenue reached $12.4M, up 18% year over year..."
chunks = split_by_tokens(report_text, chunk_size=256, overlap=25)

ingest_with_metadata(conn, "report_2024_q3", report_text, chunks, {
    "source": "annual_report_2024.pdf",
    "page_number": 42,
    "section_title": "Q3 Financial Results",
    "document_date": date(2024, 11, 15),
    "document_type": "financial_report",
    "author": "Finance Department",
    "chunking_strategy": "fixed",
})

Filtered vector search

Here is a concrete example demonstrating the implementation details.

def search_with_filters(
    conn,
    query: str,
    top_k: int = 5,
    document_type: str | None = None,
    after_date: date | None = None,
    source: str | None = None,
    section: str | None = None,
):
    """Vector search with pre-filtering on metadata columns."""
    query_embedding = openai.embeddings.create(
        model="text-embedding-3-small", input=query
    ).data[0].embedding

    # Build WHERE clauses dynamically
    conditions = []
    params: list = []

    if document_type:
        conditions.append("document_type = %s")
        params.append(document_type)

    if after_date:
        conditions.append("document_date >= %s")
        params.append(after_date)

    if source:
        conditions.append("source = %s")
        params.append(source)

    if section:
        conditions.append("section_title = %s")
        params.append(section)

    where_clause = " AND ".join(conditions) if conditions else "TRUE"

    # Add embedding to params (goes last for ORDER BY)
    params.extend([query_embedding, query_embedding, top_k])

    cur = conn.cursor()
    cur.execute(f"""
        SELECT content, source, page_number, section_title, document_date,
               1 - (embedding <=> %s::vector) AS similarity
        FROM chunks
        WHERE {where_clause}
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, params)

    return [
        {
            "content": row[0],
            "source": row[1],
            "page": row[2],
            "section": row[3],
            "date": row[4],
            "similarity": row[5],
        }
        for row in cur.fetchall()
    ]

# Search only financial reports from 2024
results = search_with_filters(
    conn, "Q3 revenue",
    document_type="financial_report",
    after_date=date(2024, 1, 1),
)

Knowing when PostgreSQL pre-filters vs post-filters

PostgreSQL's query planner decides whether to use the vector index or the B-tree index first. You can check with EXPLAIN:

EXPLAIN ANALYZE
SELECT content, 1 - (embedding <=> '[...]'::vector) AS similarity
FROM chunks
WHERE document_type = 'financial_report'
  AND document_date >= '2024-01-01'
ORDER BY embedding <=> '[...]'::vector
LIMIT 5;

If EXPLAIN shows a sequential scan or the vector index used before the B-tree filter, the planner made the wrong choice. Force pre-filtering with a CTE:

WITH filtered AS (
    SELECT * FROM chunks
    WHERE document_type = 'financial_report'
      AND document_date >= '2024-01-01'
)
SELECT content, 1 - (embedding <=> '[...]'::vector) AS similarity
FROM filtered
ORDER BY embedding <=> '[...]'::vector
LIMIT 5;

The CTE guarantees the filter runs first. This is a safe default when you have a restrictive metadata filter and a large table.

Python wrapper for CTE-based pre-filtering

Here is a concrete example demonstrating the implementation details.

def search_prefiltered(
    conn, query: str, top_k: int = 5, **filters
) -> list[dict]:
    """Vector search with guaranteed pre-filtering via CTE."""
    query_embedding = openai.embeddings.create(
        model="text-embedding-3-small", input=query
    ).data[0].embedding

    conditions = []
    params: list = []

    for col, val in filters.items():
        if val is not None:
            conditions.append(f"{col} = %s")
            params.append(val)

    where_clause = " AND ".join(conditions) if conditions else "TRUE"
    params.extend([query_embedding, query_embedding, top_k])

    cur = conn.cursor()
    cur.execute(f"""
        WITH filtered AS (
            SELECT * FROM chunks WHERE {where_clause}
        )
        SELECT content, source, page_number, section_title,
               1 - (embedding <=> %s::vector) AS similarity
        FROM filtered
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, params)

    return [
        {
            "content": row[0],
            "source": row[1],
            "page": row[2],
            "section": row[3],
            "similarity": row[4],
        }
        for row in cur.fetchall()
    ]

What goes wrong

Mistake	How you notice it	The fix
No index on the filter column	EXPLAIN shows a sequential scan; filtered searches are slow on large tables	Add a B-tree index on every column used in WHERE clauses. CREATE INDEX idx_chunks_document_type ON chunks(document_type)
Filter is too restrictive	Zero results even though relevant chunks exist	Check that your metadata is correct during ingestion. A typo in document_type means those chunks are invisible to the filter
Post-filtering drops relevant results	A highly similar chunk gets filtered out because top_k was consumed by chunks that matched the query but not the filter	Use pre-filtering (WHERE before vector search) so the vector search only runs on rows that pass the filter
NULL metadata breaks filters	WHERE section_title = 'Q3 Results' returns zero rows, but the chunks exist with section_title IS NULL	Use COALESCE or set default values during ingestion. Never leave filterable metadata as NULL if users will query on it
Metadata and content diverge	A chunk tagged as "Q3" contains Q2 data because the document had a mislabeled heading	Metadata is only as good as the ingestion pipeline that sets it. Validate metadata against content with spot checks or regex verification

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Verify that filtered search returns only matching chunks
# and that unfiltered search returns both types

results_filtered = search_with_filters(
    conn, "revenue growth",
    document_type="financial_report",
    after_date=date(2024, 1, 1),
)

for r in results_filtered:
    assert r["date"] >= date(2024, 1, 1), \
        f"Got result from {r['date']} despite after_date filter"
    # Type check: should all be financial reports

print(f"Filtered results: {len(results_filtered)}")
assert len(results_filtered) > 0, "Should return matching chunks"

# Unfiltered search should return more results (or the same if all chunks match)
results_unfiltered = search_with_filters(conn, "revenue growth", top_k=10)
print(f"Unfiltered results: {len(results_unfiltered)}")
assert len(results_unfiltered) >= len(results_filtered), \
    "Unfiltered search should return at least as many results"

# Verify CTE pre-filtering produces the same results as standard filtering
results_cte = search_prefiltered(
    conn, "revenue growth",
    document_type="financial_report",
)
cte_contents = {r["content"] for r in results_cte}
filtered_contents = {r["content"] for r in results_filtered}
assert cte_contents == filtered_contents, \
    "CTE and standard filtering should return the same chunks"

Previous: Recursive ChunkingNext: Day 3 -- Retrieval Quality & Ranking