Self-Querying Retrieval

A user asks "show me GPU benchmarks from the last month." Basic RAG embeds the whole query into a vector and hopes the right chunks come back. It won't. The vector captures "GPU" and "benchmarks" just fine, but "last month" is a date constraint, not a semantic one. The vector store has no way to filter by date unless you tell it to. Self-querying retrieval solves this by splitting the LLM's job: extract the structured constraints first, then use vector search for the semantic part. The result is a search that actually respects all the user's constraints, not just the ones that happen to be embeddable.

What you'll learn

• Self-querying uses an LLM to extract metadata filters from natural language before the vector search runs
• The LLM outputs a structured query object: {query: "GPU benchmarks", filter: {source: "benchmarks", date: ">2026-06-01"}}
• This is vastly more reliable than hoping vector similarity alone matches the right metadata — date ranges, categories, and authors are not semantic concepts

The problem

A user types "show me GPU benchmarks from the last month." Three constraints are packed into nine words: the topic (GPU benchmarks), the document type (benchmarks, not blog posts or spec sheets), and a date range (last month). When you embed this entire sentence into a 1536-dimensional vector, the semantic similarity search finds chunks about GPUs. It might even find chunks about benchmarks. But it will also return GPU-related content from three years ago with equal enthusiasm, because the vector has no concept of "last month." The date constraint is structural, not semantic, and vector search alone cannot enforce it.

Self-querying retrieval handles this by giving the LLM two outputs instead of one. The first is a rewritten semantic query stripped of structural constraints: "GPU benchmarks performance comparison." The second is a structured filter object: [{field: "date", op: ">", value: "2026-06-01"}, {field: "category", op: "=", value: "benchmarks"}]. The vector store runs both: the embedding search for relevance and the WHERE clause for constraints. Only chunks that pass both gates reach the user.

This pattern generalizes to any metadata you index alongside your documents: source, author, section, status, product, customer, language, date range, or any custom tag. The key insight is that the LLM is better at understanding which constraints exist in a natural language query than any regex or rule-based parser ever will be, but it still needs you to define the schema it can query against.

Options & when to use each

Approach	Good for	Costs you	When to pick it
Pure vector search	Simple factoid questions	No structured filtering at all	When your users ask things like "what is X" with no date/product/source constraints
Pre-filter then vector search	When you know the filter before the query	Filter must be provided by the application, not the user	When your UI has dropdown filters and the user picks from them
Vector search then post-filter	When you want to filter after retrieval	Wastes retrieval budget on items that will be filtered out	When the filter set is too complex for SQL (e.g. "documents similar to this one but more recent")
Self-querying (LLM extracts filters)	Natural language queries with implicit constraints	One extra LLM call per search, ~200ms latency	When users ask natural language questions with date/source/category constraints

Self-querying is worth the extra LLM call when your users ask questions that mix semantic intent with structural constraints. If your users only ask "what is X," stick with pure vector search. If your UI provides explicit filter dropdowns, use pre-filtering. But if you have a free-text search box where users type things like "show me the Q3 sales reports from the European team," self-querying is the only approach that works without making the user fill out a form first.

Build it

Step 1: Define your metadata schema

Before the LLM can extract filters, it needs to know what fields exist and what values are valid. Define this once and include it in every extraction prompt:

METADATA_SCHEMA = {
    "fields": {
        "source": {"type": "string", "description": "Document source: 'docs', 'blog', 'benchmarks', 'spec'"},
        "date": {"type": "date", "description": "Publication date in YYYY-MM-DD format"},
        "category": {"type": "enum", "values": ["benchmarks", "tutorial", "reference", "changelog"]},
        "author": {"type": "string", "description": "Author name"},
        "product": {"type": "string", "description": "Product name the document relates to"},
        "status": {"type": "enum", "values": ["draft", "published", "archived"]},
    }
}

Step 2: Extract structured filters with the LLM

Using the schema, prompt the LLM to parse the user's natural language into a clean semantic query and a list of structural filters. This separation is the core mechanism of self-querying.

import json
from openai import OpenAI

client = OpenAI()

def extract_filters(query, schema):
    schema_desc = "\n".join(
        f"- {name}: {info['description']}" +
        (f" (one of: {', '.join(info['values'])})" if info.get('values') else "")
        for name, info in schema["fields"].items()
    )

    prompt = f"""Given a user's natural language query, extract any metadata filters
that would help narrow the search. Also rewrite the query to remove filter language
and keep only the semantic intent.

Available metadata fields:
{schema_desc}

Rules:
- Only extract filters the user explicitly or clearly implies
- For dates, convert relative terms to YYYY-MM-DD (today is {today})
- If the user doesn't specify a filter, don't guess one
- The semantic_query should be the original question with filter language removed

User query: "{query}"

Return JSON:
{{
    "semantic_query": "rewritten query for vector search only",
    "filters": [
        {{"field": "field_name", "op": "=|>|<|>=|<=", "value": "value"}}
    ]
}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(response.choices[0].message.content)

A query like "show me GPU benchmarks from the last month" produces:

{
    "semantic_query": "GPU benchmarks performance comparison",
    "filters": [
        {"field": "date", "op": ">=", "value": "2026-06-21"},
        {"field": "category", "op": "=", "value": "benchmarks"}
    ]
}

Step 3: Build the SQL WHERE clause safely

Translate the extracted JSON filters into parameterized SQL conditions. This validation step is critical to prevent injection attacks and ignore hallucinated fields.

ALLOWED_FIELDS = set(METADATA_SCHEMA["fields"].keys())
ALLOWED_OPS = {"=", ">", "<", ">=", "<=", "IN"}

def build_where_clause(filters):
    conditions = []
    for f in filters:
        if f["field"] not in ALLOWED_FIELDS:
            continue  # Skip hallucinated fields silently
        if f["op"] not in ALLOWED_OPS:
            continue

        # Use parameterized queries to prevent injection
        if f["op"] == "IN":
            values = f["value"] if isinstance(f["value"], list) else [f["value"]]
            placeholders = ", ".join(["%s"] * len(values))
            conditions.append((f"metadata->>'{f['field']}' IN ({placeholders})", values))
        else:
            conditions.append((f"metadata->>'{f['field']}' {f['op']} %s", [f["value"]]))

    return conditions  # Return parameterized for psycopg2

Step 4: Execute the combined query

The final step combines the generated SQL filter with the vector search execution. It also includes a necessary fallback mechanism in case the structural constraints were overly aggressive.

def self_query_search(query_text, top_k=10):
    # Extract filters
    extracted = extract_filters(query_text, METADATA_SCHEMA)

    # Generate embedding from the semantic query (stripped of filter language)
    embedding = generate_embedding(extracted["semantic_query"])

    # Build WHERE clause
    where_conditions = build_where_clause(extracted["filters"])
    where_sql = " AND ".join(cond[0] for cond in where_conditions) if where_conditions else "TRUE"
    where_params = [p for cond in where_conditions for p in cond[1]]

    # Combined query: vector similarity with metadata filtering
    results = cursor.execute(f"""
        SELECT content, metadata, 1 - (embedding <=> %s) AS similarity
        FROM documents
        WHERE status = 'active' AND ({where_sql})
        ORDER BY embedding <=> %s
        LIMIT %s
    """, [embedding] + where_params + [embedding, top_k]).fetchall()

    # Fallback: if filtered results are empty, try without filters
    if not results and where_conditions:
        results = cursor.execute("""
            SELECT content, metadata, 1 - (embedding <=> %s) AS similarity
            FROM documents WHERE status = 'active'
            ORDER BY embedding <=> %s LIMIT %s
        """, (embedding, embedding, top_k)).fetchall()

    return results

The fallback to pure vector search is important. If the LLM extracts an overly restrictive filter — perhaps it interpreted "last month" too literally and the most recent benchmark is 35 days old — the user gets zero results. The fallback ensures they get something relevant, even if it doesn't perfectly match the structural constraints. Log when the fallback triggers so you can tune the extraction prompt.

What goes wrong

Mistake	How you notice it	The fix
LLM hallucinates metadata fields that don't exist	SQL error on unknown column	Validate extracted fields against ALLOWED_FIELDS. Silently skip unknown fields rather than erroring
Overly restrictive filters produce zero results	Empty result set on a query that should match	Always include a fallback to pure vector search when filtered results are empty. Log the fallback so you can debug the extraction
LLM misinterprets relative dates	"Last month" becomes a date off by a week or a year	Include today in the extraction prompt so the LLM has a reference point. Validate extracted dates are reasonable
Extraction adds cost and latency to every query	P95 latency increases by 300ms, embedding costs double	Consider caching extraction results for common query patterns. If 80% of queries don't have structural constraints, skip extraction for those
Extraction prompt drifts from schema	LLM extracts fields that existed last month but were renamed	Keep the schema definition in a single source of truth. Generate the extraction prompt from the schema programmatically, not by hand

Confirm it worked

Verify the implementation by indexing documents with conflicting dates and running a time-constrained query. The test confirms both the filtering logic and the fallback mechanism function properly.

# Test: query with date constraint should filter old results
old_doc = {"id": "old", "content": "GPU benchmarks from 2020", "date": "2020-01-01", "category": "benchmarks"}
new_doc = {"id": "new", "content": "GPU benchmarks from this month", "date": "2026-07-01", "category": "benchmarks"}
index_documents([old_doc, new_doc])

results = self_query_search("show me recent GPU benchmarks")
# Should return the new document, not the old one
assert len(results) > 0
assert results[0]["metadata"]["date"] >= "2026-06-01"

# Test: fallback when filter is too restrictive
results = self_query_search("show me GPU benchmarks from exactly July 4 2025 at 3pm")
# Should still return something via fallback
assert len(results) > 0

Next: Multi-Hop Retrieval