Query Rewriting

Users write queries the way they think, not the way your documents are written. "How do I log out?" becomes "token invalidation session termination JWT refresh token revocation" in your docs. "What's the price?" means nothing without context. "Tell me about security" is too broad to retrieve anything useful.

Query rewriting uses an LLM to transform the user's raw query into something your retrieval pipeline can work with. This lesson covers three techniques: single-query expansion, multi-query decomposition, and HyDE (hypothetical document embeddings). Each one addresses a different failure mode in retrieval.

What you'll learn

• Query rewriting bridges the vocabulary gap between how users ask questions and how your documents are written; it adds the domain-specific terms that embeddings need to find the right content
• Multi-query retrieval generates several reformulations of the same question, retrieves for each, and fuses the results: it covers more ground than a single query can
• HyDE generates a hypothetical document that would answer the question, then searches for real documents similar to it; it works because an answer-shaped text embeds closer to the real answer than a question-shaped text does

The problem: your documents and your users speak different languages

A user types "I got logged out, how do I get back in?" Your documentation says "Session expiry and re-authentication flow." The vector embedding of the user's query is close to documents about "logging out intentionally," not documents about session expiry. The user's language is colloquial; the doc's language is technical.

Query rewriting solves this by running the query through an LLM before retrieval. The LLM expands "I got logged out" into something closer to the document vocabulary: "session expiration re-authentication token refresh." Now the embedding matches the right documents.

The same technique applies to:

• Ambiguous queries: "JWT size" could mean token payload size or cryptographic key size. Rewriting adds specificity.
• Multi-part questions: "How does JWT work and when should I use it?" becomes two retrievals: one for the mechanism, one for the use case.
• Context-dependent queries: "How do I revoke it?" needs the LLM to infer "it" refers to a JWT from conversation history.

Options & when to use each

Technique	What it does	Best for	Costs you
Single-query rewriting	LLM expands the query with domain terms and fixes ambiguity	Vocabulary gap between user language and doc language	One extra LLM call per query (~100ms, ~$0.001)
Multi-query retrieval	LLM generates 3-5 reformulations, retrieves for each, fuses with RRF	Broad or ambiguous queries where a single retrieval might miss	3-5x retrieval cost; total latency is max of parallel retrievals
HyDE	LLM generates a fake answer document, embeds it, searches for similar real docs	Queries where the answer has a very different shape from the question (e.g., "What's the policy on..." vs the policy text itself)	One extra LLM call + one extra embedding
Query decomposition	LLM breaks a complex query into sub-questions, retrieves and answers each separately	Multi-hop questions that require information from different documents	Multiple LLM calls for both rewriting and answering

Build it: three query rewriting patterns

Pattern 1: single-query expansion

Here is a concrete example demonstrating the implementation details.

def rewrite_query(raw_query: str, conversation_history: list[str] | None = None) -> str:
    """
    Expand a user query with domain terminology and resolve pronouns.
    Optionally uses conversation history for context.
    """
    system = """You are a query rewriter for a RAG system about JWT authentication.
    Rewrite the user's query to be specific, self-contained, and use
    technical terminology that matches documentation language.
    Do not answer the query. Only rewrite it. Output the rewritten query only."""

    history_block = ""
    if conversation_history:
        history_block = "Conversation history:\n" + "\n".join(
            f"- {msg}" for msg in conversation_history[-3:]
        ) + "\n"

    user = f"{history_block}User query: {raw_query}\nRewritten query:"

    response = call_llm(system=system, message=user)
    return response.strip()

Here is a concrete example demonstrating the implementation details.

# Before rewriting
raw = "I keep getting kicked out, it's annoying"
print("Raw query:", raw)
# -> "I keep getting kicked out, it's annoying"

rewritten = rewrite_query(raw)
print("Rewritten:", rewritten)
# -> "JWT session expiration handling and preventing unexpected token expiry"

Pattern 2: multi-query retrieval with fusion

Here is a concrete example demonstrating the implementation details.

from concurrent.futures import ThreadPoolExecutor, as_completed

def rewrite_to_multiple(raw_query: str, n_variants: int = 3) -> list[str]:
    """Generate multiple query reformulations."""
    system = """You are a query rewriter. Generate alternative phrasings of the user's
    question that might match different documents. Include at least one formulation
    that uses highly specific technical terms and one that is more general."""

    user = f"Generate {n_variants} different search queries for: {raw_query}\nQueries:"

    response = call_llm(system=system, message=user)
    # Parse each line as a separate query
    queries = [line.strip("- 0123456789. ") for line in response.split("\n") if line.strip()]
    return [raw_query] + queries[:n_variants]  # include original


def multi_query_retrieval(
    raw_query: str,
    top_k: int = 10,
    n_variants: int = 3,
) -> list[dict]:
    """
    Generate multiple query variants, retrieve for each in parallel,
    and fuse results with reciprocal rank fusion.
    """
    queries = rewrite_to_multiple(raw_query, n_variants)

    # Parallel retrieval
    all_results = []
    with ThreadPoolExecutor(max_workers=len(queries)) as executor:
        futures = {
            executor.submit(hybrid_search, q, top_k=top_k * 2): q
            for q in queries
        }
        for future in as_completed(futures):
            all_results.extend(future.result())

    # Reciprocal rank fusion across all result sets
    doc_scores: dict[int, float] = {}
    doc_contents: dict[int, str] = {}
    k = 60

    for result_set in [all_results[i::len(queries)] for i in range(len(queries))]:
        for rank, doc in enumerate(result_set, start=1):
            doc_id = doc["id"]
            doc_scores[doc_id] = doc_scores.get(doc_id, 0) + 1.0 / (k + rank)
            doc_contents[doc_id] = doc["content"]

    # Sort by fused score and return top k
    ranked = sorted(doc_scores.items(), key=lambda x: x[1], reverse=True)
    return [
        {"id": doc_id, "content": doc_contents[doc_id], "rrf_score": score}
        for doc_id, score in ranked[:top_k]
    ]

Pattern 3: HyDE (Hypothetical Document Embeddings)

Here is a concrete example demonstrating the implementation details.

def hyde_retrieval(raw_query: str, top_k: int = 10) -> list[dict]:
    """
    Generate a hypothetical document that would answer the query,
    embed it, and retrieve real documents similar to that embedding.
    """
    system = """You are a technical writer. Write a short paragraph (3-5 sentences)
    that would appear in documentation answering the user's question.
    Write in the style of API documentation: factual, specific, uses
    correct technical terminology. Do not start with 'Here is...' or
    'The answer is...'. Write the paragraph directly."""

    hypothetical_doc = call_llm(system=system, message=raw_query)

    # Embed the hypothetical document instead of the query
    hypo_embedding = get_embedding(hypothetical_doc)

    # Search for real documents close to the hypothetical one
    results = conn.execute("""
        SELECT id, content, embedding <=> %s::vector AS distance
        FROM documents
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (hypo_embedding, hypo_embedding, top_k)).fetchall()

    return [
        {"id": row[0], "content": row[1], "distance": row[2]}
        for row in results
    ]

HyDE works because an answer-shaped text ("JWT tokens include an exp claim set to a Unix timestamp. The server rejects tokens whose exp is in the past. To set expiration, include exp in the payload when signing.") embeds much closer to a real documentation paragraph than a question-shaped text ("How do I set JWT expiration?") does. The question omits the words that appear in the answer -- exp, claim, timestamp, payload -- which HyDE generates.

When to use which

Here is a concrete example demonstrating the implementation details.

def smart_retrieve(raw_query: str, top_k: int = 10) -> list[dict]:
    """Choose the right rewriting strategy based on query characteristics."""
    words = raw_query.split()

    if len(words) <= 3:
        # Very short query: expand it
        expanded = rewrite_query(raw_query)
        return hybrid_search(expanded, top_k=top_k)

    if "?" not in raw_query and len(words) <= 5:
        # Keyword-like query, probably missing context
        expanded = rewrite_query(raw_query)
        return hybrid_search(expanded, top_k=top_k)

    if any(kw in raw_query.lower() for kw in [" and ", " or ", "compare", "versus", "vs"]):
        # Multi-part query: decompose and retrieve
        return multi_query_retrieval(raw_query, top_k=top_k)

    # Default: try HyDE for complex questions
    return hyde_retrieval(raw_query, top_k=top_k)

What goes wrong

Mistake	How you notice it	The fix
Rewritten query is too specific	Retrieval returns zero or very few results (over-constrained)	Add to the rewrite prompt: "If the query is already specific enough, return it unchanged." Fall back to original query if rewritten returns nothing
Rewritten query hallucinates terms	Retrieved documents are about a different topic entirely; the LLM added a term that doesn't exist in your corpus	Include corpus vocabulary hints in the rewrite prompt. Keep a running list of key terms from your documents and pass them as context
HyDE generates a wrong answer	Retrieved documents match the hypothetical wrong answer instead of the correct one	HyDE doesn't need to be factually correct; it needs to be structurally similar to a real answer. If the LLM hallucinates content, the embedding still lands in the right neighborhood. But if it invents an entirely different topic, retrieval fails. Validate HyDE output against known topics
Multi-query overhead is too high	Latency spikes to 2-3 seconds for a single user query	Run retrievals in parallel (ThreadPoolExecutor above). Cache rewritten queries. Skip multi-query for factual lookups
LLM cost dominates retrieval cost	Your API bill is 10x higher after adding rewriting, with minimal metric improvement	Measure recall before and after. If rewriting only improves recall@5 by 0.02, it's not worth the cost. Use the evaluation pipeline from the previous lesson to make this decision with data

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Test: a query in casual language that should match technical docs
raw_query = "why does it keep logging me out every few minutes"

# Without rewriting
raw_results = hybrid_search(raw_query, top_k=5)
raw_top = raw_results[0]["content"][:200] if raw_results else "(no results)"

# With rewriting
rewritten = rewrite_query(raw_query)
rewritten_results = hybrid_search(rewritten, top_k=5)
rewritten_top = rewritten_results[0]["content"][:200] if rewritten_results else "(no results)"

# With HyDE
hyde_results = hyde_retrieval(raw_query, top_k=5)
hyde_top = hyde_results[0]["content"][:200] if hyde_results else "(no results)"

print(f"Raw query top result:       {raw_top}")
print(f"Rewritten query top result: {rewritten_top}")
print(f"HyDE top result:            {hyde_top}")
# Expected: rewritten and HyDE results should contain technical terms
# like "exp", "expiration", "token lifetime" that the raw query missed

Next: Handling Failures