Handling Failures

Every retrieval pipeline fails sometimes. The query is about a topic not in your corpus. The relevant document exists but was split across chunk boundaries. The user asks a question in a language your embedding model doesn't support well. The failure itself isn't the problem -- every system has gaps. The problem is what your system does next.

An LLM that receives an empty or irrelevant context will still generate an answer. It doesn't know the context is bad. It sees "no documents found" and fills in the gaps from its training data, which might be outdated, wrong, or invented. Handling retrieval failures means detecting when the pipeline didn't find good results and responding appropriately: fall back, ask for clarification, or admit you don't know.

What you'll learn

• An LLM given empty or irrelevant retrieval context will still answer -- it hallucinates from training data instead of refusing; you must detect the failure before the LLM sees it
• A fallback chain (hybrid -> keyword-only -> broader search -> "I don't know") gives each strategy a chance before admitting defeat
• Admitting ignorance is a feature, not a failure: "I couldn't find information about X in the documentation" is more trustworthy than a confident wrong answer

The problem: the LLM will answer regardless

Consider this pipeline:

Here is a concrete example demonstrating the implementation details.

results = hybrid_search("What is the warranty on the WD-40-BLACK-12PK?")
context = "\n".join(doc["content"] for doc in results)
answer = call_llm(f"Answer using this context:\n{context}\n\nQuestion: What is the warranty?")

If your documents contain zero warranty information, results might be empty or contain marginally related documents about shipping and defects. The LLM still generates an answer because that's what it's trained to do. It might invent a plausible-sounding warranty policy drawn from its training data about other products. The user has no way to know the answer isn't grounded in your actual documents.

The fix is a retrieval confidence check before the LLM receives the context.

Options & when to use each

Strategy	What it does	When to use
Relevance threshold	Reject results below a minimum relevance score (from cross-encoder or vector distance)	You have reliable relevance scores and can set a meaningful threshold
Fallback chain	Try multiple retrieval strategies in sequence: hybrid, then keyword-only, then broader search	Different query types fail in different ways; a chain catches more cases
Admit ignorance	Return "I don't know" or "I couldn't find that in the documentation"	The query is out of scope or your corpus genuinely doesn't contain the answer
Ask clarifying questions	Prompt the user to rephrase or be more specific	The query is ambiguous and a rephrasing might match
Semantic "out of domain" detection	Classify whether the query is about a topic your corpus covers at all	You want to filter before any retrieval attempt

Build it: a fallback chain with confidence thresholds

Step 1: define relevance thresholds

Here is a concrete example demonstrating the implementation details.

from dataclasses import dataclass

@dataclass
class RetrievalResult:
    documents: list[dict]
    strategy: str           # which retrieval method produced these
    confidence: float       # 0.0 to 1.0, how confident we are these are good

# Thresholds
HIGH_CONFIDENCE = 0.5       # Reranker score: good enough to use directly
LOW_CONFIDENCE = 0.2        # Below this, results are likely irrelevant
VECTOR_DISTANCE_OK = 0.3    # Cosine distance threshold (lower is better)

Step 2: build the fallback chain

Here is a concrete example demonstrating the implementation details.

def retrieve_with_fallback(
    query: str,
    top_k: int = 5,
) -> RetrievalResult:
    """
    Try retrieval strategies in sequence until we get confident results.
    Returns the best result set found, or an empty result with low confidence.
    """

    # Strategy 1: hybrid search with re-ranking
    candidates = hybrid_search(query, top_k=20)
    if candidates:
        reranked = retrieve_and_rerank(query)  # from re-ranking lesson
        if reranked and reranked[0].get("relevance_score", 0) >= HIGH_CONFIDENCE:
            return RetrievalResult(
                documents=reranked[:top_k],
                strategy="hybrid+rerank",
                confidence=reranked[0]["relevance_score"],
            )

    # Strategy 2: keyword-only search (vector search might have missed)
    text_results = conn.execute("""
        SELECT id, content,
               ts_rank(text_search, websearch_to_tsquery('english', %s)) AS rank
        FROM documents
        WHERE text_search @@ websearch_to_tsquery('english', %s)
        ORDER BY rank DESC
        LIMIT %s
    """, (query, query, top_k)).fetchall()

    if text_results:
        return RetrievalResult(
            documents=[{"id": r[0], "content": r[1], "score": r[2]} for r in text_results],
            strategy="keyword-only",
            confidence=0.5,  # Moderate: we found keyword matches
        )

    # Strategy 3: broader vector search with lower threshold
    embedding = get_embedding(query)
    broad_results = conn.execute("""
        SELECT id, content, embedding <=> %s::vector AS distance
        FROM documents
        WHERE embedding <=> %s::vector < %s
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (embedding, embedding, VECTOR_DISTANCE_OK * 2, embedding, top_k)).fetchall()

    if broad_results:
        return RetrievalResult(
            documents=[{"id": r[0], "content": r[1], "distance": r[2]} for r in broad_results],
            strategy="broad-vector",
            confidence=0.3,  # Low: these are distant matches
        )

    # Strategy 4: admit we found nothing
    return RetrievalResult(
        documents=[],
        strategy="none",
        confidence=0.0,
    )

Step 3: build the response handler

Here is a concrete example demonstrating the implementation details.

def answer_with_fallback(query: str) -> str:
    """
    Full pipeline: retrieve with fallback, then decide how to respond
    based on retrieval confidence.
    """
    retrieval = retrieve_with_fallback(query)

    if retrieval.confidence >= HIGH_CONFIDENCE:
        # Confident: generate a normal answer with citations
        context = "\n---\n".join(
            f"[{i+1}] {doc['content']}" for i, doc in enumerate(retrieval.documents)
        )
        return call_llm(
            system="Answer the question using only the provided context. "
                    "Cite sources with [1], [2], etc. If the context doesn't "
                    "contain the answer, say so.",
            message=f"Context:\n{context}\n\nQuestion: {query}"
        )

    elif retrieval.confidence >= LOW_CONFIDENCE:
        # Moderate: answer but flag low confidence
        context = "\n---\n".join(
            f"[{i+1}] {doc['content']}" for i, doc in enumerate(retrieval.documents)
        )
        return call_llm(
            system="Answer the question using the provided context. "
                    "If the context doesn't fully answer the question, "
                    "state what you found and what's missing. "
                    "Begin your response with 'I found partial information...' "
                    "if the answer is incomplete.",
            message=f"Context:\n{context}\n\nQuestion: {query}"
        )

    else:
        # No relevant results: admit ignorance
        return (
            f"I couldn't find information about '{query}' in the documentation. "
            "This topic may not be covered, or you might try rephrasing your question."
        )

Step 4: add query classification

Before retrieval, classify whether the query is even in your domain:

def is_in_domain(query: str, domain_description: str) -> bool:
    """Check if a query is about a topic the corpus covers."""
    system = (
        "You classify whether a user query is about JWT authentication "
        "and token management. Reply with only YES or NO."
    )
    response = call_llm(system=system, message=query)
    return "YES" in response.upper()

Here is a concrete example demonstrating the implementation details.

def answer_safely(query: str) -> str:
    """Full safe answering pipeline with domain check."""
    if not is_in_domain(query, "JWT authentication and token management"):
        return (
            "I'm a JWT documentation assistant and can't answer questions "
            "outside that domain. Try asking about JWT tokens, authentication, "
            "or session management."
        )

    return answer_with_fallback(query)

What goes wrong

Mistake	How you notice it	The fix
Threshold too strict	Legitimate queries return "I don't know" too often; users stop trusting the system	Lower the threshold. Log rejected queries and manually check whether relevant documents existed. If >20% of rejections were wrong, your threshold is too high
Threshold too loose	Low-confidence answers are presented as authoritative; users get incorrect information	Raise the threshold. When confidence is moderate, always include a hedge: "I found partial information..." as shown above
Fallback chain is too slow	Three retrieval strategies take 2+ seconds before returning	Run strategies in parallel instead of sequentially. Return the first result that crosses the confidence threshold. If the primary strategy is fast and usually works, skip the fallback for that case
"I don't know" response is unhelpful	Users get a dead end with no path forward	Always suggest next steps: "Try rephrasing your question to include specific terms like 'token expiration' or 'refresh token.'" Or: "Here are topics I can help with: JWT structure, token signing, expiration, refresh tokens."
LLM ignores the "admit ignorance" instruction	Even when told to say "I don't know," the LLM invents an answer	Use structured output: force the LLM to output JSON with a found_answer boolean. If false, return the canned response. If true, return the answer. This is more reliable than prompting alone

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Test 1: in-domain query with good retrieval
response = answer_safely("How do I set a JWT to expire after 1 hour?")
print("In-domain:", response[:150])
# Expected: a specific answer with citation markers [1], [2]

# Test 2: in-domain query with no relevant documents
response = answer_safely("What is the warranty on JWT tokens?")
print("No-match:", response[:150])
# Expected: "I couldn't find information about..." or similar admission

# Test 3: out-of-domain query
response = answer_safely("What's the weather in Tokyo?")
print("Out-of-domain:", response[:150])
# Expected: "I'm a JWT documentation assistant..." domain refusal

# Test 4: verify the chain runs without errors
result = retrieve_with_fallback("xyzzy123_nonexistent_term_999")
assert result.strategy in ("broad-vector", "none"), \
    f"Expected fallback to broad-vector or none, got {result.strategy}"
print(f"Fallback strategy used: {result.strategy}, confidence: {result.confidence}")
# Expected: strategy="none", confidence=0.0 for a nonsense query

Next: Day 4 -- Advanced RAG Patterns