Conversational RAG

A single-turn RAG query has no memory. Ask "what's the return policy?" and the system retrieves the policy document. Ask "what about for electronics?" and the system retrieves nothing useful because "what about for electronics" means nothing without the context of the previous question. Conversational RAG threads retrieval through a conversation by reformulating each turn's query to include the relevant history. The user types "what about for electronics?" and the system searches for "what is the return policy for electronics?" because it knows what "what about" refers to.

What you'll learn

• Conversational RAG reformulates the user's query using conversation history before retrieval
• A standalone question generator rewrites "what about electronics?" as "what is the return policy for electronics?"
• The conversation history is used only for query reformulation, not stuffed into the retrieval query

The problem

A user has a three-turn conversation with your RAG system. Turn 1: "What's your return policy?" — the system retrieves the policy document and answers. Turn 2: "How long do I have?" — the system searches for "how long do I have" and finds nothing relevant. Turn 3: "What about international orders?" — the system searches for "international orders" and retrieves the international shipping policy instead of the international return policy. Each turn after the first fails because the retrieval query has lost the context that the human is still using.

The naive fix is to stuff the entire conversation history into every retrieval query. This works poorly because conversation history contains greetings, acknowledgments, and tangents that dilute the retrieval signal. The correct fix is query reformulation: before each retrieval, generate a standalone question that captures all the relevant context from the conversation history, then use that standalone question for retrieval. The history is used only for generating the question, not for the search itself.

Build it

Step 1: Reformulate the query with conversation context

The first component is a prompt that instructs the LLM to rewrite the user's latest input. By passing in the recent conversation turns, the model generates a self-contained question that is optimized for standalone retrieval.

def reformulate_query(current_question, conversation_history):
    """Generate a standalone question that captures conversation context."""
    if not conversation_history:
        return current_question

    history_text = "\n".join(
        f"User: {turn['user']}\nAssistant: {turn['assistant']}"
        for turn in conversation_history[-3:]  # Last 3 turns only
    )

    prompt = f"""Given the conversation history and the latest user question,
produce a standalone question that captures all relevant context from the history.
The standalone question should be self-contained — someone reading it without
seeing the history should understand what's being asked.

Conversation history:
{history_text}

Latest user question: {current_question}

Standalone question:"""

    return call_llm(prompt, max_tokens=100).strip()

Step 2: Full conversational retrieval pipeline

With the query reformulated, the pipeline executes the vector search and constructs the final prompt. This function combines the retrieved documents and the conversation history so the model can generate a fully contextualized answer.

def conversational_retrieve(question, conversation_history):
    # Step 1: Reformulate with context
    standalone = reformulate_query(question, conversation_history)

    # Step 2: Retrieve with the standalone question
    results = vector_search(standalone)

    # Step 3: Generate answer with full context
    context_text = format_results(results)
    history_text = format_history(conversation_history[-5:])

    answer_prompt = f"""Previous conversation:
{history_text}

Relevant documents retrieved:
{context_text}

User's latest question: {question}

Answer the question based on the retrieved documents.
If the documents don't contain the answer, say so.
Reference the conversation history where relevant."""

    answer = call_llm(answer_prompt)

    return answer, standalone, results

Step 3: Track reformulated queries for debugging

In production, it is crucial to observe how queries are being modified. This snippet introduces structured logging and similarity checks to ensure the reformulated query hasn't drifted semantically from the original intent.

import structlog
logger = structlog.get_logger()

def conversational_retrieve_with_logging(question, history):
    standalone = reformulate_query(question, history)
    
    logger.info("query_reformulated",
        original=question[:100],
        reformulated=standalone[:100],
        history_turns=len(history) if history else 0,
    )

    results = vector_search(standalone)

    # Alert if reformulation changed the meaning significantly
    if history and cosine_similarity(embed(question), embed(standalone)) < 0.3:
        logger.warning("query_reformulation_diverged",
            original=question,
            reformulated=standalone,
        )

    return results

What goes wrong

Mistake	How you notice it	The fix
Reformulated question loses original intent	"What about electronics?" becomes "Tell me about electronics" instead of "Return policy for electronics"	Log both original and reformulated queries. If cosine similarity drops below 0.3, fall back to the original
Reformulation adds hallucinated context	"What about electronics?" becomes "Return policy for electronics at Best Buy" when the user never mentioned Best Buy	Compare the reformulated query to the conversation history. Flag if it introduces entities not in the history
Too much history bloats the reformulation prompt	Reformulation prompt exceeds token limit, or takes too long	Only include the last 3-5 turns. Summarize older turns rather than including them verbatim
Follow-up question is completely unrelated	User switches topics mid-conversation, but reformulation still drags in old context	Check if the new question has low semantic similarity to recent history. If below a threshold, treat it as a new topic

Confirm it worked

Validate the conversational pipeline by simulating a multi-turn interaction. The test ensures that follow-up questions correctly resolve references and pull the necessary documents.

history = [
    {"user": "What's your return policy?", "assistant": "You can return items within 30 days for a full refund."},
]

# Follow-up should maintain context
question = "What about for electronics?"
standalone = reformulate_query(question, history)
assert "return policy" in standalone.lower()
assert "electronics" in standalone.lower()

# Verify retrieval quality
results = conversational_retrieve(question, history)
assert len(results) > 0
assert any("return" in r["content"].lower() for r in results)

Next: Day 5 — Production RAG Systems