Agentic RAG

In a fixed RAG pipeline, retrieval always happens before generation. The system retrieves chunks, stuffs them into the prompt, and the model answers. This works for simple Q&A. It fails when the model needs to retrieve multiple times, refine its query based on what it finds, or decide that retrieval isn't necessary at all. Agentic RAG gives the model a retrieval tool and lets it decide when to call it, what to search for, and when to stop. This is the pattern that powers ChatGPT's browsing, Claude's web search, and Perplexity's research mode.

What you'll learn

• Agentic RAG gives the model a retrieval tool and lets it decide when to call it, not a fixed pipeline step
• The model can retrieve multiple times, refine its query based on intermediate results, and stop when it has enough information
• This handles questions where you don't know in advance whether retrieval is needed, or how many retrievals will be required

The problem

A user asks "What's the difference between HNSW and IVFFlat indexes?" A fixed RAG pipeline retrieves chunks, stuffs them into context, and the model answers. This works fine. But the next user asks "Should I use HNSW or IVFFlat for my 10M vector dataset with high update frequency?" The fixed pipeline retrieves the same generic chunks about HNSW and IVFFlat. The model gives a generic answer that doesn't account for the specific constraints (10M vectors, high update frequency). What the model actually needs is to first retrieve general information about both index types, then realize it needs specific information about update performance at scale, then retrieve that. A fixed pipeline can't do this because it only retrieves once, before the model has seen any results.

Agentic RAG solves this by giving the model a search_knowledge_base tool. The model decides whether to search, what to search for, reads the results, and decides whether to search again with a refined query. The loop continues until the model determines it has enough information to answer. This is the same pattern as the agent loop from Day 1 of the flagship course — the model observes, decides, acts, and observes again.

Options & when to use each

Approach	Retrievals per query	When to use
Fixed RAG	Always 1	Simple Q&A where one retrieval always suffices
Agentic RAG	0 to N, model decides	Questions where retrieval need varies, or where follow-up retrieval is needed
Multi-hop RAG	Predetermined N hops	When you know the question type always needs N steps
Hybrid	Agentic with max 3 retrievals	Production — get the flexibility of agentic with a cost safety net

Build it

Implementing agentic RAG requires defining a tool schema and an execution loop. The following code sets up the tool definition and allows the model to iterate until it finds the required information.

from anthropic import Anthropic

client = Anthropic()

SYSTEM_PROMPT = """You answer questions using a knowledge base search tool.

When you need information, call search_knowledge_base with a specific query.
Read the results and decide if you need to search again with a different query.
When you have enough information, answer the user's question.

If you search and find nothing useful, tell the user you couldn't find the answer.
Do not guess or fabricate information not in the search results."""

def agentic_rag_query(question, max_turns=5):
    messages = [{"role": "user", "content": question}]
    search_count = 0

    for turn in range(max_turns):
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            system=SYSTEM_PROMPT,
            tools=[{
                "name": "search_knowledge_base",
                "description": "Search the knowledge base for relevant documents. Use specific queries. If the first search doesn't find what you need, try a different query.",
                "input_schema": {
                    "type": "object",
                    "properties": {
                        "query": {"type": "string", "description": "Specific search query. Be precise — use technical terms, entity names, and key phrases."}
                    },
                    "required": ["query"]
                }
            }],
            messages=messages,
        )

        if response.stop_reason == "tool_use":
            tool_call = response.content[-1]
            results = vector_search(tool_call.input["query"], top_k=5)
            search_count += 1

            messages.append({"role": "assistant", "content": response.content})
            messages.append({
                "role": "user",
                "content": [{"type": "tool_result", "tool_use_id": tool_call.id, "content": format_results(results)}]
            })
        else:
            return response.content[0].text, search_count

    return "I searched multiple times but couldn't find a complete answer.", search_count

What goes wrong

Mistake	How you notice it	The fix
Model retrieves too many times	Costs 5x normal, latency is high	Cap turns at 3 for production. Log search count and alert if average exceeds 2
Model doesn't retrieve when it should	Answer is wrong or missing key facts from the knowledge base	Strengthen the system prompt: "When uncertain, search first. It's better to search and find nothing than to guess."
Model retrieves the same thing repeatedly	Search count is high but results are identical	Track previous queries. If cosine similarity to any previous query exceeds 0.85, reject and force the model to answer
Model ignores search results	Retrieved facts contradict the answer, or the answer doesn't cite them	Add "Cite specific facts from the search results in your answer" to the system prompt

Confirm it worked

Test the implementation by indexing a few known facts and running queries of varying complexity. You can verify that simple queries trigger a single retrieval while complex questions drive multiple searches.

index_documents([
    {"id": "1", "content": "HNSW indexes build a graph structure. They are fast for queries but slow to update."},
    {"id": "2", "content": "IVFFlat indexes cluster vectors. They are slower for queries but easier to update incrementally."},
])

# Simple question: model should retrieve once
answer, count = agentic_rag_query("What is an HNSW index?")
assert count == 1

# Complex question: model should retrieve twice
answer, count = agentic_rag_query("I have 10M vectors and need frequent updates. HNSW or IVFFlat?")
assert count >= 2  # Should retrieve general info, then follow up on update performance

Next: Structured Data + Text