Multi-Hop Retrieval

Some questions cannot be answered from a single chunk. "Who is the CEO of the company that acquired our biggest competitor last year?" That single sentence requires retrieving three separate facts: who our biggest competitor is, which company acquired them, and who runs that company. A single vector search will find chunks about competitors, chunks about acquisitions, and chunks about CEOs, but no single chunk contains the chain of reasoning that connects them. Multi-hop retrieval chains multiple searches together, each one using the results of the last as the query for the next.

What you'll learn

• Multi-hop retrieval chains searches: the first retrieval answers part of the question, the second uses that answer as context for the next query
• Only worth it when questions genuinely require connecting facts across multiple documents — most real-world queries are single-hop
• The LLM decides whether a second hop is needed by evaluating whether the first retrieval completely answers the question

The problem

A user asks a compound question: "What year was the founder of the company that made the first GPU born?" A single vector search for this entire sentence will find chunks about GPU history, maybe chunks about company founders. But the question requires a chain: identify the first GPU maker, find its founder, then find that person's birth year. Each step depends on the previous one, and no single chunk in your knowledge base contains all three facts. If you stop after one retrieval, the LLM either guesses the founder's birth year from its training data (hallucination risk) or admits it doesn't know (correct but unhelpful).

Multi-hop retrieval models this as an iterative process. The first hop retrieves information about the first GPU. The LLM reads those results and extracts a follow-up question: "Who founded Nvidia?" The second hop searches for that. The LLM reads those results and extracts another follow-up: "When was Jensen Huang born?" The third hop answers that. Each retrieval builds on the last, and the chain terminates when the LLM determines it has enough information to answer the original question.

Options & when to use each

Approach	When to use	Cost	Risk
Single-hop retrieval	Most queries — "what is X," "how do I Y"	Lowest	Misses answers that require connecting facts
Fixed two-hop retrieval	When you know the query pattern always needs two steps	Medium	Wastes a hop on queries that only needed one
LLM-decided multi-hop	Compound questions where you don't know in advance how many hops are needed	Highest	Model might request unnecessary hops or miss needed ones
Decomposed querying	Break the question into sub-questions upfront, retrieve for each	Medium	Requires the LLM to correctly decompose the question before seeing any results

For most applications, LLM-decided multi-hop with a hard cap of 3 hops is the right balance. Single-hop queries get answered in one call. Compound queries get the extra retrievals they need. And the cap prevents runaway costs on questions that would never be answerable from your knowledge base.

Build it

Step 1: Determine if a second hop is needed

To prevent infinite loops, the system must assess whether it has enough context to answer the user. This function asks the LLM to evaluate the retrieved documents and identify any specific facts that are still missing.

def needs_another_hop(original_question, all_results_so_far):
    """Ask the LLM whether it can answer the question from current results."""
    prompt = f"""Original question: {original_question}

Information retrieved so far:
{format_results(all_results_so_far)}

Can you answer the original question completely from this information?
Answer ONLY: YES or NO

If NO, what specific information is still missing? Be specific — name the entity, fact,
or relationship you still need to look up."""
    
    response = call_llm(prompt)
    can_answer = response.strip().upper().startswith("YES")
    missing_info = response.split("\n", 1)[1] if not can_answer and "\n" in response else ""
    return can_answer, missing_info

Step 2: Generate the follow-up query

When missing information is detected, the next step is to formulate a targeted query to find it. This code prompts the LLM to write a concise search string based on the identified gaps.

def generate_hop_query(original_question, results_so_far, missing_info):
    """Generate a specific search query for the missing information."""
    prompt = f"""Original question: {original_question}

We found: {summarize_results(results_so_far)}

Missing information: {missing_info}

Generate a concise search query to find this specific missing information.
The query should be a single sentence optimized for vector search.
Do not repeat information we already have."""

    return call_llm(prompt).strip()

Step 3: Full multi-hop pipeline

The complete multi-hop pipeline iterates through retrieval, evaluation, and query generation. It enforces a maximum hop limit and uses similarity checks to prevent the system from getting stuck in redundant search cycles.

def multi_hop_retrieve(question, max_hops=3):
    all_results = []
    current_query = question

    for hop in range(max_hops):
        results = vector_search(current_query, top_k=5)
        all_results.extend(results)

        can_answer, missing = needs_another_hop(question, all_results)
        if can_answer:
            break

        current_query = generate_hop_query(question, all_results, missing)

        # Safety: if the new query is too similar to a previous one, stop
        if is_similar_to_previous(current_query, previous_queries):
            break
        previous_queries.append(current_query)

    # Deduplicate results before returning
    seen_ids = set()
    unique_results = []
    for r in all_results:
        if r["id"] not in seen_ids:
            seen_ids.add(r["id"])
            unique_results.append(r)

    return unique_results, hop + 1

The duplicate detection on hop queries is important. Without it, the LLM can get stuck in a loop: "Who founded Nvidia?" returns "Jensen Huang." The LLM asks "Who is Jensen Huang?" which returns the same biography. The LLM asks "Tell me about Jensen Huang" which returns the same results. The similarity check breaks this cycle.

What goes wrong

Mistake	How you notice it	The fix
Unnecessary hops on simple questions	"What is a GPU?" triggers 3 retrievals that all return the same definition	The "can you answer" prompt must be conservative. Default answer is YES unless specific information is clearly missing
Hop query drifts off topic	Original: "GPU founder birth year." Third hop: "History of Silicon Valley"	Compute cosine similarity between the original question and each hop query. Reject hops below a threshold (0.3 works well)
Hop limit reached without answering	Pipeline gives up after 3 hops with incomplete information	Return what was found with a note about what's missing. Never fabricate. "I found that Nvidia made the first GPU and was founded by Jensen Huang, but I could not determine his birth year from the available documents"
Identical results across hops	Each hop returns the same top-5 chunks, wasting API calls	Deduplicate results by document ID across hops. Track which chunks were already seen

Confirm it worked

Test the multi-hop logic by distributing related facts across separate documents. A successful execution will demonstrate the system traversing multiple documents to assemble the final answer.

# Set up test data with facts spread across documents
index_documents([
    {"id": "1", "content": "The first GPU was the GeForce 256, made by Nvidia in 1999."},
    {"id": "2", "content": "Nvidia was founded by Jensen Huang, Chris Malachowsky, and Curtis Priem."},
    {"id": "3", "content": "Jensen Huang was born on February 17, 1963 in Taiwan."},
])

question = "What year was the founder of the company that made the first GPU born?"
results, hops = multi_hop_retrieve(question)

# Should find the connection: GeForce 256 -> Nvidia -> Jensen Huang -> 1963
all_text = " ".join(r["content"] for r in results)
assert "GeForce 256" in all_text or "Nvidia" in all_text
assert "Jensen Huang" in all_text
assert "1963" in all_text
assert hops >= 2  # Required at least 2 retrievals

Next: Agentic RAG