Recursive Chunking

Fixed-size and semantic chunking both face the same tradeoff: pick a chunk size before you know what the user will ask. A chunk that is small enough for precise retrieval is often too small for the model to answer the question. Recursive chunking sidesteps this by building a hierarchy: small chunks for retrieval, large chunks for context.

What you'll learn

• Recursive chunking builds a tree of chunks at multiple granularities: the smallest chunks match queries precisely, and their parent chunks provide the surrounding context the model needs to answer
• The retrieval step finds the best-matching small chunk, then the pipeline expands to include its parent (and optionally grandparent) before sending context to the model
• This architecture decouples the chunk size for retrieval from the chunk size for generation, solving the most persistent tension in RAG design

The problem

You chunk a legal contract into 256-token pieces. A user asks "What are the termination conditions?" The vector search finds the chunk that says "Either party may terminate with 30 days written notice." The model answers correctly about notice periods. But it misses the sentence two paragraphs earlier that says "Termination does not apply to obligations under Section 4.2." That sentence was in a different chunk.

The root issue is that 256 tokens was the right size for retrieval -- small enough to match the query precisely -- but the wrong size for generation, because the relevant context was spread across multiple chunks. Recursive chunking solves this by keeping both sizes.

Options & when to use each

Strategy	What it is good for	What it costs you	When to pick it
Two-level hierarchy (small retrieval + large context)	Simple parent-child; small chunks find the needle, large chunks show the haystack	Extra storage for parent chunks; more complex ingestion	Most production RAG systems. Start here before going deeper
Multi-level hierarchy (small, medium, large)	Handles documents with deeply nested structure like contracts or textbooks	More storage, more retrieval logic, diminishing returns after 3 levels	Structured long-form documents where context requirements vary widely by query type
Window retrieval (small chunk + surrounding N chunks)	Simpler than parent-child; no metadata needed, just sequential offset	Context window can get large fast; no semantic grouping guarantee	Documents where adjacency is a reliable proxy for relevance, like chronological narratives

Build it

Two-level recursive chunking

The core idea: split a document at two different granularities. Store both sets of chunks with parent-child relationships. On retrieval, find the best small chunk, then expand to its parent.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def split_recursive(
    text: str,
    child_size: int = 128,
    parent_size: int = 512,
    child_overlap: int = 10,
    parent_overlap: int = 25,
) -> tuple[list[str], list[str], list[tuple[int, int]]]:
    """
    Split text into child and parent chunks with parent-child mapping.

    Returns:
        child_chunks: small chunks for retrieval
        parent_chunks: large chunks for context
        mapping: list of (child_index, parent_index) tuples
    """
    # First pass: split into parent-sized chunks
    parent_tokens = enc.encode(text)
    parent_chunks = []
    parent_starts = []
    p_start = 0
    while p_start < len(parent_tokens):
        p_end = min(p_start + parent_size, len(parent_tokens))
        parent_chunks.append(enc.decode(parent_tokens[p_start:p_end]))
        parent_starts.append(p_start)
        p_start += parent_size - parent_overlap

    # Second pass: split each parent into children
    child_chunks = []
    mapping = []

    for p_idx, (p_chunk, p_start) in enumerate(zip(parent_chunks, parent_starts)):
        child_tokens = enc.encode(p_chunk)
        c_start = 0
        while c_start < len(child_tokens):
            c_end = min(c_start + child_size, len(child_tokens))
            child_chunks.append(enc.decode(child_tokens[c_start:c_end]))
            mapping.append((len(child_chunks) - 1, p_idx))
            c_start += child_size - child_overlap

    return child_chunks, parent_chunks, mapping

Ingesting recursive chunks into pgvector

Here is a concrete example demonstrating the implementation details.

def ingest_recursive(conn, document_id: str, text: str):
    """Ingest a document with two-level recursive chunking."""
    child_chunks, parent_chunks, mapping = split_recursive(text)

    cur = conn.cursor()

    # Insert parent chunks (stored but not embedded for retrieval)
    parent_ids = []
    for i, chunk in enumerate(parent_chunks):
        cur.execute("""
            INSERT INTO chunks (document_id, chunk_index, content, level, token_count)
            VALUES (%s, %s, %s, 'parent', %s) RETURNING id
        """, (document_id, i, chunk, len(enc.encode(chunk))))
        parent_ids.append(cur.fetchone()[0])

    # Insert child chunks with embeddings and parent references
    for child_idx, parent_idx in mapping:
        chunk = child_chunks[child_idx]
        embedding = openai.embeddings.create(
            model="text-embedding-3-small", input=chunk
        ).data[0].embedding

        cur.execute("""
            INSERT INTO chunks (document_id, chunk_index, content, embedding,
                                level, parent_chunk_id, token_count)
            VALUES (%s, %s, %s, %s, 'child', %s, %s)
        """, (
            document_id, child_idx, chunk, embedding,
            parent_ids[parent_idx],
            len(enc.encode(chunk))
        ))

    conn.commit()
    print(f"Ingested {len(parent_chunks)} parent chunks, {len(child_chunks)} child chunks")

Retrieval with parent expansion

Here is a concrete example demonstrating the implementation details.

def retrieve_with_context(conn, query: str, top_k: int = 3) -> list[dict]:
    """Retrieve child chunks by vector similarity, then expand to parents."""

    query_embedding = openai.embeddings.create(
        model="text-embedding-3-small", input=query
    ).data[0].embedding

    cur = conn.cursor()

    # Find best-matching child chunks
    cur.execute("""
        SELECT id, content, parent_chunk_id,
               1 - (embedding <=> %s::vector) AS similarity
        FROM chunks
        WHERE level = 'child'
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (query_embedding, query_embedding, top_k))

    child_results = cur.fetchall()

    # Collect unique parent IDs
    parent_ids = list({row[2] for row in child_results if row[2] is not None})

    # Fetch parent content
    cur.execute("""
        SELECT id, content FROM chunks WHERE id = ANY(%s)
    """, (parent_ids,))
    parent_map = {row[0]: row[1] for row in cur.fetchall()}

    # Build results: each hit includes the child chunk and its parent
    results = []
    for chunk_id, child_content, parent_id, similarity in child_results:
        results.append({
            "child_chunk": child_content,
            "parent_context": parent_map.get(parent_id, ""),
            "similarity": similarity,
        })

    return results

# Usage
results = retrieve_with_context(conn, "termination conditions in the contract")
for r in results:
    print(f"Similarity: {r['similarity']:.3f}")
    print(f"  Child:  {r['child_chunk'][:100]}...")
    print(f"  Parent: {r['parent_context'][:150]}...")
    print()

Window retrieval: a simpler alternative

If you do not need semantic grouping and your documents are roughly sequential, window retrieval is simpler to implement than parent-child:

def retrieve_with_window(conn, query: str, top_k: int = 3, window: int = 2) -> str:
    """Retrieve top-k chunks and expand to surrounding chunks by index."""
    query_embedding = openai.embeddings.create(
        model="text-embedding-3-small", input=query
    ).data[0].embedding

    cur = conn.cursor()
    cur.execute("""
        SELECT id, document_id, chunk_index, content,
               1 - (embedding <=> %s::vector) AS similarity
        FROM chunks
        WHERE level = 'child'
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (query_embedding, query_embedding, top_k))

    hits = cur.fetchall()

    # For each hit, fetch surrounding chunks from the same document
    expanded = []
    seen = set()
    for _, doc_id, chunk_idx, content, sim in hits:
        cur.execute("""
            SELECT content FROM chunks
            WHERE document_id = %s
              AND chunk_index BETWEEN %s AND %s
              AND level = 'child'
            ORDER BY chunk_index
        """, (doc_id, chunk_idx - window, chunk_idx + window))
        for (neighbor_content,) in cur.fetchall():
            if neighbor_content not in seen:
                expanded.append(neighbor_content)
                seen.add(neighbor_content)

    return "\n\n".join(expanded)

What goes wrong

Mistake	How you notice it	The fix
Parent chunks are too large	Context window fills up after expanding 2-3 child chunks; the model runs out of room	Cap parent chunks at 512-1024 tokens. If the document needs more context than that, the model probably cannot use it all effectively anyway
Parent chunk maps to the wrong context	The expanded context is about a different topic than the child chunk suggested	Your parent chunks are too broad. Tighten parent chunk size or switch to window retrieval where context is guaranteed to be adjacent
Embedding child chunks without also storing parent content	You cannot expand to parents after retrieval because they were never persisted	Store both levels. The extra storage is small compared to the cost of re-embedding your entire corpus when you realize you need parents
Window retrieval pulls in unrelated neighbors	A chapter boundary falls within the window; context includes the end of the previous chapter	Use semantic boundaries to prevent windows from crossing chapter or section breaks. Check chunk_index against section metadata if available
Too many hierarchy levels	Retrieval latency spikes; the pipeline has too many database round-trips	Two levels is enough for most systems. Add a third only if you have a measured, documented retrieval failure at two levels

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Verify that parent expansion adds relevant context not in the child chunk
test_text = (
    "Section 1: Overview. This agreement governs the use of the service. "
    "Section 2: Payment. The customer agrees to pay all fees within 30 days. "
    "Late payments incur a 1.5% monthly interest charge. "
    "Section 3: Termination. Either party may terminate with 30 days written notice. "
    "Termination does not relieve the customer of payment obligations incurred prior "
    "to the termination date. All outstanding fees become immediately due upon termination."
)

child_chunks, parent_chunks, mapping = split_recursive(
    test_text, child_size=20, parent_size=60, child_overlap=5, parent_overlap=10
)

# Find the child chunk that contains "Termination"
termination_child = None
for i, chunk in enumerate(child_chunks):
    if "Termination" in chunk:
        termination_child = i
        break

assert termination_child is not None, "Should find a child chunk about termination"

# Find its parent
parent_idx = mapping[termination_child][1]
parent = parent_chunks[parent_idx]

# The parent should contain context the child does not
# (the payment obligations clause that follows the termination notice clause)
print(f"Child chunk:  {child_chunks[termination_child]}")
print(f"Parent chunk: {parent}")

# Verify the parent contains payment obligations context
assert "payment obligations" in parent.lower() or "outstanding fees" in parent.lower(), \
    "Parent should include the payment context that the child chunk lacks"
print("Parent contains additional context: OK")

Next: Metadata & Filtering