The Chunking Problem

You have a corpus of documents. You generate embeddings for every document and store them in pgvector. You query, and the results are wrong -- the model gets context that is too broad to be useful or too narrow to answer the question. The root cause is not your embedding model. It is the decision you made before embedding: what counts as a "document" in your vector store.

What you'll learn

• Embedding an entire document produces a vector that averages all its topics together, making it match everything poorly and nothing well
• Chunks that are too small lose the context needed to answer the question; chunks that are too large dilute the signal with unrelated content
• The right chunk size depends on your embedding model's context window, your documents' structure, and the kinds of questions users ask

The problem

When you embed a full document -- say, a 20-page whitepaper -- the resulting vector is an average of every paragraph, every table, every footnote. A user searches for "latency numbers for the RPC layer" and the closest match is... the whole whitepaper. The model gets 20 pages of context, 19 of which are irrelevant. It wastes tokens, increases latency, and confuses the model with competing signals.

On the other end: if you split every sentence into its own chunk, a sentence like "It reduced latency by 40%" has no referent. Forty percent of what? The chunk has no context because you severed it from the paragraph that named the thing being measured.

This is the chunking problem. It is the single most important design decision in a RAG system, and it is not one you can tune after the fact without re-embedding your entire corpus.

Options & when to use each

Strategy	What it is good for	What it costs you	When to pick it
Fixed-size chunking	Predictable, fast, easy to reason about	Splits sentences mid-word; loses semantic boundaries	Documents with no natural structure (transcripts, logs) or when you need consistency above all else
Semantic chunking	Preserves sentence and paragraph boundaries; chunks read like coherent units	Requires a sentence splitter or an extra embedding pass; slower	Narrative text: articles, books, documentation, anything with paragraphs
Recursive chunking	Combines small retrieval units with larger context units; handles long documents well	More complex pipeline; stores parent-child metadata; extra storage	Long documents where you need both precision (small chunk) and context (large chunk)
Agentic chunking	An LLM decides where to split based on content	Expensive, slow, non-deterministic	Only when automatic strategies fail and you need human-quality splits on a small corpus

Build it

Before you reach for a splitting strategy, measure the problem. The following script loads a document, embeds it whole versus chunked, and compares retrieval quality on a known set of questions.

import psycopg2
from pgvector.psycopg2 import register_vector
import openai

# Assume embeddings table from Day 1
conn = psycopg2.connect("postgresql://user:pass@localhost/rag_demo")
register_vector(conn)
cur = conn.cursor()

def embed(text: str) -> list[float]:
    return openai.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

def compare_whole_vs_chunked(document: str, chunks: list[str], test_questions: list[str]):
    """Embed the whole document and each chunk, then compare retrieval quality."""

    # Whole-document approach
    whole_embedding = embed(document)
    cur.execute("INSERT INTO documents (content, embedding) VALUES (%s, %s) RETURNING id",
                (document, whole_embedding))
    whole_id = cur.fetchone()[0]

    # Chunked approach
    chunk_ids = []
    for chunk in chunks:
        chunk_embedding = embed(chunk)
        cur.execute("INSERT INTO documents (content, embedding) VALUES (%s, %s) RETURNING id",
                    (chunk, chunk_embedding))
        chunk_ids.append(cur.fetchone()[0])

    conn.commit()

    for question in test_questions:
        q_embedding = embed(question)

        # Whole-document retrieval
        cur.execute("""
            SELECT content, 1 - (embedding <=> %s::vector) AS similarity
            FROM documents WHERE id = %s
        """, (q_embedding, whole_id))
        whole_result = cur.fetchone()
        print(f"Q: {question}")
        print(f"  Whole doc similarity: {whole_result[1]:.4f}")
        print(f"  Whole doc content starts: {whole_result[0][:120]}...")

        # Chunked retrieval
        cur.execute("""
            SELECT content, 1 - (embedding <=> %s::vector) AS similarity
            FROM documents WHERE id = ANY(%s)
            ORDER BY embedding <=> %s::vector LIMIT 1
        """, (q_embedding, chunk_ids, q_embedding))
        chunk_result = cur.fetchone()
        print(f"  Best chunk similarity: {chunk_result[1]:.4f}")
        print(f"  Best chunk content: {chunk_result[0][:120]}...")
        print()

    # Cleanup
    cur.execute("DELETE FROM documents WHERE id = %s OR id = ANY(%s)",
                (whole_id, chunk_ids))
    conn.commit()

# Example: a paragraph about latency buried in a long document
doc = (
    "Introduction: This document describes the architecture of the new payment system. "
    "Chapter 1 covers the data model, Chapter 2 covers API design, "
    "Chapter 3 covers performance benchmarks. Our RPC layer handles 50,000 requests per second "
    "at p99 latency of 12ms, down from 85ms in the previous version. This was achieved by switching "
    "from JSON serialization to protobuf and adding connection pooling. "
    "Chapter 4 covers deployment and Chapter 5 covers monitoring."
)

chunks = [
    "Introduction: This document describes the architecture of the new payment system.",
    "Chapter 1 covers the data model, Chapter 2 covers API design.",
    "Chapter 3 covers performance benchmarks. Our RPC layer handles 50,000 requests per second "
    "at p99 latency of 12ms, down from 85ms in the previous version. This was achieved by switching "
    "from JSON serialization to protobuf and adding connection pooling.",
    "Chapter 4 covers deployment and Chapter 5 covers monitoring.",
]

compare_whole_vs_chunked(doc, chunks, [
    "What is the p99 latency of the RPC layer?",
    "How was the latency improvement achieved?",
])

Run it. The whole-document embedding returns moderate similarity for every question -- it matches everything a little. The chunked embedding returns high similarity for the one chunk that actually contains the answer and low similarity for the rest. The difference is the entire case for chunking.

What goes wrong

Mistake	How you notice it	The fix
Chunk size is your embedding model's max token limit	Chunks fit but retrieval quality is poor because each chunk covers too many topics	Aim for 256-512 tokens per chunk. Smaller than the model's max gives each chunk room to be about one thing
Chunks are all the same size regardless of content	Some chunks end mid-sentence, some are single words	Use a strategy that respects sentence boundaries, or at minimum use overlap so partial sentences reconnect
No overlap between chunks	A query that matches a concept split across two chunks gets neither	Add 10-20% overlap between adjacent chunks. The exact number depends on your content; start at 10%
All documents use the same chunk size	Legal contracts and Slack messages get the same treatment	Tune chunk size per document type. Short messages get small chunks; long articles get medium chunks

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# After running the comparison above, verify:
# 1. The chunk containing the latency number has significantly higher similarity
#    for the latency question than the whole document does.
# 2. The chunk similarity for the latency question is above 0.7 (cosine similarity
#    with text-embedding-3-small).
# 3. The whole-document similarity is roughly the same for all three questions,
#    confirming it averages all content.

cur.execute("""
    SELECT question, embedding <=> %s::vector AS distance
    FROM queries ORDER BY distance LIMIT 5
""", (embed("RPC latency p99 12ms"),))
for row in cur.fetchall():
    print(f"  {row[0]} -> distance: {row[1]:.4f}")
# The correct chunk should be the closest match by a visible margin.

Next: Fixed-Size Chunking