Fixed-Size Chunking

The simplest chunking strategy is also the most predictable: pick a size, split the text, and move on. No model calls, no sentence boundary detection, no ambiguity about where one chunk ends and the next begins. Fixed-size chunking is the baseline every other strategy is measured against, and for many document types it is all you will ever need.

What you'll learn

• Fixed-size chunking splits text by character or token count with no regard for content boundaries -- it is fast, deterministic, and easy to implement
• Character-based splitting is simpler but can cut words in half; token-based splitting aligns with your embedding model's actual limits
• Overlap between adjacent chunks recovers the context that gets severed at split boundaries, at the cost of storing and embedding the same text twice

The problem

You need to split a document into pieces your embedding model can handle. The simplest approach is a fixed window: take 500 characters, embed them, advance 500 characters, repeat. But text does not align with round numbers. A split at character 500 might land in the middle of a sentence -- or worse, in the middle of a word. The result is chunks that are technically the right size but semantically broken.

Overlap fixes most of this. By making each chunk share some text with its neighbors, you ensure that any sentence split across a boundary appears in full in at least one chunk. The tradeoff is that you store and embed overlapping text, which increases your vector store size.

Options & when to use each

Option	What it is good for	What it costs you	When to pick it
Character-based split	Dead simple, no dependencies, fast	Splits mid-word; chunks can vary wildly in token count	Quick prototypes, documents with no sentence structure (logs, CSV rows)
Token-based split	Aligns with your embedding model's limits; chunks are predictable in token count	Requires a tokenizer; slightly more complex	Production systems where you need to stay under model limits
Character split + overlap	Recovers context lost at boundaries; most sentences appear intact in at least one chunk	10-20% storage overhead from overlapping text	Narrative text where you want simplicity but can't afford broken sentences

Build it

Character-based splitting with overlap

Here is a concrete example demonstrating the implementation details.

def split_by_characters(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split text into fixed-size character chunks with overlap."""
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

# Example
text = (
    "Retrieval-augmented generation combines a retrieval step with text generation. "
    "The system first searches a knowledge base for relevant documents, then feeds "
    "those documents to a language model as additional context. This grounds the "
    "model's output in real data and reduces hallucination. However, the retrieval "
    "step is only as good as the chunking strategy that prepared the documents."
)

chunks = split_by_characters(text, chunk_size=120, overlap=20)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i} ({len(chunk)} chars): {chunk}")

Token-based splitting

Here is a concrete example demonstrating the implementation details.

import tiktoken

def split_by_tokens(text: str, chunk_size: int = 256, overlap: int = 25) -> list[str]:
    """Split text into fixed-size token chunks with overlap."""
    enc = tiktoken.get_encoding("cl100k_base")  # OpenAI embedding tokenizer
    tokens = enc.encode(text)

    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start += chunk_size - overlap

    return chunks

chunks = split_by_tokens(text, chunk_size=30, overlap=5)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i} ({len(tiktoken.get_encoding('cl100k_base').encode(chunk))} tokens): {chunk}")

Storing fixed-size chunks in pgvector

Here is a concrete example demonstrating the implementation details.

def ingest_document(conn, document_id: str, text: str,
                    chunk_size: int = 256, overlap: int = 25):
    """Chunk a document with fixed token sizes and store in pgvector."""
    chunks = split_by_tokens(text, chunk_size=chunk_size, overlap=overlap)

    cur = conn.cursor()
    for i, chunk in enumerate(chunks):
        embedding = openai.embeddings.create(
            model="text-embedding-3-small", input=chunk
        ).data[0].embedding

        cur.execute("""
            INSERT INTO chunks (document_id, chunk_index, content, embedding, token_count)
            VALUES (%s, %s, %s, %s, %s)
        """, (
            document_id, i, chunk, embedding,
            len(tiktoken.get_encoding("cl100k_base").encode(chunk))
        ))

    conn.commit()
    print(f"Ingested {len(chunks)} chunks from document {document_id}")

Retrieval with overlap deduplication

Overlap means the same text appears in multiple chunks. When you retrieve the top-k chunks, you might get back overlapping content that wastes context window space. Deduplicate before sending to the model:

def deduplicate_chunks(chunks: list[dict], min_unique_ratio: float = 0.5) -> list[dict]:
    """Remove chunks that overlap too much with higher-ranked chunks."""
    kept = []
    for chunk in sorted(chunks, key=lambda c: c["similarity"], reverse=True):
        # Check if this chunk's content is mostly new
        if not any(
            _overlap_ratio(chunk["content"], kept_chunk["content"]) > (1 - min_unique_ratio)
            for kept_chunk in kept
        ):
            kept.append(chunk)
    return kept

def _overlap_ratio(a: str, b: str) -> float:
    """Fraction of text in the shorter string that appears in the longer string."""
    shorter, longer = (a, b) if len(a) < len(b) else (b, a)
    if not shorter:
        return 0.0
    # Simple n-gram overlap -- production systems use Jaccard or MinHash
    short_words = set(shorter.lower().split())
    long_words = set(longer.lower().split())
    return len(short_words & long_words) / len(short_words)

What goes wrong

Mistake	How you notice it	The fix
Chunk size is too small	Retrieved chunks are single sentences or fragments; the model can't answer because it lacks context	Increase chunk size. Start at 256 tokens and tune up from there. Below 100 tokens, context loss dominates
No overlap with fixed-size chunks	A question that matches the boundary between two chunks gets neither	Add 10-20% overlap. For 256-token chunks, 25-50 tokens of overlap catches most broken sentences
Character count used as a proxy for tokens	OpenAI API rejects requests because chunks exceed the embedding model's token limit	Use token-based splitting with tiktoken. Character count can be 3-4x larger than token count for non-English text
Overlap is too large	Every retrieved chunk contains mostly the same content; the context window is wasted	Reduce overlap. 10% is usually enough. Above 30%, you are embedding the same text three times
Deduplication removes the only chunk with the answer	The model says it does not have enough information, but you know the document contains it	Tune min_unique_ratio. Start at 0.5 (allow 50% overlap) and raise it only if you see duplicate content in retrieved results

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Verify that overlapping chunks preserve context at boundaries
text = (
    "The quick brown fox jumps over the lazy dog. "
    "The dog, unimpressed, continues napping in the afternoon sun."
)

chunks = split_by_tokens(text, chunk_size=12, overlap=4)
for chunk in chunks:
    print(chunk)

# If working correctly:
# - "jumps over the lazy dog" appears in at least one chunk intact
# - "dog. The dog" (the boundary sentence) appears complete in at least one chunk
# - No chunk is a single orphaned word

# Verify token counts stay under limit
enc = tiktoken.get_encoding("cl100k_base")
for i, chunk in enumerate(chunks):
    token_count = len(enc.encode(chunk))
    assert token_count <= 12, f"Chunk {i} has {token_count} tokens, exceeds limit"
    print(f"Chunk {i}: {token_count} tokens (OK)")

# Verify no two adjacent chunks are identical (overlap=chunk_size bug)
for i in range(len(chunks) - 1):
    assert chunks[i] != chunks[i + 1], f"Chunks {i} and {i+1} are identical"

Next: Semantic Chunking