Semantic Chunking

Fixed-size chunking is fast and predictable, but it does not read your documents. It will slice through the middle of a thought as cheerfully as it slices through whitespace. Semantic chunking respects what the text actually says: it splits on sentence and paragraph boundaries, and when you raise the sophistication bar, it uses embeddings to find where a document's topic shifts and cuts there.

What you'll learn

• Semantic chunking splits text at natural linguistic boundaries: sentences, paragraphs, and topic shifts
• Sentence-boundary splitting is the simplest semantic strategy; it requires a sentence tokenizer but produces readable, self-contained chunks
• Embedding-based semantic chunking compares adjacent sentences and cuts where the cosine similarity drops, detecting topic changes without an LLM

The problem

Fixed-size chunking does not know what a sentence is. It splits text at character N, even if N lands in the middle of a word. The result is chunks that read like "p99 latency of 12ms, down f" -- broken mid-thought, useless for retrieval, and confusing for the model that receives them.

Semantic chunking solves this by respecting the document's own structure. At minimum, it splits on sentence boundaries so every chunk contains complete thoughts. At the higher end, it uses the same embedding model you use for retrieval to detect topic shifts -- cutting where the semantic content changes, not where the character counter says to.

Options & when to use each

Strategy	What it is good for	What it costs you	When to pick it
Sentence-boundary splitting	Every chunk is a readable, complete thought; zero broken sentences	Requires a sentence tokenizer; chunks can vary in length	Narrative text with clear sentence structure: articles, documentation, books
Paragraph-boundary splitting	Chunks map to the author's intended grouping of ideas	Paragraphs can be very long or very short; uneven chunk sizes	Well-structured documents with consistent paragraph lengths
Embedding-based splitting	Detects topic changes the author did not mark with whitespace; cuts between ideas, not between sentences	Extra embedding calls per document; slower ingestion pipeline	Long, unstructured documents where you want the model's semantic understanding to drive splitting
Hybrid: sentence + size cap	Readable chunks that also respect a maximum token limit	Adds complexity; still breaks sentences when the cap is hit	Production systems that need semantic quality with hard model limits

Build it

Sentence-boundary splitting with spaCy

Here is a concrete example demonstrating the implementation details.

import spacy

nlp = spacy.load("en_core_web_sm")

def split_by_sentences(text: str, max_sentences: int = 5) -> list[str]:
    """Split text into chunks of up to N sentences each."""
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]

    chunks = []
    for i in range(0, len(sentences), max_sentences):
        chunk = " ".join(sentences[i : i + max_sentences])
        chunks.append(chunk)

    return chunks

text = (
    "RAG systems need chunking. The simplest approach splits by character count. "
    "But that breaks sentences. Semantic chunking avoids this. "
    "It uses sentence boundaries instead. The result is readable chunks. "
    "Models understand them better. Retrieval quality improves."
)

chunks = split_by_sentences(text, max_sentences=3)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i}: {chunk}")

Token-aware sentence splitting

Here is a concrete example demonstrating the implementation details.

import tiktoken

def split_sentences_with_token_cap(
    text: str, max_tokens: int = 256, overlap_sentences: int = 1
) -> list[str]:
    """Split at sentence boundaries, respecting a token cap per chunk."""
    enc = tiktoken.get_encoding("cl100k_base")
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]

    chunks = []
    current_chunk: list[str] = []
    current_tokens = 0

    for sentence in sentences:
        sentence_tokens = len(enc.encode(sentence))

        if current_tokens + sentence_tokens > max_tokens and current_chunk:
            # Emit the current chunk
            chunks.append(" ".join(current_chunk))
            # Start new chunk with overlap: keep the last N sentences
            current_chunk = current_chunk[-overlap_sentences:] if overlap_sentences else []
            current_tokens = sum(len(enc.encode(s)) for s in current_chunk)

        current_chunk.append(sentence)
        current_tokens += sentence_tokens

    if current_chunk:
        chunks.append(" ".join(current_chunk))

    return chunks

Embedding-based semantic chunking

This is the most sophisticated automatic strategy: embed every sentence, compute cosine similarity between adjacent sentences, and split where the similarity drops below a threshold. The intuition: when an author changes topic, the embedding vectors of consecutive sentences will be further apart.

import numpy as np

def split_by_semantic_boundaries(
    text: str, threshold: float = 0.5, min_chunk_sentences: int = 2
) -> list[str]:
    """Split text where the semantic similarity between adjacent sentences drops."""
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]

    if len(sentences) <= min_chunk_sentences:
        return [" ".join(sentences)]

    # Embed every sentence
    embeddings = []
    for sent in sentences:
        emb = openai.embeddings.create(
            model="text-embedding-3-small", input=sent
        ).data[0].embedding
        embeddings.append(np.array(emb))

    # Find split points where similarity drops
    split_indices = [0]
    for i in range(1, len(sentences)):
        similarity = np.dot(embeddings[i - 1], embeddings[i]) / (
            np.linalg.norm(embeddings[i - 1]) * np.linalg.norm(embeddings[i])
        )
        if similarity < threshold:
            split_indices.append(i)

    split_indices.append(len(sentences))

    # Build chunks, enforcing minimum size
    chunks = []
    i = 0
    while i < len(split_indices) - 1:
        start = split_indices[i]
        end = split_indices[i + 1]
        # If the chunk is too small, merge it with the next one
        if end - start < min_chunk_sentences and i + 2 < len(split_indices):
            end = split_indices[i + 2]
            i += 1
        chunks.append(" ".join(sentences[start:end]))
        i += 1

    return chunks

Putting it together in pgvector

Here is a concrete example demonstrating the implementation details.

def ingest_semantic(conn, document_id: str, text: str, strategy: str = "sentence"):
    """Ingest a document using semantic chunking and store in pgvector."""

    if strategy == "sentence":
        chunks = split_sentences_with_token_cap(text, max_tokens=256)
    elif strategy == "embedding":
        chunks = split_by_semantic_boundaries(text, threshold=0.5)
    else:
        raise ValueError(f"Unknown strategy: {strategy}")

    cur = conn.cursor()
    for i, chunk in enumerate(chunks):
        embedding = openai.embeddings.create(
            model="text-embedding-3-small", input=chunk
        ).data[0].embedding

        cur.execute("""
            INSERT INTO chunks (document_id, chunk_index, content, embedding,
                                chunking_strategy, token_count)
            VALUES (%s, %s, %s, %s, %s, %s)
        """, (
            document_id, i, chunk, embedding, strategy,
            len(tiktoken.get_encoding("cl100k_base").encode(chunk))
        ))

    conn.commit()
    print(f"Ingested {len(chunks)} chunks using {strategy} strategy")

What goes wrong

Mistake	How you notice it	The fix
Threshold is too high for embedding-based splitting	Every sentence becomes its own chunk; too many tiny embeddings	Lower the threshold. Start at 0.3 and raise it until chunks average 3-5 sentences
Threshold is too low	The entire document becomes one chunk; no splitting happens	Raise the threshold. If similarity between adjacent sentences is always above 0.8, your threshold is too low
Sentence tokenizer fails on code or lists	Chunks contain fragments of bullet points or code blocks that read like gibberish	Pre-process: extract code blocks and lists, chunk them separately with fixed-size, and treat narrative text with semantic chunking
spaCy sentence segmentation is slow on long documents	Ingestion takes minutes per document; pipeline is bottlenecked	Batch sentences for embedding: embed all sentences in one API call, then split locally. Or use a faster sentence segmenter like nltk.sent_tokenize
Embedding-based chunking is too expensive	Your OpenAI bill spikes because you are embedding every sentence twice -- once for splitting and once for storage	Cache sentence embeddings during splitting and reuse them for chunk embeddings. Combine adjacent sentence embeddings to approximate chunk embeddings

Confirm it worked

Here is a concrete example demonstrating the implementation details.

# Verify semantic chunking produces better retrieval than fixed-size
test_doc = (
    "Chapter 1: Introduction to databases. A database is an organized collection of data. "
    "Relational databases use tables with rows and columns. SQL is the standard query language. "
    "Chapter 2: Vector databases. Vector databases store embeddings. "
    "They use approximate nearest neighbor search. pgvector adds vector support to PostgreSQL. "
    "Chapter 3: Hybrid search combines vector and keyword retrieval."
)

# Semantic chunks should separate the three chapters
semantic_chunks = split_by_semantic_boundaries(test_doc, threshold=0.4)
print(f"Semantic chunks: {len(semantic_chunks)}")
for i, chunk in enumerate(semantic_chunks):
    print(f"  Chunk {i}: {chunk[:80]}...")

# Each chunk should be about one topic (database types, vector DBs, hybrid search)
# and should contain at least 2 sentences.
for chunk in semantic_chunks:
    sentences_in_chunk = len(list(nlp(chunk).sents))
    assert sentences_in_chunk >= 2, f"Chunk has only {sentences_in_chunk} sentence(s)"

# Verify retrieval: a query about vector databases should match chunk 1 most closely
embedding = openai.embeddings.create(
    model="text-embedding-3-small", input="What is a vector database?"
).data[0].embedding

similarities = []
for i, chunk in enumerate(semantic_chunks):
    chunk_emb = openai.embeddings.create(
        model="text-embedding-3-small", input=chunk
    ).data[0].embedding
    sim = np.dot(embedding, chunk_emb) / (
        np.linalg.norm(embedding) * np.linalg.norm(chunk_emb)
    )
    similarities.append((i, sim))
    print(f"  Chunk {i} similarity: {sim:.3f}")

best = max(similarities, key=lambda x: x[1])
print(f"Best match: chunk {best[0]} (similarity {best[1]:.3f})")
assert best[0] == 1, "Vector database query should match the vector DB chunk"

Next: Recursive Chunking