Generating Embeddings

You know what an embedding is. Now you need to produce them at scale. A single embedding API call is trivial -- a few lines of Python. The real work is choosing the right model for your budget and accuracy needs, batching requests so you don't burn a week and a hundred dollars embedding your document corpus, and handling the inevitable rate limits and failures that show up at production volume. This lesson covers the three major embedding providers, their cost and dimensional tradeoffs, and how to write an embedding pipeline that won't fall over at scale.

What you'll learn

• Three production-grade embedding APIs: OpenAI text-embedding-3-small (budget), Gemini text-embedding-004 (free tier), Cohere embed-v3 (multilingual)
• Batching matters: sending 10,000 documents one at a time takes minutes and wastes money; batched calls cut both time and cost by 10-50x
• Dimensionality is a tradeoff: higher dimensions capture more nuance but cost more to store and search; 768 is the sweet spot for most RAG applications
• Rate limits are real: every provider caps requests per minute. A production pipeline needs retry logic and exponential backoff

The problem: one model, one API, but many decisions

Every major provider has an embeddings API. They all work the same way: send text, get back a vector. The differences -- cost per token, maximum batch size, vector dimensionality, rate limits -- determine which one is right for your project. Pick the wrong model and you either overpay by 10x or get vectors that don't capture enough meaning for your use case. This lesson helps you pick and then build the pipeline.

The three providers compared

Provider	Model	Dimensions	Cost per 1M tokens	Max batch	Notes
OpenAI	text-embedding-3-small	512 or 1536	$0.02	2048 inputs	Can reduce to 512d at no quality loss for most tasks
OpenAI	text-embedding-3-large	256, 1024, or 3072	$0.13	2048 inputs	Overkill for most RAG; only worth it for nuanced semantic tasks
Google Gemini	text-embedding-004	768	Free (rate-limited)	2048 inputs	1,500 requests/day free; good for prototyping
Cohere	embed-english-v3.0	1024	$0.10	96 inputs	Also has embed-multilingual-v3.0 for non-English
Cohere	embed-multilingual-v3.0	1024	$0.10	96 inputs	Best choice for mixed-language corpora

Options & when to use each

Option	What it's good for	What it costs you	When to pick it
OpenAI text-embedding-3-small	Balanced cost/quality, huge batch sizes, production RAG	$0.02/M tokens adds up at millions of documents	Default choice for English-language RAG. Low cost, high quality
OpenAI text-embedding-3-large	Maximum semantic accuracy for legal, medical, or nuanced text	6.5x the cost of small model; 50% more storage for vectors	Specialized domains where recall is critical and cost is secondary
Gemini text-embedding-004	Free tier for prototyping, 768d is a good balance	Rate limits are tight at 1,500/day; not suitable for production without upgrade	Prototyping, side projects, building before committing to a paid plan
Cohere embed-v3	Multilingual RAG, good documentation on chunking strategies	Smaller batch limit (96) means more API calls; $0.10/M tokens	Mixed-language corpora or when you prefer Cohere's ecosystem

Build it: a batch embedding pipeline with retries

Here is a concrete example demonstrating the implementation details.

import time
from openai import OpenAI, RateLimitError, APIError
from typing import Iterator

client = OpenAI()

def embed_batch(
    texts: list[str],
    model: str = "text-embedding-3-small",
    dimensions: int | None = None,
    max_retries: int = 5
) -> list[list[float]]:
    """
    Embed a list of texts with automatic batching, retries,
    and exponential backoff. Returns embeddings in input order.
    """
    kwargs = {"model": model, "input": texts}
    if dimensions:
        kwargs["dimensions"] = dimensions

    for attempt in range(max_retries):
        try:
            response = client.embeddings.create(**kwargs)
            # Sort by index to preserve input order
            sorted_data = sorted(response.data, key=lambda d: d.index)
            return [d.embedding for d in sorted_data]
        except RateLimitError:
            wait = 2 ** attempt
            print(f"Rate limited, retrying in {wait}s (attempt {attempt + 1}/{max_retries})")
            time.sleep(wait)
        except APIError as e:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt
            print(f"API error: {e}, retrying in {wait}s")
            time.sleep(wait)

    raise RuntimeError("Max retries exceeded")

def embed_corpus(
    texts: list[str],
    batch_size: int = 100,
    model: str = "text-embedding-3-small",
    dimensions: int = 768
) -> Iterator[tuple[int, list[float]]]:
    """
    Generator that yields (index, embedding) pairs for a large corpus,
    processing in batches to avoid memory issues and respect rate limits.
    """
    for i in range(0, len(texts), batch_size):
        batch = texts[i : i + batch_size]
        embeddings = embed_batch(batch, model=model, dimensions=dimensions)
        time.sleep(0.1)  # Be gentle with rate limits
        for j, emb in enumerate(embeddings):
            yield i + j, emb

Usage example:

Here is a concrete example demonstrating the implementation details.

# Small test: compare providers on the same text
texts = [
    "How do I reset my password?",
    "Password reset instructions",
    "Today's weather forecast for Chicago",
]

openai_embs = embed_batch(texts, model="text-embedding-3-small")

# Large corpus: process 5,000 documents in batches
import json

corpus = [f"Document {i}: some content here" for i in range(5000)]
for idx, emb in embed_corpus(corpus, batch_size=100):
    if idx % 1000 == 0:
        print(f"Embedded {idx} documents...")

Using Google Gemini embeddings

Here is a concrete example demonstrating the implementation details.

import google.generativeai as genai

genai.configure(api_key="YOUR_GEMINI_API_KEY")

def gemini_embed(texts: list[str]) -> list[list[float]]:
    """Embed a batch of texts using Gemini."""
    result = genai.embed_content(
        model="models/text-embedding-004",
        content=texts,
        task_type="retrieval_document"  # or "retrieval_query"
    )
    return result["embedding"]

Using Cohere embeddings

Here is a concrete example demonstrating the implementation details.

import cohere

co = cohere.Client("YOUR_COHERE_API_KEY")

def cohere_embed(texts: list[str]) -> list[list[float]]:
    """Embed a batch of texts using Cohere."""
    response = co.embed(
        texts=texts,
        model="embed-english-v3.0",
        input_type="search_document"  # or "search_query"
    )
    return response.embeddings

Dimensionality: why 768 is usually enough

OpenAI's text-embedding-3-small supports a dimensions parameter that truncates the vector. You can request 512 dimensions instead of 1536 and still get excellent results for most tasks. The model learned to pack the most important information into the first dimensions during training.

# Same text, different dimensions
emb_1536 = embed_batch(["RAG with pgvector"], dimensions=1536)
emb_768  = embed_batch(["RAG with pgvector"], dimensions=768)
emb_512  = embed_batch(["RAG with pgvector"], dimensions=512)

print(f"1536d vector size: {len(emb_1536[0])}")
print(f"768d vector size:  {len(emb_768[0])}")
print(f"512d vector size:  {len(emb_512[0])}")

The 768-dimensional vector takes half the storage of 1536d and searches twice as fast. The recall difference is typically under 1% for English text. For most RAG applications, 768 is the sweet spot.

What goes wrong

Mistake	How you notice it	The fix
Embedding one document per API call	10,000 API calls take 5+ minutes and waste money on per-request overhead	Always batch. OpenAI supports up to 2,048 texts per call. Even batches of 50-100 cut time dramatically
Not handling rate limits	Pipeline crashes after a few hundred embeddings with a 429 error	Add retry logic with exponential backoff. OpenAI's free tier is 3 RPM; paid tiers start at 5,000 RPM
Embedding text that's too long	API returns an error or silently truncates input past the token limit	Check token counts before embedding. text-embedding-3-small has an 8,191 token limit. Split long texts first
Using different models for documents and queries	Search results are meaningless because vectors live in different spaces	Use the same model for everything. If you embed documents with OpenAI, embed queries with OpenAI
Forgetting input_type with Cohere	Cohere returns different embeddings depending on whether you set input_type to "search_document" or "search_query"	Set input_type="search_document" when embedding your corpus and input_type="search_query" when embedding user queries

Confirm it worked

Run the batch embedding code with a small test set:

test_texts = ["cat", "dog", "car", "bicycle"]
embs = embed_batch(test_texts, dimensions=768)

# Verify dimensions
assert all(len(e) == 768 for e in embs), "Wrong embedding dimension"

# Verify semantic relationships hold
import numpy as np

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

cat_dog = cosine(embs[0], embs[1])    # cat vs dog (animals)
cat_car = cosine(embs[0], embs[2])    # cat vs car (unrelated)
car_bike = cosine(embs[2], embs[3])   # car vs bicycle (vehicles)

print(f"cat-dog:   {cat_dog:.4f}")   # Should be relatively high
print(f"cat-car:   {cat_car:.4f}")   # Should be lower
print(f"car-bike:  {car_bike:.4f}")  # Should be relatively high

assert cat_dog > cat_car, "Animal similarity should beat animal-vehicle"
assert car_bike > cat_car, "Vehicle similarity should beat animal-vehicle"

If you see cat-car scoring higher than cat-dog, you're either using a model that wasn't trained well on these terms or there's a bug in your similarity calculation.

Next: pgvector Setup