Cost Optimization

RAG has three cost centers: embedding (per-token API calls), storage (per-GB for vector data), and generation (per-token LLM responses). Each can be optimized independently, and the biggest wins are usually in places you are not looking. This lesson covers where the money actually goes and how to spend less of it without sacrificing retrieval quality.

What you'll learn

• Embedding cost dominates at ingest time; generation cost dominates at query time
• Smaller embedding dimensions (256 vs 1536) cut storage costs by 83% with minimal quality loss for most use cases
• Caching is the single highest-ROI optimization: cache embeddings, search results, and LLM responses

Where the money goes

For a typical RAG system handling 10,000 queries per day with a 5,000-document knowledge base, the monthly costs break down roughly as follows:

Cost center	Monthly cost (approximate)	Optimization potential
LLM generation (GPT-4o, 500 tokens/response)	$150-300	30-50% reduction via caching and smaller models
Embedding API (text-embedding-3-small)	$5-15	80% reduction via caching
Vector storage (pgvector, 1536-dim)	$10-30	80% reduction via smaller dimensions
Redis caching	$5-10	Already optimized

The LLM generation cost dominates at query time. If you cut your embedding costs by 90% but your generation costs stay the same, you have saved $10/month. If you cache common LLM responses, you save $100/month. Optimize the expensive thing first.

Build it

Smaller embedding dimensions

Reducing embedding dimensions is the easiest way to cut vector storage costs. Most models support returning truncated embeddings that preserve search quality while drastically reducing the footprint.

# text-embedding-3-small supports dimensions from 256 to 1536
# 256 dimensions: 83% less storage, 83% faster search, minimal quality loss

def embed_efficient(text, dimensions=256):
    return client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
        dimensions=dimensions,
    ).data[0].embedding

# Storage comparison:
# 1536 dims * 4 bytes * 5000 docs = ~30 MB
# 256 dims * 4 bytes * 5000 docs = ~5 MB
# Plus HNSW index overhead: ~2x data size

Model selection by query complexity

Not every question requires the most expensive model. Introduce a lightweight classifier to route straightforward queries to faster, cheaper models, saving the heavy lifters for complex reasoning.

def select_model(question):
    """Route simple questions to cheaper models."""
    prompt = f"""Classify this question's complexity:
    SIMPLE: factoid lookup, definition, yes/no (use haiku)
    MODERATE: comparison, explanation, how-to (use sonnet)
    COMPLEX: analysis, synthesis, multi-step reasoning (use opus or gpt-4o)

    Question: {question}
    Complexity:"""

    complexity = call_llm(prompt, max_tokens=10, model="claude-haiku").strip()

    if "SIMPLE" in complexity:
        return "claude-haiku"
    elif "MODERATE" in complexity:
        return "claude-sonnet-4-6"
    else:
        return "claude-sonnet-4-6"  # or gpt-4o

Token budget optimization

Balancing chunk size affects both storage and inference costs. Calculate the optimal token length to ensure you are neither over-indexing nor bloating the generation prompt context.

def optimize_chunk_size_for_cost(avg_query_length, avg_response_length, model_cost_per_1k):
    """Calculate the optimal chunk size that minimizes total cost."""
    # Total cost = embedding_cost + storage_cost + generation_cost
    # Larger chunks = fewer embeddings = lower embedding and storage cost
    # But larger chunks = more tokens in LLM prompt = higher generation cost

    # Sweet spot for most use cases: 256-512 tokens per chunk
    # Below 256: too many chunks, embedding cost dominates
    # Above 512: too much context per query, generation cost dominates

    return 384  # Good default for most use cases

Cost tracking

Visibility is the prerequisite to optimization. Implement a centralized cost tracker to log token usage and calculate real-time expenditure across operations.

def track_cost(operation, model, tokens_in, tokens_out):
    PRICING = {
        "text-embedding-3-small": 0.00002,  # per 1K tokens
        "claude-haiku": (0.00025, 0.00125),  # (input, output) per 1K tokens
        "claude-sonnet-4-6": (0.003, 0.015),
        "gpt-4o": (0.0025, 0.01),
    }

    if operation == "embedding":
        cost = tokens_in * PRICING[model] / 1000
    else:
        input_cost, output_cost = PRICING[model]
        cost = (tokens_in * input_cost + tokens_out * output_cost) / 1000

    logger.info("rag_cost", operation=operation, model=model, cost_usd=round(cost, 6))
    return cost

What goes wrong

Mistake	How you notice it	The fix
Optimizing the wrong thing	Cut embedding costs 90% but generation costs stayed the same	Measure before optimizing. Track cost per query and cost per component. Optimize the biggest line item
Using the largest embedding model by default	$50/month storage when $5 would work	Start with text-embedding-3-small at 256 dimensions. Upgrade only when retrieval quality is measurably insufficient
Embedding the entire corpus on every deploy	CI/CD costs more than production traffic	Use incremental indexing and embedding cache. Only re-embed changed documents
No cost visibility	Surprised by the monthly bill	Track costs per query, per component, and per day. Set budget alerts. Review weekly

Confirm it worked

Validate the cost optimization by generating and comparing both large and truncated embeddings. This test ensures the storage reduction does not unacceptably degrade retrieval performance.

# Compare 1536-dim vs 256-dim embeddings
emb_1536 = embed_efficient("test text", dimensions=1536)
emb_256 = embed_efficient("test text", dimensions=256)

# Storage savings
assert len(emb_256) * 6 == len(emb_1536)

# Verify retrieval quality is still acceptable
results_1536 = vector_search_with_embedding(emb_1536, "test text")
results_256 = vector_search_with_embedding(emb_256, "test text")
# Top results should be similar, not identical — but the cost savings justify minor differences

Next: Capstone: Company Docs Q&A