Caching

Every embedding API call costs money and takes time. A popular query like "what's the return policy?" might get asked 50 times a day by different users. Without caching, you re-embed the query and re-run the vector search 50 times. With caching, the first request does the work and the next 49 are instant and free. Caching is the single highest-ROI optimization in any RAG system, but it introduces staleness. This lesson covers what to cache, how to cache it, and when to invalidate.

What you'll learn

• Cache document embeddings: re-embedding unchanged documents is the most expensive mistake at scale
• Cache search results for common queries: 80% of queries are repeats or near-duplicates in most production systems
• Use Redis with TTLs for production caches; a Python dict works for development

The problem

Your RAG system handles 10,000 queries per day. Each query requires an embedding API call ($0.00002) and a vector search. The top 50 queries account for 60% of traffic — "what's the return policy," "how do I reset my password," "contact support." Without caching, you pay for 10,000 embeddings per day. With caching, you pay for the first occurrence of each unique query and serve the rest from cache. The savings are immediate and compounding: fewer API calls, lower latency, and reduced load on the vector database.

What to cache

What	TTL	Invalidation trigger	Savings
Document embeddings	Permanent (no TTL)	Document content change	Avoids re-embedding on every re-index
Query embeddings	1 hour	N/A (queries are ephemeral)	Avoids embedding API call for repeat queries
Search results	5-15 minutes	Document update affecting the result set	Avoids vector search for popular queries
LLM responses	1-5 minutes	Document update or new information	Avoids generation cost for identical queries

Build it

Embedding cache

The embedding API is often the most expensive component of ingestion. Implement a persistent cache to store and retrieve document embeddings without repeatedly calling the provider.

import hashlib
import redis
import json

cache = redis.Redis(decode_responses=True)

def cached_embed(text, model="text-embedding-3-small"):
    """Cache document embeddings. No TTL — embeddings don't expire."""
    key = f"embed:{hashlib.sha256(text.encode()).hexdigest()[:16]}"

    cached = cache.get(key)
    if cached:
        return json.loads(cached)

    embedding = client.embeddings.create(model=model, input=text).data[0].embedding
    cache.set(key, json.dumps(embedding))
    return embedding

Search result cache

For popular queries, avoid running the vector search entirely. A short-lived cache stores recent search results, drastically improving latency for common questions.

def cached_search(query, top_k=10, ttl=300):
    """Cache search results with a 5-minute TTL."""
    # Normalize the query for cache key stability
    normalized = query.lower().strip()
    key = f"search:{hashlib.md5(normalized.encode()).hexdigest()}"

    cached = cache.get(key)
    if cached:
        return json.loads(cached)

    embedding = cached_embed(normalized)
    results = vector_search_with_embedding(embedding, top_k)

    cache.setex(key, ttl, json.dumps(results))
    return results

Semantic deduplication for cache hits

Exact string matches miss variations like "return policy" vs "returns policy". Enhance the cache by checking if a semantically similar query was recently processed.

def cached_search_semantic(query, top_k=10, ttl=300):
    """Cache search results with semantic deduplication for near-duplicate queries."""
    # Check if a semantically similar query is already cached
    query_embedding = cached_embed(query)
    similar_keys = find_similar_cached_queries(query_embedding, threshold=0.95)

    if similar_keys:
        key = similar_keys[0]
        return json.loads(cache.get(key))

    # No similar query found — run the search and cache
    results = vector_search_with_embedding(query_embedding, top_k)
    key = f"search:{hashlib.md5(query.encode()).hexdigest()}"
    cache.setex(key, ttl, json.dumps(results))
    return results

What goes wrong

Mistake	How you notice it	The fix
Cache returns stale results	User sees old policy after update	Invalidate cache on document update. Use short TTLs for frequently changing content. Log cache hits vs misses
Cache key collision	Different queries produce the same hash, return wrong results	Use SHA-256 with a query prefix. Collisions are astronomically rare but use a longer hash if concerned
Cache memory blowup	Redis memory usage grows unbounded	Set maxmemory and an eviction policy (allkeys-lru). Monitor cache size. Set TTLs on all cached entries
Cache hit rate is low despite caching	Most queries are unique, cache provides little benefit	Semantic deduplication catches near-duplicates. If queries are truly unique, invest in query embedding caching instead

Confirm it worked

Verify your caching implementation by measuring the latency of repeated identical requests. The second call should return results at least an order of magnitude faster.

# First call: cache miss, runs the search
start = time.time()
results1 = cached_search("return policy")
first_latency = time.time() - start

# Second call: cache hit, returns instantly
start = time.time()
results2 = cached_search("return policy")
second_latency = time.time() - start

assert results1 == results2
assert second_latency < first_latency * 0.1  # At least 10x faster

Next: Monitoring & Observability