What Are Embeddings

You type "best laptop for programming" into a search box. A keyword-based system looks for documents containing "best," "laptop," and "programming." It misses "top developer notebooks" entirely because none of the words overlap. It returns kitchen reviews that happen to mention "best laptop stand for programming." You need a system that understands what you mean, not which words you used. Embeddings are how that system works.

What you'll learn

• An embedding is a fixed-length vector of floats that positions text in a semantic space where similar meanings land close together
• Cosine similarity measures the angle between two vectors -- 1.0 means identical direction, 0.0 means orthogonal, -1.0 means opposite
• Embeddings work because they're trained to make paraphrases close and unrelated sentences far apart; they capture meaning, not keywords
• Distance in embedding space corresponds to semantic distance; "cat" is closer to "kitten" than to "refrigerator"

The problem: search that matches words but misses meaning

Keyword search fails on three common cases. Synonyms: "car" and "automobile" share zero words. Paraphrases: "how do I fix a leaky faucet" and "repairing a dripping tap" have almost no overlap. Context: "apple" could be fruit or a company, and keyword search can't tell which you meant.

Embedding-based search solves all three. An embedding of "automobile" sits near the embedding of "car" in vector space. The distance between two embeddings tells you how semantically similar the texts are, regardless of which words they share.

How embeddings are made

Take a sentence like "The cat sat on the mat." An embedding model processes it through layers of a neural network and outputs a vector -- a list of numbers like [0.023, -0.451, 0.892, ...]. For text-embedding-3-small, that list has 1,536 numbers. The model was trained so that sentences with similar meaning produce vectors that point in similar directions.

# "cat" and "kitten" are close
cat_embedding      = [0.12, 0.45, -0.33, ...]  # 1,536 dimensions
kitten_embedding   = [0.14, 0.42, -0.31, ...]  # very similar

# "cat" and "refrigerator" are far apart
fridge_embedding   = [-0.67, -0.12, 0.81, ...]  # very different

The actual similarity is measured by comparing the angle between vectors, not the raw numbers.

Cosine similarity: the math that powers search

Cosine similarity measures how parallel two vectors are, treating them as directions in high-dimensional space. It ignores magnitude (how long the vector is) and focuses on direction (which way it points). This matters because two sentences about the same topic should point the same way, even if one is a paragraph and the other is a single word.

cosine_similarity(A, B) = (A · B) / (||A|| × ||B||)

Where:
- A · B is the dot product
- ||A|| is the magnitude (length) of vector A

The result ranges from -1 to 1:

Score	Meaning	Example
1.0	Identical direction	Two copies of the same sentence
~0.85	Very similar	"best laptop for coding" vs "top developer notebook"
~0.5	Loosely related	"Python programming" vs "snake handling"
~0.0	Orthogonal (unrelated)	"mortgage rates" vs "pizza recipe"
-1.0	Opposite direction	Rare in practice with modern embeddings

Why embeddings beat keyword search

Keyword search treats "fix a leaky faucet" and "repair a dripping tap" as completely different queries because they share zero words. Embedding search sees them as nearly identical because the model learned that "fix" ≈ "repair" and "leaky" ≈ "dripping" and "faucet" ≈ "tap."

The tradeoff: embeddings need an API call or a local model to generate, and they need a vector database to search efficiently. With keyword search you can grep a file. With embeddings you need infrastructure. Day 1 is about setting up that infrastructure so you're ready when keyword search stops being enough.

Options & when to use each

Option	What it's good for	What it costs you	When to pick it
Keyword search (no embeddings)	Exact matches, predictable queries, zero setup	Misses synonyms, paraphrases, context-dependent queries	Your query vocabulary matches your document vocabulary exactly
Embedding search (dense vectors)	Semantic search, fuzzy matching, multilingual queries	API cost, vector DB infrastructure, harder to debug	You need to find relevant documents when users don't know the exact words
Hybrid search (both)	Best of both: exact matches + semantic recall	Most complex to implement and tune	Production search where you can't afford to miss either exact hits or semantic matches

Build it: your first embedding

Here is a concrete example demonstrating the implementation details.

from openai import OpenAI
import numpy as np

client = OpenAI()  # assumes OPENAI_API_KEY in environment

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a = np.array(a)
    b = np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Generate embeddings and compare
cat_emb = get_embedding("cat")
kitten_emb = get_embedding("kitten")
fridge_emb = get_embedding("refrigerator")

print(f"cat vs kitten: {cosine_similarity(cat_emb, kitten_emb):.4f}")
print(f"cat vs refrigerator: {cosine_similarity(cat_emb, fridge_emb):.4f}")

Expected output (approximate):

cat vs kitten: 0.82
cat vs refrigerator: 0.31

The numbers will vary slightly between runs, but the pattern is consistent: semantically related terms score higher.

What goes wrong

Mistake	How you notice it	The fix
Using Euclidean distance instead of cosine	"cat" and "kitten" score farther apart than "cat" and "refrigerator"	Use cosine similarity for semantic search. Euclidean distance is sensitive to vector magnitude, which varies with text length
Expecting embeddings to understand facts	"Paris is the capital of France" and "The capital of France is Paris" score ~1.0, but "What is the capital of France?" and "Paris" score lower than expected	Embeddings capture semantic similarity, not question-answer alignment. That's what the LLM in RAG is for
Comparing embeddings from different models	Similarity scores are meaningless -- different models map text to different spaces	Always use the same embedding model for all vectors in a single application. Pick one at the start and stick with it
Treating similarity as confidence	0.75 cosine similarity does not mean "75% chance of being relevant"	Similarity scores are relative within a search result. Use them to rank, not to set absolute thresholds across different queries

Confirm it worked

Run the code above. You should see:

1. cat vs kitten scores above 0.70 (high similarity)
2. cat vs refrigerator scores below 0.50 (low similarity)
3. The same code works for any pair of texts -- try "I love Python programming" vs "Python is my favorite language" and compare to "I love Python programming" vs "I prefer tropical snakes"

If you get an error about OPENAI_API_KEY, export it first: export OPENAI_API_KEY=sk-.... If you don't have an OpenAI key, skip ahead to the next lesson, which covers Gemini and Cohere alternatives.

Next: Generating Embeddings