Evaluating Prompt Quality

You can't optimize what you can't measure. Evaluating prompt quality means defining what "good" looks like, building a test set that represents real usage, and scoring outputs against a rubric. This lesson covers the three pillars of evaluation: metrics, test sets, and LLM-as-judge -- when to use each, and how to avoid the traps that make your evaluation numbers lie.

What you'll learn

• Define success metrics before you write the prompt: accuracy for classification, relevance for retrieval, format compliance for structured output, and task-specific metrics (ROUGE, BLEU, exact match) for generation
• A test set is a fixed collection of inputs and expected outputs. It must be representative of real usage, large enough to detect differences (30+ examples minimum), and never used during prompt development
• Scoring rubrics turn subjective quality into a number. A 3-point rubric (correct, partially correct, incorrect) is more reliable than a 10-point scale because human judgment is noisy at fine granularity
• LLM-as-judge uses a strong model (GPT-4, Claude) to score outputs automatically. It's cheaper than human evaluation but requires careful calibration -- the judge model has its own biases

The problem

You have two prompts that extract dates from legal documents. Prompt A feels more precise. Prompt B feels more robust. Which one is better? Without a metric and a test set, you're guessing. You might ship Prompt A because it "sounds better," only to discover it fails on the date format your users actually submit.

Evaluation answers the question definitively. You define a metric, run both prompts against the same test set, and compare the numbers. Prompt A scores 82% and Prompt B scores 91%. Ship B. No argument, no gut feeling, no "let me ask the team what they think."

Options & when to use each

Metric type	What it measures	When to use	Example
Exact match	Output == expected (after normalization)	Classification, entity extraction, any task with a single correct answer	"positive" == "positive"
Contains match	Expected string appears in output	Extraction tasks where the answer is embedded in longer text	"March 15, 2024" in "The contract was signed on March 15, 2024"
Format compliance	Output follows a specified schema or structure	Structured output, JSON, code generation	json.loads(output) succeeds without error
LLM-as-judge	A strong model scores the output against a rubric	Open-ended generation: summaries, translations, creative writing	"Rate the summary on a scale of 1-5 for completeness"
Human evaluation	A person scores the output	When LLM-as-judge isn't reliable: tone, safety, subjective quality	"Does this response sound empathetic?"

Build it

Step 1: Build a test set

A test set is the foundation of evaluation. Build it once, freeze it, and never use it for prompt development:

import json
from pathlib import Path

class TestSet:
    def __init__(self, name: str, metric: str):
        self.name = name
        self.metric = metric
        self.cases = []

    def add(self, input_text: str, expected: str, metadata: dict = None):
        self.cases.append({
            "id": f"{self.name}-{len(self.cases):04d}",
            "input": input_text,
            "expected": expected,
            "metadata": metadata or {}
        })

    def save(self, path: str):
        with open(path, "w") as f:
            json.dump({
                "name": self.name,
                "metric": self.metric,
                "cases": self.cases
            }, f, indent=2)

    @classmethod
    def load(cls, path: str) -> "TestSet":
        with open(path) as f:
            data = json.load(f)
        ts = cls(data["name"], data["metric"])
        ts.cases = data["cases"]
        return ts

    def __len__(self):
        return len(self.cases)


# Build a sentiment classification test set
test_set = TestSet(name="sentiment-v1", metric="exact_match")

# Positive examples
test_set.add("I absolutely love this product, it changed my life!", "positive")
test_set.add("Great service, would recommend to anyone.", "positive")
test_set.add("Five stars, no complaints whatsoever.", "positive")

# Negative examples
test_set.add("Complete waste of money, broke after one use.", "negative")
test_set.add("Terrible customer service, never buying again.", "negative")
test_set.add("Arrived damaged and they refused to refund.", "negative")

# Neutral / edge cases
test_set.add("It arrived on time.", "neutral")
test_set.add("The product is blue.", "neutral")
test_set.add("I haven't used it yet.", "neutral")

# Tricky cases -- mixed sentiment
test_set.add("The product works great but the packaging was terrible.", "mixed")
test_set.add("Customer service was helpful but the product is mediocre.", "mixed")

test_set.save("test_sets/sentiment-v1.json")
print(f"Test set: {len(test_set)} cases")

Step 2: Define scoring functions

Each metric type needs its own scoring function. Build them once, reuse them everywhere:

import re
import json as json_lib

def score_exact_match(output: str, expected: str) -> float:
    """Normalize and compare. 1.0 for exact match, 0.0 otherwise."""
    def normalize(s: str) -> str:
        return s.strip().lower().rstrip(".")
    return 1.0 if normalize(output) == normalize(expected) else 0.0

def score_contains(output: str, expected: str) -> float:
    """Check if the expected string appears anywhere in the output."""
    return 1.0 if expected.lower() in output.lower() else 0.0

def score_format_compliance(output: str, schema_type: str = "json") -> float:
    """Check if the output is valid JSON (or other format)."""
    if schema_type == "json":
        try:
            # Find JSON in the output (it might be wrapped in markdown)
            json_match = re.search(r'```(?:json)?\s*(\{.*?\})\s*```', output, re.DOTALL)
            if json_match:
                json_lib.loads(json_match.group(1))
                return 1.0
            json_lib.loads(output)
            return 1.0
        except (json_lib.JSONDecodeError, ValueError):
            return 0.0
    return 0.0

def score_llm_judge(output: str, expected: str, rubric: str, model: str = "gpt-4o") -> float:
    """Use a strong model to score the output against a rubric."""
    from openai import OpenAI
    client = OpenAI()

    judge_prompt = f"""You are evaluating the quality of an AI-generated response.

Rubric:
{rubric}

Reference (expected) output:
{expected}

Generated output to evaluate:
{output}

Score the generated output on a scale of 0-100 based on the rubric. Return ONLY the number."""

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0
    )
    score_text = response.choices[0].message.content.strip()
    try:
        return float(score_text) / 100.0
    except ValueError:
        return 0.0

Step 3: Run evaluation against a test set

This step executes the chosen prompt against the test set and calculates accuracy metrics. The function tracks correct answers, identifies failures for debugging, and computes the 95% confidence interval.

def evaluate(
    prompt_fn,  # Function that takes input text and returns output
    test_set: TestSet,
    metric_fn,  # Function that takes (output, expected) and returns float
    model: str = "gpt-4o-mini"
) -> dict:
    """Run a prompt against a test set and return detailed results."""
    results = []
    scores = []

    for case in test_set.cases:
        output = prompt_fn(case["input"])
        score = metric_fn(output, case["expected"])
        results.append({
            "id": case["id"],
            "input": case["input"],
            "expected": case["expected"],
            "output": output,
            "score": score
        })
        scores.append(score)

    accuracy = sum(scores) / len(scores) if scores else 0.0

    # Find failure cases
    failures = [r for r in results if r["score"] < 1.0]

    # Compute confidence interval (simple binomial proportion)
    n = len(scores)
    se = (accuracy * (1 - accuracy) / n) ** 0.5 if n > 0 else 0
    ci_lower = max(0, accuracy - 1.96 * se)
    ci_upper = min(1, accuracy + 1.96 * se)

    return {
        "test_set": test_set.name,
        "n_cases": n,
        "accuracy": accuracy,
        "ci_95": (ci_lower, ci_upper),
        "failures": failures,
        "results": results
    }

Step 4: Build a scoring rubric

A rubric turns "this looks good" into a number. The key is making it specific enough that two evaluators would give the same score:

RUBRIC_SUMMARIZATION = """
Score the summary on the following criteria, each worth 25 points (total 100):

1. COMPLETENESS (0-25): Does the summary capture all key points from the source?
   - 25: All key points present, nothing important missing
   - 15: Most key points present, 1-2 minor omissions
   - 5: Several key points missing
   - 0: Summary misses the main point entirely

2. CONCISENESS (0-25): Is the summary free of unnecessary detail?
   - 25: Every sentence adds value, no repetition
   - 15: Mostly concise, 1-2 redundant sentences
   - 5: Contains significant filler or repetition
   - 0: Longer than the original or mostly filler

3. ACCURACY (0-25): Does the summary contain any factual errors?
   - 25: No factual errors
   - 15: One minor error that doesn't change the meaning
   - 5: One significant error or multiple minor errors
   - 0: Fabricated information not in the source

4. COHERENCE (0-25): Does the summary read as a well-structured paragraph?
   - 25: Flows naturally, logical order
   - 15: Mostly coherent, one awkward transition
   - 5: Jumpy, hard to follow
   - 0: Incoherent, random sentences
"""

# Use the rubric with LLM-as-judge
score = score_llm_judge(
    output="The meeting covered Q3 budget planning. The team agreed to increase marketing spend by 15%.",
    expected="Key points: Q3 budget review, marketing spend increase of 15%, team consensus reached.",
    rubric=RUBRIC_SUMMARIZATION
)

What goes wrong

Mistake	How you notice it	The fix
Test set is too small	Accuracy jumps from 70% to 90% between runs with no prompt changes	Minimum 30 examples. For production, 100+. A test set of 5 examples has a confidence interval of +/- 40% -- useless
Test set leaks into development	Accuracy is 95% on the test set but users complain	Never look at the test set while developing prompts. Split into train/dev/test. Only run against test when you're done iterating
LLM-as-judge has positional bias	The same output scores higher when it appears first in the comparison	Randomize the order of outputs when using LLM-as-judge for pairwise comparison. Or use a pointwise rubric (score each output independently)
Exact match is too strict	A correct answer gets scored 0 because of capitalization or trailing whitespace	Always normalize before comparison: lowercase, strip whitespace, remove trailing punctuation
Rubric is too vague	LLM-as-judge gives every output 85-95, no discrimination	Make rubrics behavioral: "The summary includes the budget number" not "The summary is complete." Test the rubric on known-good and known-bad outputs before using it

Confirm it worked

Execute this script to verify the evaluation utilities function as designed. These tests ensure exact match scoring handles case insensitivity and trailing punctuation while checking that the test set loads properly.

# Test 1: Exact match scoring
assert score_exact_match("positive", "positive") == 1.0
assert score_exact_match("positive", "Positive") == 1.0  # Case insensitive
assert score_exact_match("positive.", "positive") == 1.0  # Trailing punctuation
assert score_exact_match("positive", "negative") == 0.0

# Test 2: Test set loads and saves
ts = TestSet("test", "exact_match")
ts.add("hello", "world")
ts.save("/tmp/test_set.json")
loaded = TestSet.load("/tmp/test_set.json")
assert len(loaded) == 1
assert loaded.cases[0]["expected"] == "world"

# Test 3: Evaluation produces confidence intervals
def dummy_prompt(text):
    return "positive"  # Always predicts positive

ts = TestSet("binary", "exact_match")
ts.add("I love it", "positive")
ts.add("I hate it", "negative")
ts.add("It's okay", "neutral")

result = evaluate(dummy_prompt, ts, score_exact_match)
assert result["accuracy"] == 1/3  # Only 1 of 3 correct
assert result["ci_95"][0] <= result["accuracy"] <= result["ci_95"][1]
assert len(result["failures"]) == 2  # Two incorrect predictions

# Test 4: Format compliance scoring
assert score_format_compliance('{"key": "value"}') == 1.0
assert score_format_compliance("not json") == 0.0
assert score_format_compliance('```json\n{"key": "value"}\n```') == 1.0

Next: A/B Testing Prompts -- once you can measure quality, you run controlled experiments to compare prompts.