Prompt Optimization

Writing a prompt is the easy part. Making it better is the hard part -- and "better" is a number, not a feeling. Prompt optimization is a systematic loop: measure, change, measure again, stop when the gains flatten. This lesson covers the manual optimization loop, automated optimization with DSPy, and the diminishing returns curve that tells you when to stop.

What you'll learn

• Optimization is a loop: define a metric, run your prompt against a test set, tweak one thing, measure again. If the score went up, keep the change; if not, revert
• Manual optimization works for small prompt sets (under 10 prompts) and gives you deep understanding of what each phrase does. Automated optimization (DSPy, APE) scales to hundreds of prompts
• DSPy compiles natural language prompts into optimized versions by treating the LLM as a differentiable component -- it searches over few-shot examples, instruction phrasing, and output structure
• Diminishing returns are real: the first 3-5 iterations typically yield 80% of the total improvement. Past 10 iterations, you're usually overfitting to your test set

The problem

You wrote a prompt that extracts action items from meeting notes. It works, but it misses about 15% of the items and sometimes hallucinates actions that weren't discussed. You tweak the wording, add examples, adjust the system prompt. After an hour of trial and error, you're not sure if it's better or just different.

Optimization turns this from guesswork into engineering. You measure quality on a fixed test set, make one change at a time, and keep the changes that improve the score. When you do this systematically, you can trace every percentage point of improvement to a specific change -- and you know when to stop.

Build it

Step 1: The manual optimization loop

The core loop is deceptively simple. The discipline is in sticking to it:

import json
from openai import OpenAI

client = OpenAI()

def evaluate_prompt(prompt: str, test_set: list[dict], model: str = "gpt-4o-mini") -> dict:
    """Run a prompt against a test set and return accuracy."""
    correct = 0
    results = []

    for case in test_set:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": prompt},
                {"role": "user", "content": case["input"]}
            ],
            temperature=0  # Deterministic for evaluation
        )
        output = response.choices[0].message.content
        is_correct = output.strip() == case["expected"].strip()
        if is_correct:
            correct += 1
        results.append({"input": case["input"], "output": output, "expected": case["expected"], "correct": is_correct})

    return {
        "accuracy": correct / len(test_set),
        "results": results
    }

def optimize_loop(
    base_prompt: str,
    test_set: list[dict],
    iterations: int = 5,
    model: str = "gpt-4o-mini"
) -> list[dict]:
    """Manual optimization: measure, tweak, measure, repeat."""
    history = []
    current_prompt = base_prompt

    # Baseline
    baseline = evaluate_prompt(current_prompt, test_set, model)
    history.append({"iteration": 0, "prompt": current_prompt, "accuracy": baseline["accuracy"], "errors": [r for r in baseline["results"] if not r["correct"]]})

    for i in range(1, iterations + 1):
        # Ask the model to suggest an improvement based on errors
        error_cases = "\n".join([
            f"Input: {r['input']}\nExpected: {r['expected']}\nGot: {r['output']}"
            for r in history[-1]["errors"][:5]  # Show up to 5 errors
        ])

        improvement_prompt = f"""Current prompt:
---
{current_prompt}
---

This prompt made the following errors on a test set:
{error_cases}

Suggest an improved version of the prompt. Make exactly ONE change: add a constraint, clarify an instruction, add an example, or rephrase for clarity. Return ONLY the improved prompt, no explanation."""

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": improvement_prompt}],
            temperature=0.3
        )
        candidate = response.choices[0].message.content.strip()

        # Evaluate the candidate
        result = evaluate_prompt(candidate, test_set, model)

        if result["accuracy"] > history[-1]["accuracy"]:
            current_prompt = candidate  # Keep the improvement
            history.append({"iteration": i, "prompt": current_prompt, "accuracy": result["accuracy"], "errors": [r for r in result["results"] if not r["correct"]]})
            print(f"Iteration {i}: ACCEPTED -- accuracy {history[-2]['accuracy']:.2%} -> {result['accuracy']:.2%}")
        else:
            history.append({"iteration": i, "prompt": current_prompt, "accuracy": history[-1]["accuracy"], "errors": history[-1]["errors"], "rejected": True})
            print(f"Iteration {i}: REJECTED -- accuracy stayed at {history[-1]['accuracy']:.2%}")

        # Stop if we've hit diminishing returns
        if len(history) >= 4 and all(
            h.get("accuracy", 0) == history[-1]["accuracy"]
            for h in history[-3:]
        ):
            print("Diminishing returns detected -- stopping.")
            break

    return history


# Example test set
test_set = [
    {"input": "Classify: 'I love this product!'", "expected": "positive"},
    {"input": "Classify: 'This is terrible.'", "expected": "negative"},
    {"input": "Classify: 'It's okay, I guess.'", "expected": "neutral"},
    {"input": "Classify: 'Absolutely fantastic experience.'", "expected": "positive"},
    {"input": "Classify: 'Worst purchase ever.'", "expected": "negative"},
]

base_prompt = "Classify the sentiment of the text as positive, negative, or neutral."
history = optimize_loop(base_prompt, test_set, iterations=5)

Step 2: Automated optimization with DSPy

DSPy treats prompt optimization as a compiler problem. You define your task as a signature, provide a metric, and DSPy searches over prompts, few-shot examples, and output formats:

import dspy

# Configure the LM
lm = dspy.LM("gpt-4o-mini")
dspy.configure(lm=lm)

# Define the task signature
class SentimentClassifier(dspy.Signature):
    """Classify the sentiment of text as positive, negative, or neutral."""
    text = dspy.InputField()
    sentiment = dspy.OutputField(desc="one of: positive, negative, neutral")

# Build the module
classify = dspy.ChainOfThought(SentimentClassifier)

# Define the metric
def sentiment_accuracy(example, pred, trace=None):
    return pred.sentiment.lower() == example.sentiment.lower()

# Training data
trainset = [
    dspy.Example(text="I love this product!", sentiment="positive").with_inputs("text"),
    dspy.Example(text="This is terrible.", sentiment="negative").with_inputs("text"),
    dspy.Example(text="It's okay, I guess.", sentiment="neutral").with_inputs("text"),
    dspy.Example(text="Absolutely fantastic experience.", sentiment="positive").with_inputs("text"),
    dspy.Example(text="Worst purchase ever.", sentiment="negative").with_inputs("text"),
]

# Optimize: DSPy searches over few-shot examples and prompt phrasing
optimizer = dspy.BootstrapFewShot(
    metric=sentiment_accuracy,
    max_bootstrapped_demos=4,
    max_labeled_demos=4
)

optimized_classify = optimizer.compile(classify, trainset=trainset)

# The optimized prompt is now inside the module
print(optimized_classify(text="This movie was surprisingly good!"))
# sentiment='positive'

Step 3: Track the optimization curve

The diminishing returns curve is real. Plot it to know when to stop:

import matplotlib.pyplot as plt

def plot_optimization_curve(history: list[dict], save_path: str = "optimization_curve.png"):
    iterations = [h["iteration"] for h in history]
    accuracies = [h["accuracy"] for h in history]

    plt.figure(figsize=(8, 4))
    plt.plot(iterations, accuracies, marker="o", linewidth=2)
    plt.axhline(y=max(accuracies), color="green", linestyle="--", alpha=0.5, label=f"Best: {max(accuracies):.1%}")
    plt.axvline(x=3, color="orange", linestyle=":", alpha=0.5, label="Diminishing returns zone")
    plt.xlabel("Iteration")
    plt.ylabel("Accuracy")
    plt.title("Prompt Optimization Curve")
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.savefig(save_path)
    plt.close()

    # Calculate marginal gains
    marginal = []
    for i in range(1, len(accuracies)):
        marginal.append(accuracies[i] - accuracies[i-1])
    print(f"Marginal gains per iteration: {[f'{g:.2%}' for g in marginal]}")
    print(f"Total improvement: {accuracies[-1] - accuracies[0]:.2%}")

# Use with the history from optimize_loop
# plot_optimization_curve(history)

Step 4: The optimization log

Every optimization run should produce a log you can review later:

def write_optimization_report(history: list[dict], filepath: str = "optimization_report.md"):
    with open(filepath, "w") as f:
        f.write("# Prompt Optimization Report\n\n")
        f.write(f"## Baseline\n")
        f.write(f"Accuracy: {history[0]['accuracy']:.2%}\n")
        f.write(f"Prompt:\n```\n{history[0]['prompt']}\n```\n\n")

        for h in history[1:]:
            status = "REJECTED" if h.get("rejected") else "ACCEPTED"
            f.write(f"## Iteration {h['iteration']} ({status})\n")
            f.write(f"Accuracy: {h['accuracy']:.2%}\n")
            if not h.get("rejected"):
                f.write(f"Prompt:\n```\n{h['prompt']}\n```\n")
            f.write("\n")

        final = history[-1]
        improvement = final["accuracy"] - history[0]["accuracy"]
        f.write(f"## Summary\n")
        f.write(f"Total improvement: {improvement:+.2%}\n")
        f.write(f"Final accuracy: {final['accuracy']:.2%}\n")
        f.write(f"Iterations: {len(history) - 1}\n")

What goes wrong

Mistake	How you notice it	The fix
Optimizing without a test set	You think the prompt is improving, but it's just fitting to the one example you're testing with	Always have a fixed test set of at least 20 examples. Never evaluate on the same examples you use for development
Changing multiple things at once	Accuracy changes but you don't know which change caused it	Make one change per iteration. If you change the instruction, add an example, and rephrase the output format all at once, you can't isolate what worked
Overfitting to the test set	Accuracy on the test set is 95% but drops to 70% on real data	Hold out a separate validation set that you never use during optimization. Only check it at the end to confirm real improvement
Stopping too early	You accept the first improvement and move on	Run at least 5 iterations, even if the first one works. The first improvement is usually the easiest -- later iterations find the edge cases
Not reverting bad changes	Accuracy drops but you keep the change because you "like the wording better"	Optimization is about the metric, not aesthetics. If the score drops, revert. No exceptions

Confirm it worked

Run these tests to validate the optimization loop and stagnation detection logic. They confirm the loop prevents accuracy regression and properly triggers early stopping if improvements plateau.

# Test 1: Optimization loop improves accuracy
base_prompt = "Classify sentiment."
test_set = [
    {"input": "Classify: 'I love it!'", "expected": "positive"},
    {"input": "Classify: 'I hate it.'", "expected": "negative"},
]
history = optimize_loop(base_prompt, test_set, iterations=3, model="gpt-4o-mini")
assert history[-1]["accuracy"] >= history[0]["accuracy"]  # Should not regress

# Test 2: Stagnation triggers early stopping
# (Simulated -- real test would need a prompt that can't improve further)
stagnant_history = [
    {"iteration": 0, "accuracy": 0.80},
    {"iteration": 1, "accuracy": 0.80, "rejected": True},
    {"iteration": 2, "accuracy": 0.80, "rejected": True},
    {"iteration": 3, "accuracy": 0.80, "rejected": True},
]
# Check that the stopping condition would fire
recent = [h["accuracy"] for h in stagnant_history[-3:]]
assert len(set(recent)) == 1  # All same -- should stop

# Test 3: Optimization report is written
write_optimization_report(history, "/tmp/test_optimization_report.md")
with open("/tmp/test_optimization_report.md") as f:
    content = f.read()
    assert "Baseline" in content
    assert "Accuracy" in content
    assert "Summary" in content

Next: Evaluating Prompt Quality -- the metrics and test sets that make optimization possible.