A/B Testing Prompts

Evaluation tells you which prompt scores higher on a test set. A/B testing tells you which prompt performs better in production, with real users and real data. This lesson covers designing controlled experiments, computing statistical significance with small sample sizes, blind evaluation, and recording results so you can reproduce the winner.

What you'll learn

• A/B testing compares two prompts (A and B) against the same inputs, measuring the same metric, to determine which one is better -- not just "looks better"
• Statistical significance answers the question: "Is this difference real, or could it be random noise?" With small sample sizes (under 100), use Fisher's exact test or bootstrap resampling instead of the normal approximation
• Blind evaluation removes confirmation bias: the evaluator (human or LLM) should not know which prompt produced which output. Randomize the order and strip identifiers
• Record everything: the exact prompt text, the test inputs, the outputs, the scores, and the statistical test result. A result you can't reproduce is a result you can't trust

The problem

You optimized a prompt and it scores 8% higher on your test set. Exciting. But is that 8% real, or just noise from a small test set? And will it hold up when real users send inputs that look nothing like your test cases?

A/B testing answers both questions. You run both prompts against the same inputs, compute a metric, and run a statistical test to determine whether the difference is significant. If it is, you ship the winner. If it isn't, you collect more data or accept that the prompts are effectively equivalent.

Build it

Step 1: Design the experiment

An A/B test has four components: the two prompts, the metric, the test inputs, and the statistical test. Design all four before you run anything:

import random
import json
from datetime import datetime
from pathlib import Path

class ABTest:
    def __init__(self, name: str, prompt_a: str, prompt_b: str, metric_fn):
        self.name = name
        self.prompt_a = prompt_a
        self.prompt_b = prompt_b
        self.metric_fn = metric_fn  # (output, expected) -> float
        self.results_a = []
        self.results_b = []
        self.timestamp = datetime.now().isoformat()

    def run(self, test_cases: list[dict], model: str = "gpt-4o-mini", blind: bool = True) -> dict:
        """Run both prompts against the same test cases."""
        from openai import OpenAI
        client = OpenAI()

        for case in test_cases:
            if blind:
                # Randomize order to prevent bias
                prompts = [("A", self.prompt_a), ("B", self.prompt_b)]
                random.shuffle(prompts)
            else:
                prompts = [("A", self.prompt_a), ("B", self.prompt_b)]

            for label, prompt in prompts:
                response = client.chat.completions.create(
                    model=model,
                    messages=[
                        {"role": "system", "content": prompt},
                        {"role": "user", "content": case["input"]}
                    ],
                    temperature=0
                )
                output = response.choices[0].message.content
                score = self.metric_fn(output, case["expected"])

                result = {
                    "case_id": case.get("id", "unknown"),
                    "input": case["input"],
                    "expected": case["expected"],
                    "output": output,
                    "score": score
                }

                if label == "A":
                    self.results_a.append(result)
                else:
                    self.results_b.append(result)

        return self.summarize()

    def summarize(self) -> dict:
        scores_a = [r["score"] for r in self.results_a]
        scores_b = [r["score"] for r in self.results_b]

        return {
            "name": self.name,
            "timestamp": self.timestamp,
            "n_cases": len(self.results_a),
            "prompt_a_mean": sum(scores_a) / len(scores_a) if scores_a else 0,
            "prompt_b_mean": sum(scores_b) / len(scores_b) if scores_b else 0,
            "prompt_a_wins": sum(1 for a, b in zip(scores_a, scores_b) if a > b),
            "prompt_b_wins": sum(1 for a, b in zip(scores_a, scores_b) if b > a),
            "ties": sum(1 for a, b in zip(scores_a, scores_b) if a == b),
        }

Step 2: Test for statistical significance

With small sample sizes (under 100 cases), standard z-tests are unreliable. Use Fisher's exact test for binary metrics or bootstrap resampling for continuous metrics:

from scipy.stats import fisher_exact
import numpy as np

def test_significance_fisher(results_a: list[dict], results_b: list[dict]) -> dict:
    """Fisher's exact test for binary outcomes (correct/incorrect)."""
    a_correct = sum(1 for r in results_a if r["score"] >= 1.0)
    a_incorrect = len(results_a) - a_correct
    b_correct = sum(1 for r in results_b if r["score"] >= 1.0)
    b_incorrect = len(results_b) - b_correct

    # Contingency table
    table = [[a_correct, a_incorrect],
             [b_correct, b_incorrect]]

    odds_ratio, p_value = fisher_exact(table)

    return {
        "test": "Fisher's exact test",
        "p_value": p_value,
        "significant": p_value < 0.05,
        "a_correct": a_correct,
        "b_correct": b_correct,
        "a_total": len(results_a),
        "b_total": len(results_b),
        "odds_ratio": odds_ratio
    }

def test_significance_bootstrap(
    results_a: list[dict],
    results_b: list[dict],
    n_bootstrap: int = 10_000
) -> dict:
    """Bootstrap test for continuous scores. No normality assumption."""
    scores_a = np.array([r["score"] for r in results_a])
    scores_b = np.array([r["score"] for r in results_b])

    observed_diff = np.mean(scores_b) - np.mean(scores_a)

    # Pool the scores and resample
    pooled = np.concatenate([scores_a, scores_b])
    n_a = len(scores_a)

    diffs = []
    for _ in range(n_bootstrap):
        np.random.shuffle(pooled)
        boot_a = pooled[:n_a]
        boot_b = pooled[n_a:]
        diffs.append(np.mean(boot_b) - np.mean(boot_a))

    diffs = np.array(diffs)
    p_value = np.mean(np.abs(diffs) >= np.abs(observed_diff))

    return {
        "test": "Bootstrap (permutation)",
        "p_value": p_value,
        "significant": p_value < 0.05,
        "observed_diff": observed_diff,
        "ci_95": (np.percentile(diffs, 2.5), np.percentile(diffs, 97.5)),
        "n_bootstrap": n_bootstrap
    }

Step 3: Compute required sample size

Before running an A/B test, know how many examples you need to detect a given effect size:

def required_sample_size(
    baseline_accuracy: float,
    expected_improvement: float,
    alpha: float = 0.05,
    power: float = 0.80
) -> int:
    """
    Compute the sample size needed to detect a given improvement.
    Uses the normal approximation for proportions.
    """
    from scipy.stats import norm

    z_alpha = norm.ppf(1 - alpha / 2)  # Two-tailed
    z_beta = norm.ppf(power)

    p1 = baseline_accuracy
    p2 = baseline_accuracy + expected_improvement
    p_pooled = (p1 + p2) / 2

    n = (z_alpha * np.sqrt(2 * p_pooled * (1 - p_pooled)) +
         z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2

    return int(np.ceil(n))


# Example: detecting a 5% improvement from an 80% baseline
n_needed = required_sample_size(
    baseline_accuracy=0.80,
    expected_improvement=0.05
)
print(f"Cases needed: {n_needed}")  # ~1000+ cases for 5% improvement
print(f"Cases needed for 10% improvement: {required_sample_size(0.80, 0.10)}")  # ~250
print(f"Cases needed for 15% improvement: {required_sample_size(0.80, 0.15)}")  # ~110

Step 4: Record and compare results

Every A/B test should produce a machine-readable report you can archive:

def write_ab_report(ab_test: ABTest, significance: dict, filepath: str):
    """Write a complete A/B test report."""
    summary = ab_test.summarize()

    report = {
        "experiment": summary["name"],
        "timestamp": summary["timestamp"],
        "prompt_a": ab_test.prompt_a,
        "prompt_b": ab_test.prompt_b,
        "n_cases": summary["n_cases"],
        "results": {
            "prompt_a_mean": summary["prompt_a_mean"],
            "prompt_b_mean": summary["prompt_b_mean"],
            "prompt_a_wins": summary["prompt_a_wins"],
            "prompt_b_wins": summary["prompt_b_wins"],
            "ties": summary["ties"],
            "delta": summary["prompt_b_mean"] - summary["prompt_a_mean"]
        },
        "significance": significance,
        "verdict": "B WINS" if (
            significance["significant"] and
            summary["prompt_b_mean"] > summary["prompt_a_mean"]
        ) else "A WINS" if (
            significance["significant"] and
            summary["prompt_a_mean"] > summary["prompt_b_mean"]
        ) else "NO SIGNIFICANT DIFFERENCE"
    }

    with open(filepath, "w") as f:
        json.dump(report, f, indent=2)

    print(f"\n{'='*50}")
    print(f"A/B Test: {report['experiment']}")
    print(f"{'='*50}")
    print(f"Prompt A: {report['results']['prompt_a_mean']:.2%}")
    print(f"Prompt B: {report['results']['prompt_b_mean']:.2%}")
    print(f"Delta: {report['results']['delta']:+.2%}")
    print(f"p-value: {report['significance']['p_value']:.4f}")
    print(f"Verdict: {report['verdict']}")
    print(f"Report saved to: {filepath}")

    return report


# Full example
def sentiment_metric(output: str, expected: str) -> float:
    """Binary metric: 1.0 if correct, 0.0 otherwise."""
    return 1.0 if output.strip().lower() == expected.strip().lower() else 0.0

test_cases = [
    {"id": "1", "input": "Classify: 'I love this!'", "expected": "positive"},
    {"id": "2", "input": "Classify: 'Terrible.'", "expected": "negative"},
    {"id": "3", "input": "Classify: 'It's okay.'", "expected": "neutral"},
    {"id": "4", "input": "Classify: 'Fantastic!'", "expected": "positive"},
    {"id": "5", "input": "Classify: 'Awful.'", "expected": "negative"},
    {"id": "6", "input": "Classify: 'Not bad.'", "expected": "neutral"},
    {"id": "7", "input": "Classify: 'Amazing service.'", "expected": "positive"},
    {"id": "8", "input": "Classify: 'Disappointed.'", "expected": "negative"},
    {"id": "9", "input": "Classify: 'Fine, I guess.'", "expected": "neutral"},
    {"id": "10", "input": "Classify: 'Best ever!'", "expected": "positive"},
]

ab = ABTest(
    name="sentiment-prompt-v2-vs-v3",
    prompt_a="Classify the sentiment as positive, negative, or neutral.",
    prompt_b="Classify the sentiment as positive, negative, or neutral. Return only the word, lowercase, no punctuation.",
    metric_fn=sentiment_metric
)

summary = ab.run(test_cases, blind=True)
sig = test_significance_fisher(ab.results_a, ab.results_b)
write_ab_report(ab, sig, "ab_reports/sentiment-v2-vs-v3.json")

What goes wrong

Mistake	How you notice it	The fix
Peeking at results early	You stop the test as soon as p < 0.05, but the p-value fluctuates	Set the sample size before you start and don't look at the results until you've collected all of them. Sequential testing inflates the false positive rate
Multiple comparisons without correction	You test 10 prompt variants against baseline and call the one with p < 0.05 the winner	Apply Bonferroni correction: divide alpha by the number of comparisons. For 10 comparisons, p must be < 0.005 to be significant
Small sample size, large effect claimed	p < 0.05 but the confidence interval is huge (e.g., +5% +/- 20%)	Report confidence intervals alongside p-values. A significant result with a CI that spans zero is not actionable
Non-blind evaluation	The evaluator (human or LLM) prefers Prompt B because they know it's the "new" one	Randomize the order of outputs. Strip any identifiers. If using LLM-as-judge, present both outputs and ask which is better (or score each independently)
Confusing statistical significance with practical significance	p < 0.05 but the improvement is 0.5% -- not worth the engineering effort	Define a minimum effect size before the test. If the improvement is smaller than your threshold, it doesn't matter if it's statistically significant

Confirm it worked

Run the following test suite to verify the statistical significance functions are working correctly. This confirms that Fisher's exact test and bootstrap testing behave as expected against baseline data.

# Test 1: Fisher's test detects a real difference
results_a = [{"score": 1.0}] * 8 + [{"score": 0.0}] * 2  # 80% correct
results_b = [{"score": 1.0}] * 10 + [{"score": 0.0}] * 0  # 100% correct
sig = test_significance_fisher(results_a, results_b)
assert sig["a_correct"] == 8
assert sig["b_correct"] == 10
# Note: with n=10: Fisher's test won't be significant for this small difference

# Test 2: Bootstrap detects no difference when scores are identical
results_a = [{"score": 0.8}] * 20
results_b = [{"score": 0.8}] * 20
sig = test_significance_bootstrap(results_a, results_b, n_bootstrap=1000)
assert sig["observed_diff"] == 0.0
assert not sig["significant"]

# Test 3: Sample size calculator gives reasonable numbers
n_small = required_sample_size(0.80, 0.20)  # 20% improvement
n_large = required_sample_size(0.80, 0.02)  # 2% improvement
assert n_small < n_large  # Detecting a smaller effect requires more data

# Test 4: A/B test report is complete
ab = ABTest("test", "prompt A", "prompt B", lambda o, e: 1.0 if o == e else 0.0)
test_cases = [
    {"id": "1", "input": "x", "expected": "a"},
    {"id": "2", "input": "y", "expected": "b"},
]
ab.run(test_cases, blind=False)
report = ab.summarize()
assert report["n_cases"] == 2
assert "prompt_a_mean" in report
assert "prompt_b_mean" in report

Next: Day 4 -- Advanced Techniques -- tool use, multi-turn conversations, prompt chaining, and RAG.