Testing & Evaluating Agents

You changed the system prompt, or swapped in a new model, or tweaked how a tool gets called, and now you have no idea whether the agent actually got better or just got different. This is the state most agent projects live in permanently: someone runs a handful of manual tests, decides it "feels right," and ships it. That works right up until the day it doesn't, and by then the change that broke things is buried under three more changes made without any way to check them either. This page is about replacing that feeling with a real answer.

What you'll learn

• Why "I tried it a few times and it seemed fine" doesn't scale past your first deploy
• Building a small eval set from your own real failures, not synthetic examples
• Using an LLM as a judge, and the one thing that makes that trustworthy instead of circular

The problem

An agent's behavior isn't fixed the way a normal program's is. The same prompt can produce a slightly different response on two different runs, a model provider can silently update the underlying model, and a "small" prompt tweak can shift behavior on cases you never thought to check. Manual testing catches the cases you remember to try. It never catches the regression on the case you didn't think of, which, in practice, is usually the case a real user hits first.

The fix isn't complicated in concept: build a set of representative inputs, run them through the agent automatically, and check the outputs against a standard. The part people skip is doing it before they need it, so that by the time something breaks, there's already a baseline to compare against.

Options & when to use each

Approach	Good for	Costs you	When to pick it
Manual spot-checking	Very early prototyping, before behavior has stabilized	Catches nothing systematically; doesn't scale	The first few days of building something, before you have a stable enough system to evaluate
Rule-based checks	Anything with a checkable, objective answer (does it parse? is the number in range? did it call the right tool?)	Can't judge subjective quality (was this a good answer?)	Whenever there's a mechanical way to check correctness, always prefer this over an LLM judge if one exists
LLM-as-judge	Subjective quality (tone, completeness, whether an answer actually addresses the question)	Cost of an extra model call per eval; needs its own calibration	Anything a rule can't check, especially free-text quality

Most real eval suites use both: rule-based checks for anything mechanically verifiable, and an LLM judge for the rest. Neither one replaces the other.

Build it

Start with real failures, not invented test cases

The highest-value eval cases are the ones pulled from things that actually went wrong, a support ticket where the agent gave a bad answer, a log entry where it called the wrong tool, a case a teammate flagged in review. Synthetic test cases you invent from imagination tend to test what you already believe the agent handles correctly; real failures test what it actually gets wrong.

EVAL_CASES = [
    {
        "input": "What's the refund policy for orders over 30 days old?",
        "expected_tool_call": "search_knowledge_base",
        "must_not_contain": ["I don't know", "I'm not sure"],
    },
    {
        "input": "Cancel my subscription immediately",
        "expected_tool_call": "cancel_subscription",
        "requires_confirmation": True,  # this one should trigger a human-approval gate
    },
]

Rule-based checks first

Before running expensive model-based evaluations, execute fast rule-based validation checks to verify that the agent called the expected tool names and did not output forbidden phrases:

def check_rules(case: dict, agent_output: dict) -> list[str]:
    failures = []
    if "expected_tool_call" in case:
        if agent_output.get("tool_called") != case["expected_tool_call"]:
            failures.append(
                f"Expected tool '{case['expected_tool_call']}', "
                f"got '{agent_output.get('tool_called')}'"
            )
    for phrase in case.get("must_not_contain", []):
        if phrase.lower() in agent_output["text"].lower():
            failures.append(f"Output contains forbidden phrase: '{phrase}'")
    return failures

LLM-as-judge for everything else

For quality that can't be checked with a simple rule, did the response actually answer the question, was the tone appropriate, a second model scores the output against a rubric:

JUDGE_PROMPT = """You are evaluating an AI agent's response for quality.

User asked: {input}
Agent responded: {output}

Score the response from 1-5 on:
- Relevance: does it actually address what was asked?
- Completeness: is anything important missing?

Respond with JSON: {{"relevance": <1-5>, "completeness": <1-5>, "reasoning": "<one sentence>"}}"""

def judge_response(case: dict, agent_output: dict) -> dict:
    result = call_llm(
        model="claude-sonnet-4-6",
        prompt=JUDGE_PROMPT.format(input=case["input"], output=agent_output["text"]),
    )
    return json.loads(result)

The thing that makes an LLM judge trustworthy instead of just another unreliable model call: use a different, typically stronger model as the judge than the one being evaluated, and periodically spot-check the judge's scores against your own read of a handful of cases. A judge that consistently agrees with your own assessment is doing its job; one that doesn't needs a clearer rubric, not more trust.

Run it as a gate, not a report

The eval suite only earns its keep if it actually blocks something:

def run_eval_suite(cases: list[dict]) -> bool:
    all_passed = True
    for case in cases:
        output = run_agent(case["input"])
        rule_failures = check_rules(case, output)
        judge_scores = judge_response(case, output)

        if rule_failures or judge_scores["relevance"] < 3:
            all_passed = False
            print(f"FAILED: {case['input'][:50]}... ,  {rule_failures or judge_scores}")

    return all_passed

# Wire this into your deploy step, don't ship on a failed eval run
if not run_eval_suite(EVAL_CASES):
    raise SystemExit("Eval suite failed, not deploying")

What goes wrong

Mistake	How you notice it	The fix
Eval set never grows	The same 5 cases keep passing while real users hit new failure modes	Add a new eval case every time a real failure gets reported, this is how the suite earns its keep over time
Using an LLM judge for things a rule could check	Slower, more expensive evals than necessary, and occasional judge inconsistency on things that have a clear right answer	Use a rule wherever one exists (did it call the right tool? is the number in range?); save the judge for genuinely subjective quality
Judge model same as the model being evaluated	Judge tends to rate the model's own style favorably, masking real quality issues	Use a different (ideally stronger) model as the judge
Evals run manually, occasionally	Regressions ship because nobody happened to run the suite that day	Wire the eval run into your deploy process as a gate, not an optional check

Confirm it worked

To verify that your test suite successfully identifies regressions and halts deployment pipelines when failure conditions are encountered, run this test validation check:

# Confirm the suite actually catches a known-bad case
BROKEN_CASE = {"input": "Cancel my subscription", "expected_tool_call": "cancel_subscription"}
bad_output = {"tool_called": None, "text": "I've noted your request."}  # simulated bug: no tool called
assert check_rules(BROKEN_CASE, bad_output) != []  # should report a failure, not pass silently

# Confirm it's wired into your deploy path
python deploy.py --dry-run
# should print "Eval suite failed" and exit non-zero if you intentionally break a case first

If the suite catches a deliberately broken case and blocks a deploy on it, it's doing the job a "seemed fine when I tried it" check never could.

Next: Day 5 , Spec-Driven Production Development.