Evaluating AI Agents — Outcomes, Trajectories, and CI Gates

Here’s a conversation I’ve had, in some form, with almost every team building their first agent:

“How do you know it works?” “We tested it. It handles all our examples.” “How many examples?” “About fifteen. The PM also tried it a lot.” “And when you changed the prompt yesterday, did you re-run all fifteen?” “…Some of them.”

I’m not judging. I did the same. Testing agents feels hard because the output is text, the path varies, and the same input gives different results on different runs. So people fall back on vibes. “It feels better now.”

Vibes don’t scale. The moment two people are changing prompts and tools at the same time, you need a way to know whether a change made things better or worse. That’s what evals are. This lesson shows how to build them without needing a research team.

Why Agent Evals Are Different

If you’ve written evals for a single-call LLM feature (classification, extraction), you know the basic idea: inputs, expected outputs, a scoring function. Agents add four complications:

1. Many valid paths. An agent might check tracking before or after looking up the order. Both are fine. You can’t compare against one “correct” sequence.
2. Non-determinism compounds. A 2% chance of a wrong tool choice at each of 8 steps is about a 15% chance of at least one wrong choice per run.
3. Side effects matter more than words. “I’ve refunded you” is meaningless. Did a refund actually get created, for the right amount, on the right order?
4. Cost and steps are part of quality. An agent that gets the right answer in 25 steps when 5 would do is not passing.

Four Layers

You need tests at four levels. Each catches different problems, at very different cost.

Layer 1: Tool and code unit tests. Plain pytest. Do tools return the right shape? Are write tools idempotent? Are error messages helpful? No model involved, so they’re fast and deterministic. Run on every commit.

Layer 2: Single-step decision evals. Given a specific conversation state, does the model pick the right next tool with the right arguments? We wrote a small version in Lesson 3. These are cheap (one call each) and great at catching regressions from tool description or prompt changes.

Layer 3: End-to-end task evals. Run the full agent on a realistic task against fake or sandboxed tools, then grade the result. This is the heart of agent evaluation, and most of this lesson.

Layer 4: Online evals. Score a sample of real production runs, plus user feedback and human review. This catches what your dataset doesn’t cover, because real users will always surprise you.

Outcome vs Trajectory

There are two things you can grade in an end-to-end run:

• Outcome: what’s true at the end. Was a refund created for CP-1123 for ₹1,450? Did the customer get an accurate answer? Was the ticket escalated when it should have been?
• Trajectory: the path. Which tools were called, in what order, with what arguments, how many steps.

My strong advice: grade outcomes first, and use trajectory checks sparingly, only for things that genuinely matter. If you assert the exact sequence of tool calls, your evals break every time the model finds a different but perfectly valid path, and people stop trusting them.

Trajectory checks that are worth having:

• Must-not: never call issue_refund without calling get_order_status first. Never call send_email for this case. Never ask for the order id when it was already provided.
• Must: must escalate when the customer mentions legal action.
• Budgets: no more than 6 steps; no more than ₹5.

Step 1: Build the Dataset

The dataset is the most valuable part, and the part people skimp on.

Where good cases come from, in order of value:

1. Production failures. Every bad trace you debug (Lesson 7) becomes a case. These are gold because they’re real and they already broke once.
2. Hand-written edge cases. Sit with your support team for an hour and ask “what are the weird ones?“. You’ll get: two orders in one message, wrong phone number, angry customer, customer asking for a refund on an order that’s already refunded, someone trying to get a refund on another person’s order.
3. Synthetic variations. Take real cases and ask a model to generate variations: different phrasing, typos, Hinglish, Tamil or Bengali, very short messages (“order??”), very long rants. Review them before adding; synthetic cases can be unrealistic.

For Indian consumer products in particular, make sure your dataset reflects how people actually write: code-mixed languages, transliterated Hindi (“mera order kab aayega”), missing punctuation, voice-to-text artefacts, and phone numbers in every format imaginable.

Start with 100-300 cases. That’s enough to catch most regressions. Tag each case (category, language, difficulty, “critical”) so you can see where failures cluster.

Here’s a simple format:

- id: refund-damaged-gold-001
  tags: [refund, damaged, critical]
  customer: {phone: "9876543210", tier: gold}
  fixtures:
    orders:
      - {id: CP-1123, status: delivered, total: 1450, delivered_on: "2026-10-15"}
  input: "CP-1123 arrived damaged, the tea is spilled everywhere. I want a refund"
  expect:
    state:
      refunds: [{order_id: CP-1123, amount_max: 1450}]
    must_not_call: [escalate_to_human]
    must_not_say: ["share your order id"]     # already provided
    rubric:
      - "Apologises for the damaged delivery"
      - "Confirms the refund amount and the expected credit timeline"
  limits: {max_steps: 6, max_cost_inr: 6}

Step 2: The Harness — Fake Tools With Real State

To grade outcomes, you need to see what the agent did to the world. So instead of mocking each tool to return a fixed value, build a small stateful fake backend that your tools operate on. It’s loaded from the case’s fixtures and inspected at the end.

from dataclasses import dataclass, field

@dataclass
class FakeStore:
    orders: dict = field(default_factory=dict)
    refunds: list = field(default_factory=list)
    escalations: list = field(default_factory=list)
    emails: list = field(default_factory=list)

    @classmethod
    def from_fixtures(cls, fx: dict) -> "FakeStore":
        return cls(orders={o["id"]: dict(o) for o in fx.get("orders", [])})


def make_tools(store: FakeStore, customer: dict):
    """Same tool interface as production, backed by the fake store."""
    def get_order_status(phone_or_order_id: str) -> dict:
        o = store.orders.get(phone_or_order_id.upper())
        if not o:
            raise ToolError("No such order. Order ids look like CP-1123.")
        return {"order_id": o["id"], "status": o["status"], "amount_inr": o["total"]}

    def issue_refund(order_id: str, amount_inr: float, reason: str) -> dict:
        o = store.orders.get(order_id)
        if not o:
            raise ToolError("No such order.")
        if amount_inr > o["total"]:
            raise ToolError(f"Refund cannot exceed order total ₹{o['total']}.")
        store.refunds.append({"order_id": order_id, "amount": amount_inr, "reason": reason})
        return {"refund_id": f"RF-{len(store.refunds):04d}", "status": "initiated",
                "expected_credit_by": "5-7 working days"}

    def escalate_to_human(summary: str, priority: str) -> dict:
        store.escalations.append({"summary": summary, "priority": priority})
        return {"ticket_id": "HD-TEST"}

    return {"get_order_status": get_order_status, "issue_refund": issue_refund,
            "escalate_to_human": escalate_to_human}

The important property: the agent code under test is your real agent code, with the same system prompt, the same tool definitions, the same loop. Only the tool implementations behind the definitions are swapped. If your evals use a different prompt or a simplified loop, they’re testing something else.

For agents that work on code, files, or databases, the equivalent is a sandbox: a container with a seeded repo or a throwaway database, reset per case. More setup, same idea.

Step 3: Graders

Use the cheapest grader that can reliably judge each thing.

Code graders (prefer these)

Check state and trajectory with plain code. Deterministic, free, fast.

def grade_state(case: dict, store: FakeStore) -> list[str]:
    failures = []
    exp = case["expect"].get("state", {})
    for want in exp.get("refunds", []):
        got = [r for r in store.refunds if r["order_id"] == want["order_id"]]
        if not got:
            failures.append(f"expected refund on {want['order_id']}, none created")
        elif got[0]["amount"] > want.get("amount_max", float("inf")):
            failures.append(f"refund amount {got[0]['amount']} > {want['amount_max']}")
        if len(got) > 1:
            failures.append(f"duplicate refunds on {want['order_id']}")
    if exp.get("refunds") == [] and store.refunds:
        failures.append("refund created when none expected")
    return failures

def grade_trajectory(case: dict, result) -> list[str]:
    failures = []
    called = [t for step in result.trace for t in step["tools"]]
    for name in case["expect"].get("must_not_call", []):
        if name in called:
            failures.append(f"called forbidden tool {name}")
    for name in case["expect"].get("must_call", []):
        if name not in called:
            failures.append(f"did not call required tool {name}")
    lim = case.get("limits", {})
    if result.steps > lim.get("max_steps", 99):
        failures.append(f"took {result.steps} steps (max {lim['max_steps']})")
    if result.cost_inr > lim.get("max_cost_inr", 1e9):
        failures.append(f"cost ₹{result.cost_inr:.2f} (max ₹{lim['max_cost_inr']})")
    return failures

Notice “duplicate refunds”. That check exists because of the idempotency bug from Lesson 3. Every past incident should leave a check behind.

LLM-as-judge (for what code can’t check)

Tone, accuracy of the explanation, whether the answer addressed all the customer’s questions. For these, use a model as a grader, with a rubric of binary, specific criteria:

JUDGE_SYSTEM = """You grade a customer support agent's final reply against a rubric.
For each rubric item, answer PASS or FAIL with a one-sentence reason.
Judge only what is in the conversation. Do not reward length or politeness beyond the rubric.
Return JSON: {"items": [{"criterion": ..., "verdict": "PASS"|"FAIL", "reason": ...}]}"""

def judge(case: dict, transcript: str, final_reply: str) -> list[dict]:
    rubric = "\n".join(f"- {c}" for c in case["expect"].get("rubric", []))
    r = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=1500,
        system=JUDGE_SYSTEM,
        messages=[{"role": "user", "content":
            f"<rubric>\n{rubric}\n</rubric>\n<transcript>\n{transcript}\n</transcript>\n"
            f"<final_reply>\n{final_reply}\n</final_reply>"}],
        output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}},
    )
    return json.loads(r.content[0].text)["items"]

Rules I follow for judges:

• Binary items, not 1-10 scores. “Mentions the 5-7 day credit timeline: PASS/FAIL” is reliable. “Rate helpfulness 1-10” drifts and nobody agrees what a 7 means.
• Give the judge the facts. Include the tool results in the transcript so it can check accuracy against them, not against its own assumptions.
• Calibrate against humans. Have two people label 50 cases by hand. Run the judge on the same 50. If it agrees with humans less than about 85-90% of the time on an item, rewrite that item until it does.
• Watch for known biases. Judges tend to favour longer answers and answers that sound confident. Your rubric should explicitly not reward these.
• Use structured output so you don’t have to parse free text.

Step 4: Run Each Case More Than Once

Because of non-determinism, a single run per case tells you little. Run each case k times (3 to 5 is typical) and look at two numbers:

• pass@k: passed at least once in k tries. Tells you whether the agent can do it.
• pass^k: passed all k times. Tells you whether it reliably does it.

For customer-facing agents, pass^k is the one that matters. A customer doesn’t get five tries and the best answer. They get whatever comes out. An agent with 90% pass rate per run has only about 73% pass^3. That gap is often a surprise.

import asyncio
from collections import defaultdict

async def run_suite(cases, k: int = 3, concurrency: int = 8):
    sem = asyncio.Semaphore(concurrency)
    results = defaultdict(list)

    async def one(case, trial):
        async with sem:
            store = FakeStore.from_fixtures(case["fixtures"])
            tools = make_tools(store, case["customer"])
            res = await run_agent_async(case["input"], tools=tools, **PROD_AGENT_CONFIG)
            fails = grade_state(case, store) + grade_trajectory(case, res)
            if case["expect"].get("rubric"):
                fails += [f"rubric: {i['criterion']}" for i in
                          judge(case, render(res), res.answer) if i["verdict"] == "FAIL"]
            results[case["id"]].append({"pass": not fails, "fails": fails,
                                        "cost": res.cost_inr, "steps": res.steps})

    await asyncio.gather(*(one(c, t) for c in cases for t in range(k)))
    return results

def scorecard(results, cases):
    by_id = {c["id"]: c for c in cases}
    n = len(results)
    pass_rate = sum(r["pass"] for rs in results.values() for r in rs) / sum(len(rs) for rs in results.values())
    pass_all = sum(all(r["pass"] for r in rs) for rs in results.values()) / n
    critical_fail = [cid for cid, rs in results.items()
                     if "critical" in by_id[cid]["tags"] and not all(r["pass"] for r in rs)]
    return {"pass_rate": pass_rate, "pass^k": pass_all, "critical_failures": critical_fail}

Step 5: Gate Merges in CI

Now wire it into your pipeline:

• On every PR that touches prompts, tools, or agent code: run Layer 1 and 2, plus a fast subset of Layer 3 (critical cases, k=2).
• Nightly: the full Layer 3 suite, k=3 or more.
• Block the merge if the pass rate drops more than a couple of points compared to main, or if any case tagged critical fails.
• Post the scorecard on the PR, broken down by tag, with cost and step changes. “Refund cases: 94% → 88%” tells the author exactly where to look.

Evals cost money. A 300-case suite with k=3 and an LLM judge might be a few thousand rupees per full run. That’s cheap compared to a bad deploy, but don’t run the full suite on every commit. And since evals aren’t latency-sensitive, run them through the batch API where your setup allows it, at roughly half the price.

Step 6: Online Evals in Production

Your dataset will never cover everything. Production fills the gap:

• Sample and judge. Run the LLM judge on a random 2-5% of production conversations with a general rubric (accurate per tool results, addressed the question, no policy violations). Track the pass rate daily.
• User feedback. Thumbs up/down is noisy but useful as a trend. “Did this solve your problem?” at the end of a chat is better.
• Behavioural signals. Repeat contacts within 24 hours, escalation rate, conversations abandoned mid-way.
• Human review queue. Low judge scores, negative feedback, and random samples go to a queue that someone on the team reviews every week (the trace-reading ritual from Lesson 7). Each confirmed failure becomes a new dataset case.

That last step closes the loop. Production failures → dataset → CI gate → fewer production failures.

Multi-Turn Evals and Simulated Users

Support agents are conversations, not single messages. For multi-turn cases, you can use another model to play the customer, given a persona and a goal (“You are an annoyed customer whose order CP-1123 arrived damaged. You want a refund. You only give your phone number if asked twice.”). The agent and the simulated user talk until the agent finishes or a turn limit is hit, then you grade as usual.

This works, with caveats. Simulated users are usually more cooperative and articulate than real ones. Use them to test conversation flow and policy adherence, and keep real transcripts in the dataset too.

Pitfalls to Avoid

1. Testing on vibes or 15 examples. Start with 100+ cases, tagged, and grow from production failures.
2. Asserting exact tool sequences. Brittle. Grade outcomes; use trajectory checks only for must/must-not rules.
3. Evaluating a different agent than production. Same prompt, same tool definitions, same loop. Only swap tool implementations.
4. Mocks without state. You can’t check what the agent did if your mocks don’t record it.
5. 1-10 judge scores. Use binary rubric items and calibrate against human labels.
6. One run per case. Non-determinism hides failures. Run k times and track pass^k.
7. Ignoring cost and steps in grading. A correct answer in 25 steps is a failing answer.
8. Synthetic-only datasets. They miss how real users write, especially code-mixed Indian languages and messy inputs.
9. Evals that nobody looks at. Post scorecards on PRs and gate merges, or they’ll rot.
10. Not feeding production failures back. The dataset should grow every week.

Quick Recap

• Four layers: tool tests, single-step decisions, end-to-end tasks, online production evals.
• Grade outcomes on a stateful fake backend; add trajectory checks only for rules that matter.
• Prefer code graders; use LLM judges with binary, calibrated rubrics.
• Run each case several times; pass^k is what customers experience.
• Gate merges in CI and feed production failures back into the dataset.

So far we’ve assumed everyone talking to your agent is a customer with a genuine problem. Next lesson, we drop that assumption. Securing agents: prompt injection, the lethal trifecta, and how to design so one bad email can’t empty your database.