Build an agent eval suite in five days.

Evals feel like overhead when you're racing to ship a pilot. The team is optimistic, the demo works, and writing forty test cases feels like it's slowing down momentum. Then, six weeks in, someone tweaks the system prompt to fix a complaint, and a different, previously-working case silently breaks. Nobody notices for two weeks because there's no regression check running. This is the single most common gap behind stalled pilots, more common than model choice or infrastructure problems.

The cost compounds. Every fix becomes a coin flip. Confidence in the system erodes. Stakeholders start asking for a human to double check everything, which quietly kills the ROI case the agent was supposed to deliver.

The common advice is "use an LLM-as-judge to score everything, it scales better than human review." That's true as far as it goes, but it skips the harder and more important step: deciding what to grade and against what standard. An LLM judge without a concrete rubric just replaces one form of vibes with a more expensive form of vibes. Teams that adopt LLM-as-judge without first defining pass/fail criteria from real cases end up with a suite that gives a false sense of rigor while still missing regressions that matter to actual users.

The order matters: build the case set from real production behavior first, define deterministic checks wherever possible, and only reach for an LLM judge for the genuinely subjective slice, like tone or completeness, where deterministic checks don't apply.

Pull 30 to 50 real interactions from production logs, support tickets, or the manual process the agent replaces. Prioritize:

• Common cases that represent the bulk of real traffic.
• Known failure cases that have already caused a complaint or an incident.
• Edge cases that are rare but high stakes, like a refund request over a dollar threshold.

Synthetic cases written by a team member imagining what users might ask are a distant second choice. They tend to be cleaner and easier than real traffic, which makes your suite look healthier than the system actually is.

For each case, write down what "correct" means as concretely as possible. This is the step teams rush, and it's the step that makes or breaks the suite. Three kinds of expected outcomes cover most cases:

1. Exact match. A specific field, number, or action that must be correct, like a calculated refund amount.
2. Structural check. The output must contain certain elements or call a specific tool, regardless of exact wording.
3. Rubric-graded. Subjective quality, like tone or completeness, scored against a written rubric with 3 to 5 concrete criteria.

Turn each expected outcome into code. Exact match and structural checks should be deterministic assertions, no model involved. For rubric-graded cases, write a grading prompt that gives a second model the case, the output, and the rubric, and asks for a pass/fail plus a one-sentence justification. Log every grading decision so a human can audit the judge's calls later.

Run the current system against the full suite and record the pass rate honestly, even if it's low. This number is your baseline, not a target. Set the deploy gate at the baseline. If it's 74%, that's the bar until a deliberate improvement moves it up. Never lower the bar to make a deploy easier, that defeats the entire purpose of the suite.

Here's the checklist we use to confirm the suite is actually load-bearing, not decorative:

1. Suite runs automatically on every pull request touching prompts, tools, or model config.
2. A failing run blocks merge, no manual override without a written justification.
3. Suite also runs nightly against a fresh sample of production traffic to catch drift.
4. Pass rate and failure categories are visible on a dashboard the workflow owner checks weekly.
5. New failure cases discovered in production get added to the suite within a week, not "eventually."

This is the same discipline we build into every MCP server engagement and every Claude implementation project, because a tool surface without a regression suite is a tool surface you can't safely change.

An eval suite that never grows becomes stale within a quarter. Every production incident, every support escalation tied to the agent, and every "that's not quite right" from a stakeholder should become a new case within days, not get filed away and forgotten. Suites that grow this way become genuinely predictive of production quality. Suites that get built once and never touched again become theater. If you're not sure whether your current suite still reflects real usage, that's a fast thing for an outside team to check in a technical audit, alongside the rest of your production readiness gaps covered in our production readiness guide.