You can't ship agents on vibes — evals for agentic systems

Here’s the failure mode nobody warns you about. You tweak an agent’s system prompt to fix one thing, it looks fine in a quick test, you ship it — and three other behaviors you’d carefully tuned quietly regress. Nobody notices until a user hits one. Agents are non-deterministic and their behavior is smeared across a prompt, a model, a temperature, and a tool list. You cannot eyeball that.

The fix is old news in software and overdue in agents: a test suite. In this world it’s called an eval harness — a golden set of inputs and expected properties, run against every agent (and every workflow) on every change.

An eval is an input plus checks, run against a target

The unit is small. An eval names a target (an agent or a workflow), an input to send it, and a list of checks the output must satisfy. Running it is mechanical:

flowchart LR
  E["Eval: input + checks"] --> R["Run the target
(agent or workflow)"]
  R --> O["Output + cost + latency"]
  O --> C{"Apply checks"}
  C -->|all pass| P["✅ pass"]
  C -->|any fail| F["❌ fail + which check"]
  P --> L["Log the run"]
  F --> L

A run captures more than the text: the output, the token cost, the latency, and any error. That lets your checks assert on behavior and budget — “answers correctly” and “under $0.02” and “under 3s.”

Two kinds of checks: deterministic and judged

Most properties are cheap to check with code. A few need judgment, so you borrow another model.

Check	Asserts	Cost
contains / not_contains	a string is (n)present	free
regex	pattern matches	free
equals / min_len	exact output / minimum length	free
json_valid	the output parses as JSON	free
max_cost / max_ms	budget and latency ceilings	free
no_error	the run didn’t throw	free
judge	an LLM grades against a rubric	one model call

Reach for deterministic checks first — they’re free, fast, and unambiguous. json_valid alone catches a huge class of “the model wrapped it in prose” regressions. Use the judge only for the genuinely fuzzy properties: “is this a polite refusal?”, “does it cite a source?”

The judge: a strict, independent grader

LLM-as-judge is a second model, independent of the agent under test, handed a rubric and the answer and asked for a single verdict. Keep it strict and structured:

You are a strict evaluator. Given a RUBRIC and an ANSWER, decide whether the
answer satisfies the rubric. Reply with a single word PASS or FAIL, then a
colon and one short sentence.

Temperature 0, a short token budget, PASS/FAIL first so parsing is trivial. The independence matters — never let an agent grade its own homework with its own prompt.

Run-on-change: the regression net

An eval suite you have to remember to run is an eval suite you won’t run. The leverage is automatic regression: when someone edits an agent’s behavior-affecting fields — its prompt, model, tools, temperature — fire its evals in the background and flag any that flipped to failing.

flowchart LR
  Edit["Edit agent
(prompt / model / tools)"] --> Q{"has evals?"}
  Q -->|yes| Run["run its golden set"]
  Q -->|no| Skip["skip"]
  Run --> Diff["report pass/fail
+ Telegram on regressions"]

Now the golden set is a safety net stretched under every change, not a chore. A red result the moment you save is worth ten green results you ran a week ago.

Grade workflows, not just agents

Single-agent evals are table stakes. The interesting failures live in multi-agent workflows — an orchestrator delegating to specialists, or a node graph fanning out and back in. Those need the same treatment: run the whole workflow on a golden input, then check the final synthesized output.

The trick is to make the target polymorphic. An eval’s target_kind is agent or workflow; the harness runs the right executor and applies the identical checks to whatever comes out. If your eval runner only knows how to run single agents, your most complex — and most fragile — systems are the ones going ungraded.

The golden set is a living spec

The quiet payoff: your evals become the executable definition of what each agent is for. “Refuses off-topic requests.” “Always returns valid JSON.” “Cites at least one source.” “Costs under two cents.” Every check is a promise, and the suite is the promise you can run. New behavior starts as a new eval; fixing a bug starts by writing the eval that would have caught it.

You don’t need a platform to start — you need three columns (input, check, target) and the discipline to run them on every change. Agents earn trust the same way code does: not by looking right, but by passing.