Skip to content

Fixtures, assertions and verdicts in the portal Tester

When you replay a saved test in the portal Tester, the screen answers one question: did this agent's reply change, and does it still say what you expected? It answers it with a handful of words (pass, fail, not evaluated, still passing, newly failing, no baseline) that look like one scale and are not. This page explains what each one claims, so that a red or grey badge sends you to the right place.

The words are defined in the portal, not in the platform's shared rules, so the statements below are pinned to the portal's source at commit 147f5eda (src/lib/agent-tester/verdict-types.ts, src/lib/server/agent-tester/, src/routes/api/agent-chat/fixtures/). The Tester is changing; if the screen disagrees with this page, the screen is current.

This page is about the fixture half of the Tester: one message, one saved expectation. Running a scripted user through whole conversations is a different mechanism with a different meaning of pass; see Testing a chatbot with personas, and the difference below.

A fixture is one message and what you expect back

In the Tester you send one message to your agent and read the reply, the conversation and stage it landed in, and, if you asked for one, a verdict. Save as fixture keeps that as a named test.

A fixture holds:

  • a label (up to 80 characters), such as welcome-greeting;
  • the message you send, and optionally the name of the persona it is sent as (a label only; see below);
  • an optional assertion: the check to apply to the reply;
  • if it was saved from a run, a baseline: the reply, the route (the conversation and the stage) and the verdict that run produced.

Fixtures belong to the agent. They are stored with it, not in your browser, so the list is there when you come back. This version of a fixture holds a single message and its reply, not a multi-turn conversation.

An assertion is the check you apply to the reply

There are three kinds:

Kind Passes when
Literal contains The reply includes your exact text.
Regex The reply matches your pattern. The match runs in a time-limited worker, so a pathological pattern cannot hang the portal.
LLM judge A model, given your criterion in plain words, scores the reply from 0 to 1, and the score is at least your threshold.

A fixture with no assertion is a probe: it exists to capture a reply and later notice when the reply changes. A probe is recorded as a pass by convention, because nothing was checked and nothing could fail.

A passing assertion says the reply satisfied that one check on that one run. It does not say the reply was good, and a model judge in particular is a second model's opinion against your wording, not a measurement.

A verdict says what happened to the check, not just what the reply was

Every assertion run ends in one of three verdicts:

Verdict It means It is a claim about the reply?
Pass The check ran to the end and the reply satisfied it. Yes
Fail The check ran to the end and the reply did not satisfy it. Yes
Not evaluated The check did not run to the end. No

Not evaluated is the one that matters. It happens when a regex pattern does not compile, when a regex does not finish inside its time budget, when the worker that runs it fails to start, or when the judge cannot be reached or does not return a score. In each case nothing was compared to the reply, so the portal does not call it a failure. Reading it as one would send you to debug an agent that was never measured.

So a red Fail means look at the reply, and a grey Not evaluated means run it again, or look at the assertion. The detail line under the badge says which.

An errored run is a different thing again

A fixture can also error: the replay itself did not complete. The agent call failed, or the conversation ended during its welcome turn so there was no reply to check. An errored fixture has no verdict at all, because no reply exists. When you run all fixtures, an error in one does not stop the rest; the summary counts it separately.

The difference from not evaluated: there, the agent replied and the check did not run; here, there was no reply.

A baseline is what you decided was right

When you save a fixture from a run, that run becomes its baseline: the reply, the route and the verdict at that moment. Later replays compare against it.

Only pass and fail can be stored as a baseline verdict. Not evaluated is not an expectation, so it has no stored form, and the Tester will not accept a not-evaluated run as a new baseline: the Accept new baseline button is not offered for it, and a request to do so is refused with a reason rather than silently ignored. Run the fixture again instead.

A replay never overwrites the baseline by itself. You replace it deliberately with Accept new baseline, which is what keeps drift detection meaningful.

Drift is how a replay compares to its baseline

A replay's drift report compares three things separately:

  • Reply: the same text as the baseline, or different. This is an exact comparison, so a reworded sentence counts as different even when it means the same.
  • Route: the same conversation and stage as the baseline, or different.
  • Verdict: how the new verdict relates to the baseline verdict.

The verdict comparison has six values:

Shown as Means
Still passing / Still failing Baseline and replay agree.
Newly passing / Newly failing The verdict changed since the baseline.
No baseline The fixture was never run with its result saved, so there is nothing to compare to.
Not evaluated The replay produced no verdict, so whether it drifted is unknown.

Not evaluated is not a direction of change. A fixture that was green and whose worker then failed reads Not evaluated, never Newly failing: the replay produced no result to have regressed.

The current verdict and the drift label are two questions

Is there a verdict this time? and how does it compare to the baseline? are separate, and the drift label can only carry one answer. When a fixture has no baseline, the drift label reads No baseline even if the assertion could not be evaluated this time. Both are true; the label shows the first.

This is why Accept new baseline follows the current verdict and not the drift label. A fixture with no baseline whose assertion did not evaluate shows No baseline, and still cannot be accepted, because there is no verdict to accept.

The reply and route comparisons stay meaningful when the verdict is not evaluated: the chat turn ran, and only the check did not.

The run summary counts, and the cost is an estimate

After a run, a summary line shows how many fixtures ran and the tallies: passed, failed, not evaluated and errored. Zero tallies are hidden, except passed. Not evaluated and errored are grey on purpose, because neither says anything about your agent's replies.

Next to the tallies is an estimated cost, in US dollars. It is computed before the run from a flat per-call guess (at the pinned commit, about one cent per agent turn, two turns per fixture, and half a cent per judge call) and is there to stop an expensive run-all by accident. When the estimate for a run-all passes the cap, currently ten cents, the portal asks you to confirm before it spends anything, and the summary notes that it ran above the cap because you did. The figure is a rough guide to order of magnitude. It is not what any provider will bill you.

An example

One fixture, refund-policy, sends "Can I get my money back?" with the assertion regex 30 days. It was saved when the reply mentioned 30 days, so its baseline is a pass.

Replay Verdict Drift label What to do
Reply mentions 30 days, worded differently Pass Still passing (reply: different) Probably nothing; read the new wording.
Reply no longer mentions 30 days Fail Newly failing Look at the reply and the design behind it.
The regex worker failed to start Not evaluated Not evaluated Run it again; the reply was not judged.
A fixture saved without a run, and the worker fails Not evaluated No baseline Run it again; nothing can be accepted yet.

Not a persona run

Test personas are a separate mechanism. A persona's pass means every conversation it was expected to visit was visited: a claim about routing, taken over a whole scripted conversation. A fixture's verdict is a claim about one reply against one assertion.

The two use similar words and answer different questions, so do not read one in the terms of the other. A passing fixture says nothing about whether the right conversations were reached. A passing persona says nothing about what any single reply said. A fixture does record the route it landed in, and drift reports a change in it, but that is a comparison with the baseline, not a coverage claim.