Skip to content

Testing a chatbot with personas

A chatbot agent in Hadron is a design, not a prompt: conversations made of stages, edges that route between them, goals that track what the user still owes you, and extraction specs that pull structured data out of free text. The conversation-routing reference has the machinery.

Designs like that have a testing problem that prompts do not.

Why poking at it by hand does not work

The obvious way to check a conversation is to open a chat and type. It fails for three reasons, and only the first is obvious.

The model is non-deterministic. One good run tells you the flow can work, not that it does. Two runs of the same input can route differently, and the one you happened to try is the one you will remember.

You are the wrong user. You know what the bot wants, so you supply it — in the right order, in the phrasing the extraction spec expects. Real users answer a different question than the one asked, volunteer their budget before their name, and change the subject. Your manual run exercises the path you designed and never the paths you did not.

It does not survive the next edit. A conversation design is a graph, so adding an edge can reroute traffic that used to work. You cannot notice that by re-testing the thing you just changed, which is the thing you will re-test.

A test persona answers all three: a scripted user, defined once, that can be run again on demand.

What a persona is

A name, a short background, an opening message, an ordered list of follow-ups, and — optionally — the conversations it is expected to visit.

That last field is what turns a transcript into a test. Without it you have a replay to read; with it you have an assertion.

Personas live in the agent's own system memory, so they travel with the agent rather than with whoever wrote them. A design and its tests stay together.

What pass actually means

pass is true when every expected conversation was visited. That is the entire definition, and it is narrower than it sounds.

It does not mean the answers were good, the tone was right, or the user would have been satisfied. It means routing went where you said it should. A chatbot can be unhelpful for eight turns and pass. And a conversation that gave a genuinely excellent answer fails if it never reached the second conversation you expected — the quality of what happened is not what is being measured.

Nor is the route: reaching every expected conversation by an unexpected path still passes, because the assertion is over the set visited, not the order or the journey. routeHistory is where you find out how it got there.

That narrowness is deliberate — routing is the part that is mechanically checkable, and it is also the part that breaks silently when a design changes. Quality still needs a human reading the transcript. Treat pass as "the graph behaved", not "the bot was good."

The rest of the report is the diagnosis rather than the verdict:

  • issues — missing conversations, off-track turns, dead ends.
  • visitedConversations — where it actually went, which is the thing to compare against where you thought it would go.
  • routeHistory — every stage entered and exited and every edge traversed. When a persona fails, this says where it diverged, which is usually one edge earlier than the symptom.
  • goalStack — what was completed, still active, or abandoned. A flow that visits the right conversations while abandoning its goals is passing for the wrong reason.

Two modes, two different questions

Dry run Live test
Who plays the chatbot your coding agent's own model the agent's configured LLM
Cost free one LLM call per turn
Feel interactive — you watch each turn automatic — all turns at once
Answers is the design right? does it work with the model that will ship?

This is not a cheap version and a real version. They answer different questions, and the difference is which model is on the other side.

A dry run substitutes a different model for your agent's. That makes it free and fast, and it means a passing dry run proves the graph is right — the stages connect, the edges fire, the goals resolve. It cannot tell you whether your configured model will follow the prompts you wrote, because it is not that model.

So the sequence is: iterate with dry runs until the design stops surprising you, then live-test to find out whether the model you are shipping agrees. Dry runs are for design defects; live tests are for model defects. Reaching for the expensive one first mostly buys slow feedback on problems the cheap one would have found.

A useful set of personas

Three or four, each deliberately unlike the others:

  • The happy path — answers everything, in order, as designed. Proves the flow works at all. Write it first and stop being reassured by it.
  • The withholder — refuses a question, dodges another. Proves the design handles missing data rather than assuming it, and that conditional edges fire when a value is absent.
  • The off-topic one — changes the subject mid-flow. Proves off-track detection and re-routing, which is the machinery that never runs on your own manual test because you would not do that.
  • The traveller — follow-ups that naturally span two conversations. Proves routing across a boundary, which is where edge-based designs break.

Name them for their scenario rather than numbering them. "Maria — freelance designer, partial data" tells you what a failure means; "test-user-3" makes you re-read the definition every time.

The part that pays for itself

Personas are cheap to write once, and a dry run costs nothing to repeat — which makes them a regression suite rather than a one-time validation. (A live test bills per turn, so the loop you run constantly is the dry one; live tests are for before you publish.) The value shows up on the change you did not think was risky: adding one edge, reordering two stages, tightening a prompt.

That is also the argument for running them again after every change rather than only before publishing. A conversation design has no compiler — hadron_validate checks structural and index health, not whether your graph still routes — so nothing fails when an edge stops being reachable. Short of someone re-reading the design, the personas are the only thing that notices.