Skip to content

Test a chatbot with scripted personas

Portal onlyIntermediate~15 min

Run a scripted user through your chatbot agent and get a pass/fail report of where it went: every turn of the conversation, each stage it moved to, and whether it reached the conversations you expected. You do it in the portal, and it takes one form and one button.

Each run uses your agent's own model, so it is billed by your LLM provider per turn (see what a run costs), and it creates a real chat. What a persona is and what pass does and does not prove are covered in Testing a chatbot with personas.

Before you start

  • A chatbot agent with at least one conversation. See Building a chatbot agent.
  • Membership of the organization that owns the agent. The Tester only opens agents owned by an organization you belong to. An agent you reach only through an App or a subscription, or one you own personally with no organization, reports "Agent not found or no longer available".
  • An LLM provider configured on the agent. See Configure an LLM provider. The Tester needs one to run.
  • A script: an opening message, the follow-ups in order, and which conversations the user should end up in. Writing good personas below has four worth starting with.

Run a persona

  1. Open the agent in the portal and go to Chatbot Control → Overview.
  2. Click Open Tester →.
  3. Expand Scripted persona (multi-turn).
  4. Fill in the form:
    • Persona name — who this user is, e.g. Maria — freelance designer, partial data. Required.
    • Start conversation (optional) — the conversation's bare name, e.g. onboarding. Leave it empty to start where a new chat normally starts.
    • Opening message — the first thing the user says. Required.
    • Follow-up messages — one per box, in order. Add follow-up adds a box.
    • Expected conversations (optional) — bare names, comma-separated, e.g. onboarding, strategy. This is what turns a transcript into a pass or a fail.
  5. Click Run script.

The agent greets the persona, then answers each message in turn. The run stops at the end of your script, or earlier if the chatbot ends the chat.

Use bare conversation names, not conversations:…

The form's placeholders suggest conversations:setup, but a chat is started by the bare name, and the report lists conversations by bare name. A prefixed Start conversation is refused as not found, and a prefixed Expected conversation never matches, so the run reports Fail.

Read the report

  • Pass or Fail. A run passes when every expected conversation was visited, in any order. If you set no expectations, it passes.
  • Issues name each expected conversation that was missed, and count the turns the chatbot marked off-track. Off-track turns are reported, but they do not fail the run.
  • Conversations visited is where the persona actually went. Compare it with where you thought it would go.
  • The transcript shows each User and Assistant turn. A turn that moved to a new stage is marked with the stage and its conversation, and an off-track turn is marked off-track.

When a run fails, read the transcript from the top. The turn where it went wrong is usually one earlier than the turn where you notice it.

What a run costs, and what it leaves behind

Every turn is one call to your agent's LLM. Hadron does not bill per turn: you pay the provider directly through the key on the agent's AI configuration. A run costs roughly the number of turns times the cost of a turn, which depends on the provider, the model, the prompt size and the length of the history. As a rough order of magnitude for a typical chatbot turn (1–3K input tokens, 200–500 output tokens):

Provider / model Per-turn cost (rough) A 6-turn persona
OpenAI gpt-4o-mini ~$0.001 – $0.003 ~$0.01
OpenAI gpt-4o ~$0.01 – $0.03 ~$0.10
Anthropic Haiku ~$0.001 – $0.005 ~$0.02
Anthropic Sonnet ~$0.01 – $0.03 ~$0.10
GLM (Z.AI) typically lower than OpenAI ~$0.01

These are ballpark figures. Check your provider's pricing page for current rates, and its billing dashboard for what you actually spent:

A run also leaves things behind, and has limits:

  • It is a real chat. It appears in your chat history under the agent, and facts the chatbot extracts can be written to your personal memory.
  • The persona is not saved. The form forgets it when you leave the page. Keep your scripts somewhere you can paste them from.
  • Up to 50 follow-up messages and 100,000 bytes of message text per run.
  • Up to 10 runs a minute. After that, wait a minute before running again.

Writing good personas

Cover the happy path

Start with a persona that follows the expected flow perfectly: it gives its name, answers the questions, and provides all the data the extraction spec expects. This proves the basic flow works.

Test partial data

Write a persona that withholds information: it refuses to give its location, or dodges the team-size question. This shows whether the chatbot handles missing data gracefully.

Test off-topic messages

Write a persona whose follow-ups go off-topic: "Actually, can I ask about billing instead?" The report marks the turns the chatbot judged off-track.

Test conversation transitions

Write a persona whose follow-ups naturally span two conversations, e.g. onboarding then strategy, and list both under Expected conversations. This checks that routing between conversations works end to end.

Name personas clearly

Use a real-sounding name with a specific scenario. "Maria Santos — freelance designer, partial data" is much more useful than "test-user-3".

A test routine

  1. Write 3–5 personas covering the happy path, partial data, off-topic, and cross-conversation flows.
  2. Run each one in the Tester. Fix what fails in the conversation design: prompts, stages, edges, goals.
  3. Re-run after each change. Personas are what catch the edit that broke a route you were not looking at.
  4. Save and publish a revision once they all pass. In the agent's Conversation Editor, click Save revision. Then open Revisions and click Publish on the one you saved.

Keep the set small. Every run bills per turn, so a handful of personas you actually re-run beats a large set you skip.