Skip to content

Chatbot agent end-to-end test

MCPPortalIntermediate~30 min

A step-by-step guide that builds an agent, designs conversations, runs chats as test personas, and verifies that everything landed correctly in memory. Follow every step — the expected results at the end are specific and testable.

Path B no longer works

Path B drives the chats with the hadron_chatbot_* MCP tools, which were removed in hadron-server#1143. Use Path A, which runs the chats in an App's Chats tab, or run each persona as a scripted run in the agent's Tester (Chatbot Control → Overview → Open Tester →, see Test a chatbot with scripted personas). Path B's rewrite is tracked in hadron-docs#288.

Choose your test path

This tutorial supports two ways to run the chats. The agent and conversation design (Parts 1–3) are the same; only the run-and-verify steps differ.

Path A — Portal (App agent chat) Path B — Claude Code (MCP), unavailable
Plays the chatbot Your configured LLM (Anthropic / OpenAI / GLM) Claude Code's own model
Cost LLM API costs per turn Free
Iteration speed Start a new chat between edits Edit + test in same session
Fidelity Exact production model Claude Code, not the production model
Best for Final validation Iterating on prompts, stages, extraction

Use Path A. Path B doesn't work today: the hadron_chatbot_* MCP tools it drives were removed (see the warning above), and its tabs below are kept only until #288 rewrites them. For a fast loop while you iterate, run a persona as a scripted run in the agent's Tester instead.

The Path A and Path B sections are presented together using tabs in Parts 4 and 5. Switch tabs based on what you chose; the steps inside each tab are self-contained.

What you will build

By the end of this test you will have:

  • 1 agent ("Sage") with a system memory and a knowledge memory.
  • 2 conversations: onboarding (learn about the user) and strategy (help with a business problem).
  • 3 stages per conversation, each with an extractionSpec that pulls structured data from the chat.
  • 2+ personas (Alex Chen and Maria Santos), each with their own per-user memory created on first chat.
  • Chat transcripts stored in each persona's memory, including message history, extracted data and stage transitions.

What you need

  • A Hadron portal account at hadronmemory.com with admin access to an organization.
  • Claude Code installed with the Hadron MCP server connected (used in Part 3 for both paths to author conversation designs).
  • An API key for one of: Anthropic, OpenAI, GLM, or AWS Bedrock.
  • Owner or Admin role in the organization — only they can add the AI configuration.
  • A Workstation app (only needed for Part 3 / authoring conversations).
  • That's it for Path A — chats run in the browser, in an App you install Sage into below.
  • A Workstation app with the Hadron MCP proxy connected (see Connecting an MCP host).
  • You do not need an LLM API key — Claude Code plays both the user and the chatbot, no production LLM is called.

Part 1 — Create the agent (Portal)

  1. Go to your organization's Agents page and click New Chatbot (next to New Agent). That opens the Create Chatbot Agent wizard.
  2. Fill in:
    • Agent name: Sage
    • Description: AI mentor for small business owners
    • Visibility: Personal
  3. On the System Prompt step, enter: 
    You are Sage, a warm and practical AI mentor for small business
owners. You ask clarifying questions before giving advice. You are
direct but encouraging. When you don't know something, say so.
  4. On the Memories step:
    • System memory: auto-named Sage System (URN: sage-system) — leave as-is.
    • Knowledge memory: keep checked. Auto-named Sage Knowledge (URN: sage-knowledge).
  5. Review and create.
  6. On the agent detail page → Settings tab, check that surfaces includes mcp (chatbot agents default to ["api"] only). Add mcp if missing — Part 3 needs it.

Checkpoint: On the agent detail page you should see:

  • Agent "Sage" with system memory set.
  • Two memories listed: Sage Knowledge (read-write).
  • The Chatbot Control tab showing one conversation (setup) with one stage (onboard).

Configure the LLM provider on the agent, then install it as an App to chat in.

  1. On the Sage agent page, open the AI Providers tab → Add configuration:
    • Name: default.
    • Pick your provider (e.g. OpenAI).
    • Enter model (e.g. gpt-4o-mini).
    • Paste your API key.
    • Test — expect "✓ Works" with a short reply — then Save.
  2. Go to your organization's Agents page. On Sage's row, click Install.
  3. Set App type to Chatbot. It defaults to Workstation, whose agent chat stays blank, and changing the type later doesn't fix it (hadron-portal#922). Keep the suggested name, then click Install.

Checkpoint: The AI Providers tab lists default with your provider and model, and no key is shown. You land on the new Sage App, on its Chats tab, and Agent chat is in the row of chats at the top. See Configure your LLM provider if Test fails.

Wire your Workstation app to the Sage agent so Claude Code can reach it.

  1. Go to Apps → your Workstation app → Settings tab → Agents.
  2. Add the Sage agent with the Develop role (read/write).

No LLM provider configuration is needed for Path B — Claude Code plays the chatbot itself.

Checkpoint: In Claude Code, ask:

List my memories using hadron_list_memories.

You should see Sage System and Sage Knowledge.

Part 2 — Set the active memory (Claude Code)

Open Claude Code in the working directory of the Workstation app you configured. Set the active memory to the Sage system memory:

Set the active memory to Sage System.

All subsequent node operations in Part 3 will target this memory.

Part 3 — Design conversations (Claude Code + MCP)

Both paths author the conversations the same way: through Claude Code with the Hadron MCP tools.

3.1 — Verify the wizard scaffold

The wizard already created a setup conversation with one onboard stage. Confirm:

List all nodes with prefix conversations.

Expected — 3 nodes:

conversations             (system)
conversations:setup       (system) — data: { isSetup: true, stageOrder: ["onboard"] }
conversations:setup:onboard  (system) — data: { promptRef: "prompts:setup:onboard", extractionSpec: [...] }

Read the node at prompts:setup:onboard with raw: true.

You should see the wizard's default onboard prompt.

3.2 — Create the onboarding conversation

Create a conversation called onboarding in the Sage system memory with 3 stages: welcome, background, and goals. This is not a setup conversation (isSetup: false).

Stage order: welcome → background → goals.

welcome stage:

  • promptRef: prompts:onboarding:welcome
  • extractionSpec:
    • memory.name (string): "The user's full name"
    • memory.location (string): "City and state/country"
  • Prompt content: "Greet the user warmly. Ask for their name and where they're based. Use the respond tool."

background stage:

  • promptRef: prompts:onboarding:background
  • extractionSpec:
    • memory.business_type (string): "What kind of business they run"
    • memory.business_age (string): "How long they've been in business"
    • memory.team_size (string): "Number of employees or solo"
  • Prompt content: "Ask about their business: what do they do, how long have they been at it, team size? Summarize before moving on. Set next_stage to goals when done. Use the respond tool."

goals stage:

  • promptRef: prompts:onboarding:goals
  • extractionSpec:
    • memory.top_goal (string): "Their #1 business goal right now"
    • memory.biggest_challenge (string): "The main obstacle to that goal"
  • Prompt content: "Ask the user what their #1 business goal is right now, and the biggest challenge in the way. Reflect back what you heard. Tell them you'll switch to strategy mode. Set next_stage to null. Use the respond tool."

Checkpoint: Run hadron_list_nodes with prefix conversations:onboarding. You should see 4 nodes (parent + 3 stages). Read the parent's data and verify:

{ "isSetup": false, "stageOrder": ["welcome", "background", "goals"] }

Read each stage's data and verify each has a promptRef and an extractionSpec with the listed fields.

3.3 — Create the strategy conversation

Create a conversation called strategy with 3 stages: diagnose, options, action-plan. Not a setup conversation. Stage order: diagnose → options → action-plan.

diagnose stage:

  • promptRef: prompts:strategy:diagnose
  • extractionSpec:
    • memory.current_problem (string): "The specific problem being discussed"
    • memory.problem_severity (string): "How urgent: low, medium, high"
  • Prompt: "Ask the user to describe a specific business problem. Probe: when did it start, what have they tried, how urgent? Use the respond tool."

options stage:

  • promptRef: prompts:strategy:options
  • extractionSpec:
    • memory.options_discussed (string): "Comma-separated list of options"
    • memory.preferred_option (string): "Which option the user leaned toward"
  • Prompt: "Suggest 2-3 concrete options. For each, give a one-sentence pro and con. Ask which resonates most. Set next_stage to action-plan when the user picks. Use the respond tool."

action-plan stage:

  • promptRef: prompts:strategy:action-plan
  • extractionSpec:
    • memory.next_steps (string): "Agreed next steps, semicolon-separated"
    • memory.timeline (string): "When they'll start and any deadlines"
  • Prompt: "Turn the preferred option into 2-4 concrete next steps with a rough timeline. Confirm with the user. Set next_stage to null when agreed. Use the respond tool."

Checkpoint: hadron_list_nodes with prefix conversations:strategy should show 4 nodes. Verify stageOrder and each stage's extractionSpec.

3.4 — Verify all prompts exist

List all nodes with prefix prompts.

You should see at minimum:

prompts
prompts:setup
prompts:setup:onboard
prompts:onboarding
prompts:onboarding:welcome
prompts:onboarding:background
prompts:onboarding:goals
prompts:strategy
prompts:strategy:diagnose
prompts:strategy:options
prompts:strategy:action-plan
prompts:partials
prompts:partials:metadata-spec

Each leaf prompt should have non-empty content.

Part 4 — Run chats as test personas

This is where the two paths diverge. Pick the tab that matches the path you chose at the top.

Go back to the portal and open the Sage App you installed in Part 1: its Chats tab → Agent chat.

4.1 — Persona 1: Alex Chen

Click + New chat. Leave the scope on Private.

The agent should greet you with the welcome turn from onboarding.

Which conversation a new chat opens in isn't fixed

With both onboarding and strategy defined, a new chat from the portal opens in one of them, and which one isn't fixed; the portal can't choose. Check the first reply. If it's the strategy welcome, the onboarding rows below won't match that chat. See Building a chatbot agent.

Play the role of Alex Chen:

Turn You (as Alex) What to watch for
1 "Hi! I'm Alex Chen, based in Portland, Oregon." Agent should extract name + location.
2 "I run a specialty coffee shop. Been at it about 2 years. Just me and two part-time baristas." Agent should extract business_type, business_age, team_size. Stage transition from welcome → background (stage toast).
3 "My goal is to break even consistently — we're profitable some months but not others. Biggest challenge is foot traffic dropping in winter." Agent should extract top_goal + biggest_challenge. Stage transition to goals.
4 (Agent should wrap up onboarding and suggest switching to strategy.) The chat stays on goals: nothing moves it to strategy unless you drew a routing rule from goals to the strategy conversation. Otherwise start a new chat (the warning above applies).
5 "My winter foot traffic drops 40%. I've tried seasonal drinks but it didn't move the needle." Diagnose: current_problem + problem_severity.
6 (Respond to the agent's options.) Pick whichever sounds best. Extract options_discussed + preferred_option.
7 Confirm the action plan the agent proposes. Extract next_steps + timeline.

Stage transitions to watch for: welcome → background → goals in the onboarding chat. diagnose → options → action-plan in the strategy chat. Each transition shows a stage toast in the UI.

4.2 — Persona 2: Maria Santos

Click New chat again. Play Maria — a freelance graphic designer who wants to grow beyond solo work.

Turn You (as Maria) What to watch for
1 "Hey, I'm Maria Santos, I'm in Austin, Texas." Name + location extracted.
2 "I'm a freelance graphic designer. 4 years in, still solo — no employees." Business info extracted. Stage transition.
3 "Goal: double my revenue this year. Challenge: can't take on more clients without help, but hiring feels risky." Goal + challenge extracted.
4 Start a strategy chat. "I'm stuck doing everything myself — design, invoicing, client calls. 60-hour weeks, can't scale." Problem + severity.
5–7 Follow the agent through options and action plan. Full stage progression.

Maria is a different user only if you can log in as a second account. If not, both personas share the same per-user memory: the chats stay separate, but the facts don't. Every persona's facts are merged into that memory's one data node, so Maria's answers overwrite Alex's, and a field a later persona withholds keeps the earlier value. With one account, check each persona's facts (5.3) right after that persona's chat, before you start the next one.

4.3 — Partial data extraction test

Start one more chat. Withhold information deliberately:

  • Give your name; refuse to say where you're based ("I'd rather not say").
  • Give your business type; dodge the team size question.

The fields you shared should be updated. The ones you withheld are absent only on a fresh per-user memory (a separate account). On a memory shared with the earlier personas, they keep the earlier values, because facts are merged, never removed.

Claude Code will play the chatbot (generate responses from the compiled prompt) while you type the user's messages.

Key concepts before you start

  • hadron_chatbot_start starts a chat and returns the compiled prompt + tool schema. Claude Code reads the prompt and generates the welcome message.
  • hadron_chatbot_process sends the chatbot's response back to Hadron. Pass message, data (extracted fields), next_stage.
  • hadron_chatbot_send sends the user's reply and returns an updated prompt + history for the next chatbot response.
  • The chatbot's response must use the respond tool shape: Claude Code generates message, data, next_stage — not free-form text.
  • All data goes to memory automatically. Do not use hadron_create_node or hadron_update_node to save user data — the Chat API handles it.

4.1 — Persona 1: Alex Chen (onboarding → strategy)

Start the onboarding chat:

Start a chat with the Sage agent for user alex-chen, conversation onboarding. Use hadron_chatbot_start.

Read the returned systemMessage and tools. Generate a welcome message as Sage. Call hadron_chatbot_process with:

  • message: your welcome text
  • data: {} (no data to extract yet)
  • next_stage: null (stay in welcome stage)

Show me the welcome message and pause.

Write down the chat ID returned by hadron_chatbot_start (e.g. chats:20260507-abc12345-onboarding). You'll need it later.

Continue through the turns:

Turn Alex says Extract Transition to
1 "Hi! I'm Alex Chen, based in Portland, Oregon." memory.name, memory.location background
2 "I run a specialty coffee shop. 2 years. Me + 2 part-time baristas." memory.business_type, memory.business_age, memory.team_size goals
3 "Break even consistently. Winter foot traffic drops 40%." memory.top_goal, memory.biggest_challenge null (end)

Each turn: hadron_chatbot_send → generate response → hadron_chatbot_process. Watch for stageTransitioned: true and newStageName.

Then start a new chat for the strategy conversation:

Start a new chat with Sage for user alex-chen, conversation strategy. Generate the welcome, process it. Then continue.

Turn Alex says Extract Transition to
1 Winter foot traffic problem memory.current_problem, memory.problem_severity: "high" options
2 Responds to Sage's options memory.options_discussed, memory.preferred_option action-plan
3 Confirms the action plan memory.next_steps, memory.timeline null (end)

4.2 — Persona 2: Maria Santos (onboarding only)

Start a chat with Sage for user maria-santos, conversation onboarding. Same flow as Alex.

Turn Maria says Extract
Welcome (Sage greets) —
1 "Hey, I'm Maria Santos, Austin, Texas." memory.name, memory.location
2 "Freelance graphic designer. 4 years, solo." memory.business_type, memory.business_age, memory.team_size
3 "Double revenue. Can't scale without help; hiring feels risky." memory.top_goal, memory.biggest_challenge

4.3 — Partial data test

Start a chat with Sage for user partial-test, conversation onboarding.

  • Give your name: "I'm Pat." → extract memory.name: "Pat".
  • Refuse location: "I'd rather not say." → omit memory.location or send null.
  • Give business type: "I sell handmade candles." → extract memory.business_type.
  • Dodge team size: "It's complicated." → omit memory.team_size.

Part 5 — Verify the results

5.1 — Check the chats

Open the Sage App → Chats → Agent chat. All your chats should be in the sidebar, most recently active first. Each shows a title (auto-derived from the first user message), a Private badge, and when it was last active. Click each chat — the full message history should be intact.

5.2 — Check per-user memories

Per-user memories are private and scoped to the agent. To find them, use Claude Code:

List my memories. Set the active memory to my user memory for the Sage agent.

Or read chat data directly:

List all nodes with prefix chats in my user memory for the Sage agent.

You should see chat nodes per session, with a messages child holding the transcript. The extracted facts are not on the chat node: they are on the memory's data node (see 5.3).

5.3 — Check extracted data

Extracted facts are saved on the data node of the memory the chat lives in, each under the last segment of its field (memory.name is saved as name). That node holds the latest value of each fact, so if every persona used one account, do this check right after Alex's chat (see 4.2). For Alex Chen's onboarding chat, the node's data should contain:

{
  "name": "Alex Chen",
  "location": "Portland, Oregon",
  "business_type": "specialty coffee shop",
  "business_age": "2 years",
  "team_size": "3 (1 owner + 2 part-time baristas)",
  "top_goal": "break even consistently",
  "biggest_challenge": "winter foot traffic drop"
}

For the partial-data chat, withheld fields are absent only if it ran on a fresh per-user memory (a separate account). On a memory shared with the earlier personas, they keep the earlier values (see 4.3).

5.4 — Check stage transitions

Read the chat node's data:

  • Onboarding chats: progression through welcome → background → goals.
  • Strategy chats: diagnose → options → action-plan.

If the agent didn't transition when expected:

  1. Did the LLM include next_stage in the respond tool call?
  2. Does the stage's extractionSpec match what the LLM returned?
  3. Is stageOrder correct on the conversation node?

5.1 — Verify chat transcripts

List all nodes with prefix chats in the user memory for alex-chen.

You should see:

  • 2 chat nodes (one onboarding, one strategy).
  • Under each: a messages node with the full transcript.

Read the messages node for Alex's onboarding chat.

Verify the message history contains all turns (user + assistant).

5.2 — Verify extracted data

Read the data node of Alex's user memory for the Sage agent.

Look for:

{
  "name": "Alex Chen",
  "location": "Portland, Oregon",
  "business_type": "specialty coffee shop",
  "business_age": "2 years",
  "team_size": "3 (1 owner + 2 part-time)",
  "top_goal": "break even consistently",
  "biggest_challenge": "winter foot traffic drop"
}

The node accumulates: after Alex's strategy chat it also holds the 6 strategy fields. Maria's and the partial-data chat's facts land on the same node if they use the same memory, overwriting Alex's and leaving any field they withhold at its earlier value, so check each persona right after its chat (see Path A, 4.2).

5.3 — Verify stage transitions

Chat Expected transitions
Alex onboarding welcome → background → goals (2)
Alex strategy diagnose → options → action-plan (2)
Maria onboarding welcome → background → goals (2)
Partial test welcome → background (1, may stop early)

Expected end state

Item Count How to verify
Agent 1 (Sage) Agent detail page
System memory 1 (Sage System) hadron_list_memories
Knowledge memory 1 (Sage Knowledge) hadron_list_memories
Conversations 3 (setup + onboarding + strategy) hadron_list_nodes prefix conversations, depth 1
Stages 3 per real conversation (6 total) hadron_list_nodes per conversation
Prompts 6+ (one per stage) hadron_list_nodes prefix prompts
Chats 4+ (2 Alex, 1–2 Maria, 1 partial) Agent chat sidebar (Path A) / hadron_list_nodes prefix chats (Path B)
Per-user memories 1 per user Created when a user starts their first chat with the agent (Path A: + New chat)
Extracted data Varies per user the data node of each per-user memory
Stage transitions 2 per full conversation Stage toasts (Path A) / hadron_chatbot_process return values (Path B)

Test report checklist

  • Agent created with correct system + knowledge memory
  • mcp in agent surfaces
  • Conversations and stages match the spec above
  • All 6 stage prompts created with correct content
  • Each stage has extractionSpec in data
  • Path A: Sage App (type Chatbot) shows Agent chat, with a default AI configuration
  • Path B: Workstation app has Sage agent attached with Develop role
  • Alex Chen onboarding: 5 fields extracted, 2 stage transitions
  • Alex Chen strategy: 6 fields extracted, 2 stage transitions
  • Maria Santos onboarding: 5 fields extracted, 2 stage transitions
  • Partial data chat: present fields correct; withheld fields missing on a fresh memory, or keeping earlier values on a shared one
  • Chat transcripts intact (messages preserved on reload)
  • Per-user memory exists and contains chat nodes
  • Any bugs, unexpected behavior, or unclear steps noted

Troubleshooting

Path A: Agent chat is blank, or won't start — Check, in order: (1) the App was installed with App type: Chatbot — with any other type the Agent chat panel stays blank; (2) the chat doesn't read "Connect an LLM provider to enable agent chat" — if it does, add an AI configuration with a key (Part 1); (3) the agent's system memory is set and has at least one non-setup conversation (the wizard's setup conversation has isSetup: true and doesn't count).

Path B: hadron_list_memories shows nothing — Your app doesn't have the agent attached, or the agent doesn't have mcp in its surfaces.

Path B: hadron_chatbot_start says "agent has no system memory" — The agent's systemMemoryId isn't set. Check agent Settings.

Path B: hadron_chatbot_start says "no non-setup conversation found" — You haven't created onboarding or strategy yet, or one of them has isSetup: true. Only the wizard's setup conversation should have that flag.

Stage didn't transition — The LLM must include next_stage in the respond tool call. If the prompt is vague ("move on when ready"), the LLM may not emit the field. Make the instruction explicit: "set next_stage to <name> when done."

Extraction data is missing — Check two things: (1) the stage's extractionSpec lists the fields, (2) the LLM's response (or your hadron_chatbot_process call in Path B) includes those fields in data. The Chat API stores what's sent — it doesn't infer from the message text.

Path A: "No AI config saved yet" on test — Save the config before testing. The Test button can also test unsaved values — enter provider, model, key, then click Test before saving.

Path A: MCP tools don't see the agent — The agent needs mcp in its surfaces. Chatbot agents default to ["api"] only. Add mcp in Settings.

Path B: Claude Code generates weird responses — Claude Code is playing the chatbot, not being one. If responses don't match the persona, just tell it what to say. The point is testing the Chat API flow (extraction, transitions), not the quality of Claude's roleplay.

Data model reference (Path B)

Useful for verifying state via MCP:

Concept Where it's stored How to inspect
Conversations, stages System memory, conversations:* hadron_list_nodes / hadron_get_node
Prompts System memory, prompts:* hadron_get_node with raw: true
Extraction specs data.extractionSpec on stage nodes hadron_get_node on the stage
Chat sessions Per-user memory, chats:* hadron_list_nodes in user memory
Message history Per-user memory, chats:<id>:messages hadron_get_node
Extracted user data the memory's data node hadron_get_node
Knowledge content Knowledge memory, any structure hadron_list_nodes / hadron_get_node

The content field is markdown text; the data field is structured JSON. Don't mix them up.

Automated testing with personas

After running through this manually, script it: turn Alex and Maria into scripted personas in the portal Tester, and re-run them whenever you change the chatbot. Each run talks to the same model your real users get, and produces a pass/fail report without you typing each turn. See Test a chatbot with scripted personas.