Skip to content

Portal Agent Chat — Manual Test Checklist

Portal onlyBeginner~10 min

End-to-end smoke test for agent chat in the portal, which runs inside an App: App → Chats → Agent chat. Exercises the full loop: AI config → conversation design → chat turns → stage transitions → chat management.

This is a manual checklist, not an automated suite. Run through it after any change to agent chat, the Chat API, or the conversation engine. For a scripted, repeatable run against the agent itself, use the agent's Tester instead (Chatbot Control → Overview → Open Tester →) — see Test a chatbot with scripted personas.

Prerequisites

  • An organization you are an Owner or Admin of. Only they can manage AI configurations.
  • An agent with:
  • A chatbot system memory attached (role read-write).
  • At least one non-setup conversation defined under conversations:* with a real starting stage and prompt.
  • The agent installed as an App with App type: Chatbot (org Agents → Install). With any other type the Agent chat panel stays blank, and changing the type afterwards doesn't fix it (hadron-portal#922).
  • An API key for one of: Anthropic, OpenAI, GLM, or AWS Bedrock.

1. AI config

AI configurations are managed on the App's Settings tab, the agent's AI Providers tab, or the organization's Integrations tab. A chat asks for a configuration by name (default unless one is picked) and uses the first enabled configuration with that name, walking App → agent → organization → host — see Where a configuration lives.

  1. On one of those surfaces, confirm it reads "No AI configurations yet."
  2. Add configuration → name it default, pick a provider, enter a model, paste an API key.
  3. Test, before saving → expect ✓ Works with a reply from the model. Test uses the values in the form, so this works on every surface.
  4. Replace the key with a bad one → Test → expect a clear provider error message, not a stack trace. Put the good key back → Save. The key is not shown afterwards.
  5. On the agent or App surface only: Edit the saved configuration and Test with the key field left blank → the server tests the stored key. On the organization's Integrations tab this isn't supported and reads "Enter a key to test this configuration."
  6. Delete the configuration, or turn off Enabled. (Don't just clear its key: a keyless, enabled configuration still wins its name, so no fallback takes over.) If it was the only usable configuration in the chain — none on the App, agent or organization, and none on the host — the App's Agent chat shows "Connect an LLM provider to enable agent chat". Otherwise the chat keeps working on the next usable configuration: check that it does. The chat's AI config picker may still show the same name, because a configuration with that name at a lower level (say, the organization's default) now takes its place.
  7. Restore a working configuration (add it again with its key, or turn it back on) and confirm Agent chat works before continuing: the sections below need it.

2. Agent chat availability

In the App's Chats tab, the Agent chat entry and a working chat behind it depend on three things. Check each:

  • A linked agent. Without one, Agent chat isn't in the row of chats.
  • App type Chatbot (the App's surfaces include chat). Without it, Agent chat is listed but its panel renders nothing.
  • A usable AI config (section 1). Without one, it shows "Connect an LLM provider to enable agent chat". If you can't list the App's configs (you aren't an App member), the portal doesn't block the chat on this check.

3. Start a new chat

  1. Open the App's Chats tab → Agent chat. The sidebar shows "No chats yet. Start a new one above."
  2. Click + New chat.
  3. Expect:
  4. A new chat row appears in the sidebar, selected.
  5. The assistant greets you with the conversation's welcome prompt (served by /api/agent-chat/start).
  6. No user bubble precedes the welcome.
  7. Stage toast shows the initial stage, if the conversation defines one.

4. Send messages

  1. Type a message → Enter (or click Send).
  2. User bubble appears immediately (optimistic).
  3. Assistant reply streams in once the LLM responds.
  4. On error (kill network, wrong key), the optimistic user bubble rolls back and turnError is shown below the input.
  5. Shift+Enter inserts a newline; Enter alone submits.

5. Stage transitions

  1. Drive the conversation through a prompt designed to transition stages (e.g., answer the question the stage is gating on).
  2. Expect a stage toast when next_stage is returned.
  3. Subsequent assistant messages should reflect the new stage's prompt.

6. Resume an existing chat

  1. Start a second chat so there are at least two in the sidebar.
  2. Reload the page.
  3. Open Chats → Agent chat → the sidebar lists both chats, sorted by lastMessageAt (most recent first).
  4. Click an older chat → messages load via /api/agent-chat/resume.
  5. Send another message → lastMessageAt updates and the row jumps to the top of the sidebar.

7. Rename + delete

  1. Hover a chat row → click rename → edit title → Enter to commit, Escape to cancel.
  2. Click delete → confirmation modal → confirm → chat disappears from sidebar, underlying Chat is soft-deleted.
  3. Cancel on the modal keeps the chat.

8. Per-user scoping

  1. As user A, create a Private chat in the App's Agent chat.
  2. Log in as user B (a member of the same App) → Chats → Agent chat.
  3. Expect an empty sidebar — user A's chats must not leak.
  4. Starting a chat as B provisions a fresh per-user memory (userMemoryOfAgentId) and does not touch A's memory.

9. Multi-provider sanity

Run sections 3–5 once per provider you support:

  • Anthropic (claude-*)
  • OpenAI (gpt-*)
  • GLM (glm-*)
  • AWS Bedrock

Tool-call forcing must succeed for every provider — the LLM should always call respond({ message, data, next_stage }), never return plain text. If a provider returns text, treat it as a bug in the portal's agent-chat turn handling.

10. Welcome latency

The welcome message is served synchronously by /api/agent-chat/start, which runs a full LLM turn before returning. Expect a 1–3s delay on New chat. This is known; streaming is not yet implemented.

Known non-goals (do not test)

  • Password-based encryption of API keys (deferred; JWT only for now).
  • Streaming responses.
  • Billing / usage metering for agent chat turns.