Testing your agent

What a test is

A test is a short scripted conversation plus a description of what should happen. You'll find tests under Build → Tests for each agent. Every test has:

• What the customer says — one message, or up to eight sent in order for a multi-step conversation.
• What should happen — a plain-language expectation, for example "The agent explains returns are accepted within 30 days and asks for the order number." An AI evaluator reads the conversation and decides whether it was met, and tells you why when it wasn't.
• Expected actions (optional) — tick the actions the agent must call. The test fails unless the agent calls exactly those actions. Tick none to require that it calls no action at all.

When to run tests

Run them any time, and always after you change the agent's instructions, model, data sources, procedures or actions. Run all runs every test; the play button on a row runs just that one. Each test keeps its last result, and the run history shows how the agent did over time.

Actions in tests

Tests never really run your actions, so nothing is created, sent or charged. For each action you choose what it returns:

• Mock: succeeds — the default. The agent is told the action worked.
• Mock: return my data — paste the JSON (or text) the action should return, such as {"status": "shipped", "eta": "2 days"}, to check how the agent uses it.
• Mock: return an error — for example Order not found, to check the agent handles failure gracefully.
• Run live — only for read-only store lookups (Shopify product search, order lookup and cart). Actions that change data, and custom actions, can't run live.

Suggested tests

Suggest tests drafts a handful from your agent's own sources and actions, including one question your sources don't cover, to check the agent admits it doesn't know. Drafts aren't saved until you add them, and you can edit each one first.

Test credits

Every paid plan includes free test credits each month: 200 on Hobby, 500 on Standard and 1,000 on Pro and Enterprise. Each agent reply in a test uses credits at the same rate as a normal reply for that model. When the free test credits run out, and on the Free plan, tests use your message credits instead. Test credits reset with your monthly credit period.

Good tests

• Write expectations someone could check from the reply alone: a fact, a question the agent must ask, an action it must take.
• Cover your most common questions, each action, and at least one thing the agent should refuse or not know.
• When a test fails after a change, open it to read the conversation and the reason before changing the test.