Test an AI Chatbot UI With Plain-English AI Tests

If you have ever tried to write a Playwright spec against a chat product, you already know the pain. To test an AI chatbot UI you have to send a prompt, wait for tokens to stream in one character at a time, and then assert something about a reply that is different every single run. Selectors drift as the design team ships. The "typing" indicator appears and disappears on its own schedule. And the actual answer, the thing you most want to check, is generated by a model that will phrase it a new way tomorrow. Traditional locator-and-toHaveText testing was built for deterministic forms, not for a streaming, probabilistic conversation surface.

This guide walks through a different approach: describing what a chatbot should do in plain English and letting an AI agent drive a real browser to verify it. We will cover sending prompts, waiting for streamed replies to finish, and the part everyone gets stuck on, asserting non-deterministic chat output by intent rather than by exact string. The examples use BrowserBash, a free and open-source natural-language browser automation CLI, but the patterns apply to any AI-driven testing setup you build.

Why testing a chatbot UI breaks normal automation

A conventional end-to-end test is a contract of exact expectations. Click this button, expect this text, see this URL. A chat interface violates almost every assumption baked into that contract.

The reply is non-deterministic. Ask a support bot "how do I reset my password?" ten times and you may get ten different sentences. A toHaveText("To reset your password, go to Settings") assertion is guaranteed to flake because you are pinning a hash of a moving target.

The reply streams. The DOM node for the assistant message exists almost immediately, but it is empty, then partial, then finally complete. If your test reads it too early it captures half a sentence. Naive waitForTimeout(3000) sleeps are the usual band-aid, and they make suites slow and still flaky when a cold model takes four seconds instead of three.

The DOM is hostile to selectors. Modern chat UIs render markdown, code blocks, citations, and retry buttons inside deeply nested React trees with hashed class names. Writing page.locator('div.msg-content-wrapper__inner span:nth-child(3)') is a promise to rewrite that test the next time a designer nudges the layout.

State carries across turns. A real conversation test is multi-turn. Message two depends on the model having "remembered" message one, so your test is a small scripted dialogue where each step waits for the previous reply to land before typing the next prompt.

Put those four together and you understand why most teams either skip UI-level chatbot tests entirely (and ship regressions) or maintain a brittle mess that the on-call engineer mutes every Friday.

The plain-English approach: assert intent, not strings

The mental shift is this: stop asserting the exact words the model produced and start asserting the intent the reply should satisfy. A human QA tester reading a chatbot reply does not diff it against a golden string. They read it and ask, "Did this answer the question? Did it mention the right feature? Did it refuse the thing it should refuse?" That is a judgment about meaning, and it is exactly what an AI agent driving the browser can make on your behalf.

With BrowserBash you write an objective in English. An AI agent takes a real Chrome browser, reads the accessibility tree and the visible page, decides what to click and type, waits for the page to settle, and then reports a deterministic verdict. There are no selectors and no page objects. You install it once from npm:

npm install -g browserbash-cli

browserbash run "Open https://chat.example.com, type 'How do I reset my password?' into the message box, send it, wait for the assistant's reply to finish streaming, and confirm the reply explains how to reset a password" --agent --headless --timeout 120

Read that objective the way the agent does. "Wait for the assistant's reply to finish streaming" is a real instruction the agent honors: it watches the page until the response stops growing before it evaluates anything. "Confirm the reply explains how to reset a password" is an intent assertion. The agent is not matching a string, it is judging whether the visible answer is about password resets. That judgment survives the model rephrasing itself tomorrow, which is precisely the property you need when you test an AI chatbot UI.

The --agent flag makes the run emit NDJSON, one JSON event per line, so a CI pipeline or another AI coding agent can consume the verdict without parsing prose. Exit codes are stable: 0 passed, 1 failed, 2 error, 3 timeout. That combination, a semantic assertion plus a machine verdict, is the whole trick.

Where agent judgment is the right tool and where it is not

Be honest about the tradeoff. LLM-judged assertions are the right tool when correctness is about meaning: relevance, tone, refusal behavior, whether a citation was included, whether the bot stayed on topic. They are the wrong tool when correctness is about a fact you can pin exactly, like a URL, a page title, a specific button being visible, or an element count. For those, judgment is overkill and you want a deterministic check. BrowserBash gives you both, and the next section is about mixing them.

Deterministic Verify steps for the parts that are exact

Not everything in a chatbot flow is fuzzy. The "Send" button should be visible. The URL should contain /chat. After you send a message, exactly one new user bubble should appear. The composer should clear. These are deterministic facts, and pinning them with model judgment would be slower and less reliable than it needs to be.

BrowserBash Verify steps compile to real Playwright checks with no LLM judgment involved. Supported forms include URL contains, title is or contains, text visible, a named button/link/heading being visible, element counts, and a stored value equalling something. A pass means the condition literally held. A fail comes with expected-versus-actual evidence in the run_end.assertions block and in the written Result.md table, so triage is a five-second read instead of a screenshot hunt.

You write these in a committable *_test.md file, which is where the workflow gets pleasant. Here is a chatbot smoke test that mixes deterministic structure checks with an intent check:

# Chatbot answers a billing question

- Open https://chat.example.com
- Verify: "Send" button visible
- Type "Where can I see my invoices?" into the message input and send it
- Wait for the assistant reply to finish streaming
- Verify: text "invoice" visible
- Confirm the assistant's reply tells the user where to find their invoices or billing history

The Verify: "Send" button visible line is deterministic. The Verify: text "invoice" line is deterministic too, a cheap sanity check that the reply at least surfaced the right keyword. The final "Confirm the assistant's reply tells the user where to find their invoices" line is outside the deterministic grammar, so it runs as an agent-judged assertion and gets flagged judged: true in the results. That flag matters: when you read the report you can instantly tell which passes were exact and which were judgment calls. You are never guessing whether a green check was real math or a model's opinion.

Run it and write a human-readable Result.md at the same time:

browserbash testmd run ./.browserbash/tests/chatbot_billing_test.md --agent

For more on the assertion grammar and how it fits into a suite, the BrowserBash tutorials walk through Verify steps end to end.

Handling streamed replies without flaky sleeps

The single biggest source of flake in chatbot tests is timing. The answer arrives token by token. Read too early, you assert against a half sentence. Read too late with a fixed sleep, you waste seconds on every fast reply and still lose to the occasional slow one.

The plain-English instruction "wait for the assistant's reply to finish streaming" hands that problem to the agent, which watches the page for stability rather than counting seconds. But you can be more explicit when a UI has a clear signal. Many chat products show a stop button or a pulsing cursor while generating and swap it for a copy/retry button when done. Encode that:

• "Send the message and wait until the stop-generating button disappears and a copy button appears on the latest reply"

Now the wait is anchored to a real UI state transition, not a guess. If your product exposes an explicit "generating" aria-state, mention it in the objective and the agent will key off it.

Multi-turn conversations

Real coverage means testing memory across turns. Because a *_test.md file runs its steps in order against one browser session, a multi-turn dialogue is just a sequence of send-and-wait steps:

• "Type 'My name is Priya' and send it, wait for the reply to finish"
• "Type 'What is my name?' and send it, wait for the reply to finish"
• "Confirm the assistant's reply says the name is Priya"

The third step is the assertion that the bot carried context forward. Notice it is still an intent check ("says the name is Priya") rather than a string match, so it passes whether the bot says "Your name is Priya" or "You told me your name is Priya." This is the kind of behavioral coverage that catches a context-window regression before your users do.

Seeding chat state deterministically with testmd v2

Some chatbot tests need setup that has nothing to do with the UI. You want to test how the bot renders a conversation that already has ten messages, or how it behaves for a user on a paid plan, or what it does when a knowledge-base article exists. Driving that setup through the chat UI itself is slow and circular. You want to seed it through the API and then verify it through the UI.

That is what testmd v2 is for. Add version: 2 to the frontmatter and steps execute one at a time against a single browser session, with two deterministic step types that never call a model: API steps for seeding, and Verify steps for checking. API steps speak GET/POST/PUT/DELETE/PATCH, can carry a JSON body, and can store a value from the response for later use.

---
version: 2
---

# Bot surfaces a freshly created support ticket

- POST https://api.example.com/tickets with body {"subject": "Refund request", "status": "open"}
- Expect status 201, store $.id as 'ticketId'
- Open https://chat.example.com
- Type "What is the status of my latest ticket?" into the message box and send it
- Wait for the assistant reply to finish streaming
- Verify: text "Refund request" visible
- Confirm the assistant's reply says the ticket is open

The API step creates the ticket deterministically and stashes its id. The plain-English steps then drive the chat UI and check that the bot picked up the new ticket. You get a hybrid API-plus-UI test where the fuzzy part (reading a generated reply) is the only part that uses the model, and the setup is fast and reproducible.

One honest caveat: testmd v2 currently drives the builtin engine, which speaks the Anthropic API, so it needs ANTHROPIC_API_KEY or a compatible ANTHROPIC_BASE_URL gateway. It does not yet run directly on Ollama or OpenRouter. Plain v1 files (no version: frontmatter) still run on the default local-first setup exactly as before.

Running chatbot tests locally with free models

You do not need a cloud account or an API key to start. BrowserBash is Ollama-first: it defaults to free local models and, by design, nothing leaves your machine. The resolver looks for a local Ollama install first, then falls back to ANTHROPIC_API_KEY, then OPENAI_API_KEY, then OpenRouter, and only errors with setup guidance if it finds nothing.

For chatbot testing, model choice matters more than for a simple form flow, because judging whether a streamed reply satisfies an intent is itself a reasoning task. Be realistic: very small local models (around 8B and under) can be flaky on long objectives and nuanced intent judgments. The sweet spot is a mid-size local model in the Qwen3 or Llama 3.3 70B class, or a capable hosted model for harder flows.

A pragmatic pattern is to plan on a strong model and execute the mechanical steps on a cheaper one using --model-exec. The strong model decides strategy; the cheap model does the clicking. The BrowserBash learn hub covers model selection and the engine and provider matrix in more depth.

Engines and providers

Two engines ship in the box. The default stagehand engine (MIT, by Browserbase) handles most flows. The builtin engine is an in-repo Anthropic tool-use loop and is required for grid providers. On the provider side, local (your own Chrome) is the default, and cdp, browserbase, lambdatest, and browserstack are available when you need remote or cross-browser runs.

Migrating an existing Playwright chatbot suite

If you already have a Playwright suite for your chat product, you do not have to throw it away to try this approach. browserbash import converts Playwright specs to plain-English *_test.md files heuristically, with no model involved, so it is deterministic and reproducible. It translates goto, click, fill, press, check, selectOption, getBy* locators, and common expects. Environment reads like process.env.API_URL become {{API_URL}} variables.

browserbash import ./e2e/tests --out ./.browserbash/tests

Anything it cannot translate confidently, it does not invent or silently drop. It lands in an IMPORT-REPORT.md so you can see exactly what needs a human pass. In practice the navigation and message-sending scaffolding converts cleanly, and the assertions that used brittle toHaveText on model output are precisely the ones you will want to rewrite as intent checks anyway.

Wiring chatbot tests into CI and monitoring

A chatbot test is only worth writing if it runs on every deploy and, ideally, on a schedule against production. Two features make that clean.

For CI, run a whole folder of tests in parallel with the memory-aware orchestrator, and shard across machines deterministically:

browserbash run-all ./.browserbash/tests --shard 2/4 --junit out/junit.xml --budget-usd 2.00

The --shard 2/4 slice is computed on sorted discovery order, so four parallel CI machines agree on who runs what without any coordination. --budget-usd 2.00 is a hard stop: once the suite crosses the budget, remaining tests are reported skipped, the suite exits 2, and the spend lands in the results and JUnit properties. That budget guardrail matters for chatbot suites specifically, because model-judged assertions cost tokens and you do not want a runaway loop draining an API key. There is also a self-updating GitHub Action that installs the CLI, runs the suite, uploads artifacts, and posts a verdict-table PR comment; the GitHub Action docs cover the matrix and budget inputs.

For production monitoring, run a single canary chatbot flow on an interval and alert only when the pass/fail state flips:

browserbash monitor ./.browserbash/tests/chatbot_smoke_test.md --every 10m --notify https://hooks.slack.com/services/XXX/YYY/ZZZ

The monitor fires on state changes in both directions, never on every green run, so you get a ping when the bot breaks and a ping when it recovers, and silence in between. The replay cache is what makes this affordable: a green run records its actions, the next identical run replays them with zero model calls, and the agent only steps back in when the page actually changed. An always-on chatbot monitor ends up nearly token-free, which is the difference between "nice idea" and "we actually left it on."

Saved logins for gated chat products

Most real chatbots sit behind auth. Rather than re-logging-in on every run, save the session once:

browserbash auth save chatapp --url https://chat.example.com/login

You log in by hand once, press Enter to save the Playwright storageState, and then pass --auth chatapp on any run, testmd, run-all, or monitor invocation (or set auth: in the test frontmatter). If a saved profile does not cover the target start URL, BrowserBash prints a warning instead of silently doing nothing, so a stale login fails loud.

A realistic chatbot test plan

Coverage for a chat product falls into a few buckets. Here is a compact map from behavior to the right assertion style.

The judgment call for each row is: can I pin this exactly? If yes, use Verify and keep it fast and free of model calls. If it depends on the meaning of generated text, use an agent-judged line. Mixing the two in one file is not a compromise, it is the point.

When a different approach is the better fit

Be balanced about it. If your "chatbot" is a scripted decision-tree bot with a fixed set of canned responses, you do not have a non-deterministic output problem at all, and a plain Playwright suite with exact assertions will be faster. If you need to evaluate raw model quality at scale (thousands of prompts scored offline against a rubric), that is an eval-harness job, not a browser-UI job. BrowserBash shines specifically when you care about the rendered chat experience: the streaming, the DOM, the multi-turn UI, and whether the visible answer makes sense to a real user.

You can compare how teams structure this on the BrowserBash case studies page, and if you want the agent inside your own coding assistant, browserbash mcp serves the CLI over the Model Context Protocol so Claude Code, Cursor, or Codex can run these tests as tools.

FAQ

How do you test non-deterministic AI chatbot responses?

You assert on intent rather than exact strings. Instead of matching a golden response, you tell an AI agent what a correct reply should accomplish (answer the question, mention a feature, refuse a request) and let it judge whether the visible reply satisfies that intent. Pair those semantic checks with deterministic Verify steps for the exact parts (button visible, URL contains, a required keyword present) so you get evidence where facts are pinned and judgment where meaning matters.

How do you wait for a streamed chatbot reply to finish in a test?

Avoid fixed sleeps, which are slow and flaky. With a plain-English instruction like "wait for the assistant's reply to finish streaming," the agent watches the page for stability before it evaluates anything. If your UI has a clear signal, anchor the wait to a real state transition instead, for example waiting until the stop-generating button disappears and a copy button appears on the latest message.

Can I run chatbot UI tests without an API key?

Yes. BrowserBash is Ollama-first and defaults to free local models, so nothing leaves your machine and no key is required to start. Keep in mind that judging whether a streamed reply satisfies an intent is a reasoning task, so very small local models can be flaky on nuanced judgments. A mid-size local model in the 70B class or a capable hosted model is the sweet spot for hard flows.

How do I test a chatbot that requires login?

Save the session once with the auth command, which opens a browser so you can log in by hand and stores the Playwright storageState. After that, pass the saved profile with the --auth flag on any run, test file, suite, or monitor invocation, or set it in the test frontmatter. If the saved profile does not cover your start URL, BrowserBash warns you instead of silently failing, so a stale login is easy to spot.

Ready to test your own chat product? Install the CLI with npm install -g browserbash-cli, write your first objective in plain English, and let an AI agent drive a real browser to a verdict. An account is optional, but if you want the free cloud dashboard and hosted history you can sign up here and start validating your chatbot UI today.