A Test-Data Strategy for AI Browser Suites

Most flaky AI browser suites do not fail because the agent misread a button. They fail because two tests tried to register the same email at the same time, or because a run left a coupon marked "used" and the next run assumed it was fresh. A solid test data strategy is the difference between a suite you trust in CI and one you rerun three times and merge anyway. This guide walks through how to give every parallel test its own clean data, how to keep shards from colliding, and how to make a failure reproducible instead of a coin flip, specifically for suites where an AI agent drives a real browser from plain-English steps.

The stakes are a little different when an agent is in the loop. A traditional Selenium test does exactly what its selectors say. An AI agent reads the page and decides what to do, so non-deterministic data can send it down a slightly different path. If the data is stable and unique, the agent behaves consistently. If it is shared and mutable, you inherit both the classic parallel-collision problems and a new one: the agent adapting to state you did not intend. Getting the data right keeps the intelligence useful instead of chaotic.

Why AI suites need a data strategy more, not less

There is a tempting assumption that because an AI agent is flexible, it will "figure out" messy data. It often does, and that is exactly the trap. When an agent quietly works around a duplicate account or a stale cart, your test passes for the wrong reason and you lose the signal you were paying for. An automated check should fail loudly when something is off. Data that drifts between runs turns a deterministic gate into a mood ring.

Parallelism makes this sharper. BrowserBash runs suites through a memory-aware orchestrator that derives concurrency from real CPU and RAM, so on a decent machine you might have six or eight tests hitting the same staging environment at once. Every one of them that creates a user, posts an order, or claims a promo code is now racing the others. Shared fixtures that worked fine when tests ran one at a time start producing intermittent failures that only reproduce under load, which is the worst kind of bug to chase.

The strategy that holds up has three legs. First, generate unique data per test so nothing collides. Second, namespace data per shard so parallel CI machines never step on each other. Third, seed your generators so a failure you saw at 2am reproduces exactly at 10am. Miss any one of these and you get a suite that is either brittle, slow, or impossible to debug.

Generate unique data per test

The baseline rule: no two tests should ever depend on the same mutable record. If a test needs a user, it creates a user. If it needs an order, it creates an order. The cheapest way to guarantee uniqueness is to bake it into the value itself.

For emails, the plus-address trick works on most systems: qa+run7d3f2a@example.com routes to the same inbox but reads as a distinct address to your app. For usernames and other fields, append a short random token or a timestamp. The goal is that even if the same test runs twice in the same second on two machines, the values differ.

In a BrowserBash *_test.md file you keep this in the {{variables}} layer rather than hardcoding it into steps. A variables file supplies the values, and secret-marked ones get masked as ***** in every log line so a leaked password never lands in CI output. Here is a signup test that leans on templated, per-run values:

browserbash run "Open https://staging.example.com/signup, register with email {{signup_email}} and password {{signup_password}}, then confirm you land on the dashboard" \
  --var signup_email="qa+$(date +%s)@example.com" \
  --var signup_password="Test-$(openssl rand -hex 4)" \
  --agent --headless

The --agent flag emits NDJSON so CI reads a verdict instead of parsing prose, and the shell substitutions guarantee a fresh identity on every invocation. Nothing about this run can collide with another, because the uniqueness lives in the data, not in a hope that the tests never overlap.

Faker-style values without a faker library

You do not always need realistic-looking data, but sometimes a field validates format (a real-looking phone number, a plausible postal code) and pure randomness gets rejected. When that happens, precompute the values in your test harness or a small seed step and pass them in as variables. Keep the generation logic outside the plain-English steps so the agent never has to reason about what a valid ZIP code looks like. The agent's job is to drive the browser. The harness's job is to hand it clean, valid, unique inputs.

A useful discipline here is to treat every generated value as disposable. In most cases, do not try to clean up test users after the fact. It adds a whole teardown surface that itself can fail and leave your suite hanging. If your staging store tolerates accumulation, let it accumulate and prune on a schedule. Uniqueness plus disposability beats uniqueness plus fragile cleanup.

Seed data deterministically for API-first setup

Creating a user through the UI on every test is slow and, worse, couples your data setup to the very flow you might be testing. If the signup page breaks, every test that needs a logged-in user also breaks, and now you cannot tell a data problem from a feature regression. The fix is to seed through the API and verify through the UI.

testmd v2 makes this a first-class pattern. Add version: 2 frontmatter and your steps execute one at a time against a single browser session, with two deterministic step types that never call a model: API steps for seeding, and Verify steps for checking. An API step can POST to your backend, assert the status, and store a value from the JSON response for later steps to use.

---
version: 2
---
# Order appears in account history

POST https://staging.example.com/api/orders with body {"sku": "WIDGET-1", "qty": 2}
Expect status 201, store $.id as 'order_id'

Open the account order history page
Verify text "WIDGET-1" is visible
Verify "Order #{{order_id}}" link is visible

The API step seeds the order deterministically, with no browser and no LLM in the loop, then the agent-driven step navigates the UI and the Verify steps compile to real Playwright checks (text visible, link visible) with no model judgment. A pass means the condition actually held. A fail comes back with expected-versus-actual evidence in the assertions block. You have separated data creation (fast, deterministic, API) from behavior verification (the thing you actually want to test).

One honest caveat: testmd v2 currently drives the builtin engine, which needs an ANTHROPIC_API_KEY or a compatible ANTHROPIC_BASE_URL gateway. It does not yet run directly on Ollama or OpenRouter. If you are committed to a fully local, no-key setup, keep your seeding in shell or a pretest script for now and use v1 files for the UI checks. The tutorials walk through both shapes.

Store and reuse ids across steps

The store $.path as 'name' mechanic is the backbone of any multi-step data flow. Create a resource, capture its id, then reference {{order_id}} or {{user_id}} in every downstream step. This keeps your test honest: it verifies the specific record it just created, not "some order that happens to exist." That distinction matters enormously in a shared environment where dozens of orders from other runs are sitting in the same database. Assert against your id, and other runs become invisible to you.

Namespace data per shard so parallel machines never collide

Sharding is how you cut a slow suite's wall-clock time: run-all --shard 2/4 runs a deterministic slice of the tests, computed on sorted discovery order so four CI machines agree on who runs what without any coordination between them. That determinism is a gift, and it extends naturally to data if you plan for it.

The risk is subtle. Even with unique-per-test data, if all four shards draw from the same pool of "prefixes" or the same counter, you can get overlap. The clean solution is to fold the shard index into your namespace. Machine 2 of 4 stamps everything it creates with s2, machine 3 uses s3, and so on. Now collisions are impossible by construction, not by luck, because the shard identity is baked into every value the shard generates.

SHARD=2
TOTAL=4
browserbash run-all .browserbash/tests \
  --shard ${SHARD}/${TOTAL} \
  --var ns="s${SHARD}" \
  --var run_id="$(date +%s)" \
  --junit out/junit-s${SHARD}.xml \
  --agent

Inside your tests, build values from the namespace: qa+{{ns}}-{{run_id}}@example.com, or Company {{ns}} {{run_id}} for a tenant name. The shard prefix guarantees cross-machine isolation, the run id guarantees isolation across time, and together they mean a value from shard 2 today can never match a value from shard 3 tomorrow. Each shard also writes its own JUnit file, so your CI can merge four reports without them clobbering each other, which is the reporting equivalent of the same namespacing discipline.

Viewport matrix multiplies your data needs

If you run --matrix-viewport 1280x720,390x844, every test runs once per viewport. That doubles the number of live test instances, and if two viewport runs of the same test both create a user with the same email, you are back to collisions. Fold the viewport into the namespace too, or rely on a fresh run_id per invocation. The general principle: any axis that multiplies your test count (shard, viewport, retry) is an axis that can multiply collisions, so every dimension needs to be part of the uniqueness key.

Make failures reproducible with seeded runs

Here is the scenario that erodes trust fastest. A test fails in CI overnight. You open it in the morning, run it locally, and it passes. Was it a real bug that self-resolved? A flake? A data race? Without reproducibility you cannot answer, so you shrug and rerun the pipeline. Do that enough times and the suite stops meaning anything.

Seeding is the cure. If your random data comes from a seeded generator, then given the same seed you get the same values, and a failure becomes a recipe you can replay. The practical move is to derive every random value in a run from a single run_id (or an explicit seed) that you log in the run output. When a test fails, you grab that seed from the NDJSON stream, pass it back in, and you get byte-identical data. Now you are debugging a real, stable failure instead of chasing ghosts.

BrowserBash helps here in a few ways. The --agent NDJSON output records the run's events, and because your variables were derived from a logged seed, the failing data is right there in the trace. On the builtin engine, a Playwright trace captures what the agent actually did. And the run history and memory store orders previously-failed and slowest tests first on the next run, so a test that failed once gets re-attempted early, where you will notice it, rather than buried at the end of a long suite.

Seed once, thread it everywhere

Discipline matters more than tooling here. Pick one seed per run. Derive the email, the username, the company name, the order sku suffix, everything, from that single seed. The failure of a seeded suite is that people seed some values and leave others truly random, so the "reproducible" run still differs in the one field that triggered the bug. If it feeds into the app, it comes from the seed. That is the rule that turns "works on my machine" into "here is the exact seed, run it yourself."

Isolate mutable state, not just identities

Unique identities solve the creation problem. They do not solve the consumption problem. Some data is single-use by nature: a one-time coupon, an invite token, a limited-stock item, a rate-limited endpoint. Two parallel tests that both try to redeem the same coupon will race, and one will fail in a way that looks like a product bug but is really a data-sharing bug.

The strategy for consumable data is to give each parallel worker its own pool. If you need single-use coupons, seed a batch keyed by shard: coupons s2-0001 through s2-0050 belong to shard 2 alone. Better yet, mint the consumable inside the test via an API step so it exists only for that run and cannot be touched by anyone else. The general rule is that anything with a "used" state must be owned by exactly one test at a time, and namespacing by shard plus a fresh run id gets you there.

A quick reference for what to isolate

Read-only reference data is the exception worth calling out. A product catalog you only browse, a help article you only read, a pricing page you only assert against, these can be shared with zero risk because nothing mutates them. Do not namespace data that no test writes to. Spend your isolation budget where the writes are.

Reuse auth without reusing identity

Logging in through the UI on every test is both slow and a data cost: each login is another chance for a rate limit or a captcha to bite. BrowserBash lets you save a login once and reuse the session:

browserbash auth save qa-user --url https://staging.example.com/login
# log in once in the browser that opens, press Enter to save

browserbash run-all .browserbash/tests --auth qa-user --agent

The saved profile is a Playwright storageState, and every test that passes --auth qa-user starts already logged in. This is a genuine speed and stability win, but it comes with a data caveat: a shared auth profile means shared account state. If your tests mutate the logged-in user's data (change a profile setting, add to a persistent cart, delete an address), they will interfere with each other even though they never collided on creation.

So the pattern splits by intent. Read-heavy tests that only browse as a logged-in user can safely share one auth profile. Tests that mutate account state should create their own fresh user per run and log in as that user, which costs a login but buys total isolation. Do not try to force one auth strategy across the whole suite. Match the isolation level to what each test actually writes. If a profile's saved origins do not cover a test's start URL, BrowserBash prints a warning rather than silently doing nothing, so a stale profile fails visibly.

Wire it into CI and monitoring

A data strategy only pays off if it runs unattended. In CI, the pieces fit together cleanly: shard across machines with --shard, namespace by shard index, cap spend with --budget-usd so a runaway suite cannot burn your token budget, and let the GitHub Action post a verdict table on every PR. Each shard runs its slice with its own namespace, writes its own JUnit and NDJSON artifacts, and the action stitches the results together.

browserbash run-all .browserbash/tests \
  --shard 1/3 --var ns="s1" \
  --budget-usd 2.00 \
  --junit out/junit-s1.xml --agent

For always-on synthetic monitoring, the data rules still apply but the economics shift. browserbash monitor <test> --every 10m --notify <webhook> runs on an interval and alerts only when a check flips between pass and fail, never on every green run. Because monitors run the same objective over and over, the replay cache makes them nearly token-free: a green run records its actions and the next identical run replays them with zero model calls, stepping the agent back in only when the page changed. But a monitor that creates data every ten minutes will flood your staging database, so monitors should lean toward read-only checks or self-cleaning API-seed-then-verify flows. Design the monitor's data footprint to be sustainable at ten-minute cadence for weeks, not to look fine for one run.

Putting the strategy together

The whole approach reduces to a short checklist you can apply to any suite. Generate unique data per test so nothing collides, ideally by baking uniqueness into the value with plus-addressing and random tokens. Seed setup through the API with testmd v2 so data creation is fast and decoupled from the flows you are testing, and store returned ids to assert against exactly what you created. Namespace by shard index so parallel CI machines are isolated by construction, and extend that namespace across any axis (viewport, retry) that multiplies your test count. Derive every random value from one logged seed per run so a failure is a recipe you can replay. Isolate mutable and single-use data into per-worker pools, and share only genuinely read-only reference data.

None of this is exotic. It is the same test-data discipline good SDETs have always practiced, adapted for two new realities: an AI agent that will quietly adapt to messy state and hide your bugs, and a parallel orchestrator that turns any shared record into a race. Get the data layer right and the agent becomes what you wanted, a reliable driver that gives the same verdict every time. Get it wrong and no amount of model intelligence saves you from a suite you cannot trust. To see how teams structure this end to end, the case studies show real suites in production.

FAQ

How do I stop parallel AI tests from colliding on the same data?

Make every test create its own data with a value that is unique by construction, such as a plus-addressed email with a random token or timestamp appended. Then fold the shard index into a namespace prefix so parallel CI machines cannot generate the same values, and derive time-based uniqueness from a per-run id. With uniqueness baked into the data itself, collisions become impossible rather than merely unlikely.

Should I seed test data through the API or the UI?

Seed through the API and verify through the UI. Creating a user or order through the browser on every test is slow and couples your setup to the exact flow you might be testing, so a broken signup page takes down every test that needs a user. With testmd v2 you can run deterministic API steps to seed data and store the returned ids, then use agent-driven steps and Verify assertions to check the result in the browser.

How do I make a flaky AI browser test reproducible?

Derive every random value in a run from a single seed or run id that you log in the output. When the test fails, grab that seed from the NDJSON stream and pass it back in to regenerate byte-identical data, which turns an intermittent failure into a stable one you can debug. The common mistake is seeding some fields but leaving others truly random, so make the rule absolute: if a value feeds into the app, it comes from the seed.

Can I reuse a saved login across a whole parallel suite?

You can reuse a saved login profile for tests that only read as the logged-in user, and it is a real speed win because you skip the login step every time. But a shared auth profile means shared account state, so any test that mutates the user's data will interfere with the others. The safe split is to share auth for read-heavy tests and give mutating tests their own fresh user per run.

Get started

A test data strategy is only as good as the tooling that runs it, and BrowserBash gives you the namespacing, seeding, and API-seed-then-verify pieces in one CLI. Install it with npm install -g browserbash-cli, point it at your staging environment, and start with a single unique-data run before you scale to sharded CI. Creating an account is optional and everything runs locally by default, but if you want the free cloud dashboard and hosted history you can sign up here.