use-case· 15 min read· by Pramod Dutta

Test a Magento Store Checkout With Plain-English AI

Learn how to test a Magento store checkout end to end with plain-English AI objectives: catalog, cart, and multi-step checkout, no selectors.

If you have ever tried to test a Magento store checkout with Selenium or a hand-rolled Playwright suite, you already know the pain. Magento 2 renders a Knockout.js checkout with dynamically generated form field IDs, an accordion of shipping and payment steps, and a mini-cart that reloads over AJAX. The selectors rot every time you touch a theme. This guide shows a different approach: you test a Magento store the way a real shopper experiences it, by writing plain-English objectives that an AI agent executes against a real Chrome browser. No XPath, no page objects, no waiting on a data-bind attribute that changed in the last patch release.

BrowserBash is a free, open-source (Apache-2.0) command-line tool from The Testing Academy that does exactly this. You describe what a customer should be able to do, an AI agent drives a real browser step by step, and you get back a deterministic verdict plus structured results you can wire into CI. The multi-step checkout and the payment sandbox are where intent-based testing earns its keep, and also where you have to be honest about what a test can and cannot verify. We will cover both.

Why Magento checkout is hard to automate

Magento 2 is not a simple form. The default Luma theme and most commercial themes build the checkout as a single-page Knockout application that mounts components asynchronously. Field IDs like #customer-email, #shipping-method-buttons-container, and the payment method radios are generated by layout XML and change when you swap themes, enable a one-page-checkout extension, or upgrade the platform. A shipping method only appears after a valid address triggers the rate request. The place-order button stays disabled until the payment component reports it is ready.

Traditional automation fights all of this with explicit waits, retry loops, and brittle CSS selectors. A single theme update or a new checkout extension can break dozens of tests at once. That is the real cost of selector-coupled tests: not the writing, but the endless maintenance.

When you test a Magento store by intent instead, you sidestep the selector churn. An objective like "add the first product to the cart, go to checkout, fill the shipping form, choose the cheapest shipping method, and reach the payment step" describes shopper behavior, not DOM structure. The agent reads the page, decides what to click, and adapts when the layout shifts. You are testing the outcome a customer cares about.

How BrowserBash tests by intent

BrowserBash positions itself as the open-source validation layer for AI agents, but it works just as well as a plain testing tool. You install it globally and run a one-line objective:

npm install -g browserbash-cli

browserbash run "Open the Magento demo store, open the first product in the catalog, add it to the cart, and confirm the mini-cart shows 1 item" --headless --timeout 120

The agent navigates to the store, snapshots the page, reasons about which element is the first product, clicks through to it, adds it to the cart, and checks the mini-cart count. Every step is logged. At the end you get a verdict: passed, failed, error, or timeout. Those map to exit codes 0, 1, 2, and 3, so CI can gate a deploy on them without parsing any prose.

By default BrowserBash is Ollama-first. It looks for a local Ollama install before anything else, so you can run these tests with free local models and nothing leaves your machine. If no local model is available it resolves in order to your ANTHROPIC_API_KEY, then OPENAI_API_KEY, then OpenRouter. For a long Magento checkout flow, be realistic about model size. Very small local models (around 8B parameters and under) tend to get lost across many steps. A mid-size local model in the Qwen3 or Llama 3.3 70B class, or a capable hosted model, handles the multi-step checkout far more reliably. You can read more about the engine and model options on the features page.

Engines and providers

There are two engines under the hood. The default stagehand engine (MIT, built by Browserbase) interprets your English against the live page. The builtin engine is an in-repo Anthropic tool-use loop driving Playwright, needed for a few advanced features below. Providers control where the browser runs: local uses your own Chrome by default, and you can also point at a cdp endpoint, Browserbase, LambdaTest, or BrowserStack. For most Magento testing on your laptop or in CI, the local provider with a headless browser is all you need.

Writing a Magento catalog and cart test

Start small before checkout. The catalog and cart are high-value flows that break often after a theme change, so they are worth a dedicated test. BrowserBash tests live in committable Markdown files, so they read like documentation and diff cleanly in review.

Here is a magento_cart_test.md file. The # line is the title, dashes are steps, and {{variables}} get substituted at runtime:

# Magento catalog and cart

- Open {{store_url}}
- Open the "Gear" category from the top navigation
- Open the first product in the listing
- Choose size M and color Blue if those options are shown
- Add the product to the cart
- Open the mini-cart and store the item count as "cart_count"
- Verify text "Shopping Cart" is visible

Run it against the file:

browserbash testmd run ./magento_cart_test.md --headless

The store URL comes from a variable so the same test runs against your local Magento, a staging instance, or the public demo store without edits. After each run BrowserBash writes a human-readable Result.md next to the test, so a non-technical stakeholder can see exactly what the agent did. If you are new to the Markdown test format, the tutorials walk through it from scratch.

Notice the Verify text "Shopping Cart" is visible line. That is a deterministic assertion, not an AI judgment, and it matters a lot for a checkout suite. Let us look at why.

Deterministic Verify assertions for checkout

An AI agent is great at figuring out how to reach a state. You do not want it improvising on whether that state is correct. If the agent decides the checkout "looks about right," you have a test that passes when it should fail. BrowserBash separates these two jobs.

Verify steps in a test file compile to real Playwright checks with no model involvement. The supported grammar covers the assertions a storefront test actually needs: URL contains a path, title is or contains text, text is visible, a named button or link or heading is visible, element counts, and a stored value equals an expected value. A pass means the condition genuinely held. A fail comes with expected-versus-actual evidence in the run_end.assertions block and in the Result.md assertion table, so you are not guessing why it broke.

For Magento checkout, that means you can assert the concrete facts:

# Magento checkout to payment step

- Open {{store_url}}
- Add the first available product to the cart
- Proceed to checkout
- Fill the email field with {{guest_email}}
- Fill first name {{first_name}}, last name {{last_name}}, address, city, and zip
- Select the country and state
- Choose the flat rate shipping method
- Continue to the payment step
- Verify "Payment" heading is visible
- Verify text "Place Order" is visible
- Verify url contains "checkout"

Those three Verify lines are deterministic. The agent navigates the flow with judgment, and then Playwright, not the model, confirms you actually reached the payment step with a place-order control present. A Verify line that falls outside the supported grammar still runs, but agent-judged, and it is flagged judged: true in the output so you always know which assertions were deterministic and which were the model's opinion.

The payment sandbox: being honest about what you can verify

Here is where a lot of "AI tests your store" pitches get dishonest, so let us be straight. You generally should not, and often cannot, complete a real payment in an automated end-to-end test. Live payment gateways block automation, use 3-D Secure redirects and one-time codes, and charging a real card in CI is a bad idea. What you can and should verify is that the checkout correctly reaches the payment step and offers the expected methods.

Magento gives you a few honest options for testing past that point:

Match the assertion to reality. A test that claims to verify payment but really just clicked a button is worse than no test. Pick the scope you can verify honestly and label it as such.

Handling test card data safely

When you drive a sandbox card form, keep the values in variables and mark the sensitive ones as secrets. Secret-marked variables are masked as ***** in every log line, so a published test card or a sandbox API key never shows up in CI output or in Result.md. Store non-secret values like the guest email and address in a plain variables file, and keep sandbox credentials in secret-typed variables the run substitutes at execution time without ever printing them.

Running the full suite in CI

A real storefront has more than one flow. You test the catalog, the cart, guest checkout, registered-user checkout, coupon codes, and maybe a couple of category filters. BrowserBash has a memory-aware parallel orchestrator for exactly this. Point run-all at a folder and it discovers every test, derives a safe concurrency level from your real CPU and RAM, runs previously-failed and slowest tests first, and flags flaky ones.

browserbash run-all ./.browserbash/tests --junit out/junit.xml --shard 2/4 --budget-usd 2

Two things here matter for a Magento suite. The --shard 2/4 flag runs a deterministic quarter of the suite, computed on sorted discovery order so four parallel CI machines each take a distinct slice without any coordination. The --budget-usd 2 flag is a hard spend cap: once the suite crosses two dollars of estimated cost, no new tests launch, the remaining ones are reported skipped, and the suite exits with code 2. That estimate comes from a cost_usd value in each run's run_end, backed by a bundled per-model price table. If you are running local models the cost is effectively zero, but the budget guard is a good safety net for hosted-model runs.

The suite writes a RunAll-Result.md summary and a JUnit XML file your CI already knows how to display. For wiring this into a pipeline, the official GitHub Action documentation covers the marketplace action that installs the CLI, runs the suite, uploads artifacts, and posts a self-updating PR comment with the verdict table.

The replay cache keeps runs cheap

The first green run of a test records the actions the agent took. The next identical run replays them with zero model calls, and the agent only steps back in when the page actually changed. For a Magento checkout that mostly stays the same between commits, this turns an expensive multi-step reasoning flow into a near-instant, near-free replay. When a theme update genuinely changes the page, the replay misses, the agent re-plans that step, and the cache updates. You get speed on the stable path and adaptation on the changed one, with no selectors to maintain.

Testing logged-in checkout with saved sessions

Guest checkout is one path. Registered-customer checkout, with saved addresses and a stored payment method, is another, and logging in on every single test is slow and wasteful. BrowserBash solves this with saved logins. You log in once, interactively, and reuse that session everywhere.

browserbash auth save magento-customer --url https://your-store.test/customer/account/login

That opens a browser, you sign in as a test customer, press Enter, and BrowserBash saves the session as a Playwright storageState profile. From then on you attach it with a flag on any run:

browserbash run "Go to checkout, use my saved shipping address, choose the default shipping method, and reach the payment step" --auth magento-customer --headless

You can also set auth: in a test file's frontmatter so the whole suite runs as that customer. If the saved profile does not actually cover the store's origin, BrowserBash prints a warning instead of silently pretending it worked, which saves you from a whole class of confusing "why is it logged out" failures.

Monitoring your live checkout

Testing before deploy is half the job. A checkout that passed in CI can still break in production because a third-party script changed, a payment provider had an incident, or a config was flipped in the admin. Monitor mode runs a test on an interval and alerts you only when the pass or fail state changes, in either direction, so you are not drowning in green notifications.

browserbash monitor magento_checkout_test.md --every 10m --notify https://hooks.slack.com/services/XXX/YYY/ZZZ

Every ten minutes the agent runs your checkout test against the live store. If it was passing and starts failing, you get a Slack message. If it recovers, you get told that too. Slack incoming-webhook URLs get proper Slack formatting automatically; any other URL receives the raw JSON payload. Because the replay cache handles the stable path, an always-on monitor is nearly token-free, which makes ten-minute synthetic checks genuinely affordable. This is the kind of validation the case studies dig into.

Migrating and seeding data for advanced flows

If you already have a Playwright suite for your Magento store, you do not have to start over. The browserbash import command converts Playwright specs into plain-English test files heuristically and deterministically, with no model in the loop. It translates goto, click, fill, press, check, selectOption, getBy* locators, and common expects into English steps, and turns process.env.X references into {{X}} variables. Anything it cannot translate cleanly lands in an IMPORT-REPORT.md file rather than being silently dropped or invented.

browserbash import ./tests/e2e/checkout.spec.ts

The output is a starting point, not a finished suite. You read the generated English, tighten a few steps, and add Verify assertions where the original spec had expect calls. It saves the tedious first pass and never hides what it could not convert. There is also a browserbash record command that opens a visible browser, lets you click through a flow once, and writes a plain-English test on Ctrl-C, with password fields captured as a secret marker so a real password never leaves the page.

For a checkout test you sometimes need known data: a product in stock, a customer that exists, a valid coupon. Rather than clicking through the admin, testmd v2 lets you seed data with deterministic API steps and verify it through the UI in one file. Add version: 2 to the frontmatter and steps execute one at a time against a single browser session. An API step like POST a cart or coupon runs with no model at all, checks the status, and can store a value from the JSON response for later steps. A following Verify step confirms the effect in the storefront UI. The honest caveat: testmd v2 currently runs on the builtin engine, so it needs an ANTHROPIC_API_KEY or a compatible ANTHROPIC_BASE_URL gateway, and does not yet run directly on Ollama or OpenRouter. You can follow the format details on the learn hub.

When intent-based testing is the right choice, and when it is not

Be balanced about this. Plain-English AI testing is not the answer to every problem, and pretending otherwise would be exactly the kind of hype this tool is built to avoid.

Intent-based testing shines when the UI changes often (theme-heavy Magento stores are a classic case), when you want tests non-engineers can read, when you are validating shopper outcomes rather than internal DOM structure, and when you want an always-on production monitor without a maintenance burden. It is also the natural fit if you are wiring a storefront into an AI agent workflow through the Model Context Protocol, since the same tool that tests your checkout can be exposed as an MCP tool to Claude Code, Cursor, or Codex.

It is the weaker choice when you need microsecond-precise timing assertions, when you are unit-testing a single Knockout component, or when you have a hard requirement for fully offline, zero-model execution across every step (the replay cache gets you most of the way, but the first run and any changed step still need a model). For pure API contract testing with no UI, a dedicated API framework is simpler. And if your team already has a mature, low-maintenance Playwright suite that nobody complains about, there is no urgent reason to rip it out. Import a slice, see if the plain-English version is easier to live with, and decide from evidence.

The honest summary: use intent-based tests for the flows that break because the UI moved, and keep deterministic tools for the flows that need microsecond precision or pure API assertions. Most Magento teams have plenty of the former.

FAQ

Can BrowserBash complete a real Magento checkout with payment?

It can complete a full order when you use an offline payment method like Check / Money Order or Cash On Delivery, or a properly configured sandbox gateway such as Braintree or Stripe with published test cards. It should not charge a real card in an automated test, and live gateways plus 3-D Secure challenges usually block automation anyway. For most CI runs the honest and reliable scope is to verify that checkout reaches the payment step with the correct methods visible, and to reserve full sandbox orders for staging.

Do I need API keys to test my Magento store?

No. BrowserBash is Ollama-first, so it defaults to free local models and nothing leaves your machine. It only falls back to a hosted model if you have an ANTHROPIC_API_KEY, OPENAI_API_KEY, or OpenRouter key set, and only in that order. The one exception is testmd v2 with its deterministic API steps, which currently needs the builtin engine and therefore an Anthropic key or a compatible gateway.

How does this handle Magento theme changes that break selectors?

Because you describe shopper intent instead of DOM selectors, a theme change that renames field IDs or reshuffles the checkout does not break your test the way a CSS selector would. The AI agent reads the current page and adapts its clicks to whatever layout it finds. The replay cache stores the fast path for unchanged pages and re-plans only the specific step that actually moved, so you get speed on stable flows and automatic adaptation on changed ones without editing a locator.

Is BrowserBash free for testing an e-commerce store?

Yes. The CLI is free and open-source under Apache-2.0, including the engines, the local dashboard, the replay cache, the MCP server, and NDJSON agent output. Running with local Ollama models has no per-run cost at all. Optional paid features are limited to hosted retention and team services, and anything that runs on your own machine stays free.

You can test a Magento store checkout by intent in the next few minutes. Install the CLI, point it at your storefront, and write your first objective in plain English:

npm install -g browserbash-cli

Start with a simple cart test, add Verify assertions for the payment step, then wire run-all into CI and a monitor job against production. If you want the hosted dashboard and 15-day run retention, create a free account at browserbash.com/sign-up, though an account is entirely optional and everything above works fully offline without one.

Try it on your own appnpm install -g browserbash-cli
Start learning