It's 4:40 p.m. on a Thursday, a release is scheduled for the morning, and your Playwright suite just went from all green to fourteen failures in one CI run. Nobody touched checkout. What shipped was a marketing banner: a "Limited time, 20% off" strip above the product grid, with its own "Add to cart" button that happens to share the accessible name of the real one. Every test that calls page.getByRole('button', { name: 'Add to cart' }) now throws strict mode violation: locator resolved to 2 elements. The feature works fine. The regression is fictional. And someone on your team is opening the Trace Viewer at 5 p.m. to confirm, again, that this is a locator problem and not a real bug.
That afternoon costs real money, and it has nothing to do with how fast Playwright executes a click. Playwright is fast, arguably the fastest mainstream browser automation tool available. The question this article answers is a different one: over the life of a suite, what does it cost in human hours to keep locators pointed at the right elements as the app changes underneath them, and what changes when you replace the locator with a plain-English description that an AI agent re-derives on every run.
Auto-waiting solved timing, not targeting
Give Playwright credit where it's due. Before it acts on an element, Playwright checks that the target is attached to the DOM, visible, stable (not mid-animation), able to receive events (not covered by another element), and enabled. That's the actionability model, and it genuinely killed a category of race-condition flakiness that plagued older Selenium scripts built on manual sleeps and brittle explicit waits. getByRole, getByLabel, and getByText are also a real improvement over raw CSS: they target by accessible role and visible text instead of a fragile DOM path, and Playwright's own docs steer you toward them for exactly that reason.
None of that touches the actual maintenance problem, because auto-waiting answers "is this ready yet," not "is this the right element." A locator will patiently wait out the full timeout for an element that no longer exists, then fail anyway. And strict mode, Playwright's default refusal to act on an ambiguous locator, is a genuinely good design decision: it surfaces ambiguity immediately instead of silently clicking the first match the way older find_element calls did. But it's also exactly why the banner in the story above breaks fourteen tests that never touched the banner. Waiting patiently for the right thing and correctly identifying the right thing are two different problems, and auto-waiting only solves the first one.
Where Playwright test maintenance cost actually accumulates
The Page Object Model file is the visible cost. fixtures/pages/CheckoutPage.ts grows a getter for every field, and somebody owns that file for the life of the suite. But the POM is only the part you can see in a diff. The rest is quieter, and it's where most of the real Playwright test maintenance cost hides.
| Cost center | What drives it |
|---|---|
| Page Object Model files | A getter per element, patched every time markup shifts, owned indefinitely |
data-testid conventions |
Only as stable as cross-team discipline; one component ships without one and someone falls back to a CSS class |
| Codegen output | npx playwright codegen records a working script fast, complete with nth() chains and generated classes nobody goes back to refactor into getByRole |
| Design-system upgrades | A major version bump to your component library can restructure internal DOM wholesale, breaking every locator that wasn't role-based |
| Multi-project fan-out | playwright.config.ts projects mean one broken locator turns into several red CI jobs (chromium, firefox, webkit, mobile viewports) for a single root cause |
| Retry and quarantine creep | retries: 2 plus a spreading habit of test.fixme on "known flaky" tests quietly narrows what a green suite actually proves |
None of this shows up in a sprint estimate, because none of it is "write the test." It's what happens after, on a suite that has to survive months of a product actually changing shape.
The cost curve a sprint estimate never captures
Authoring a Playwright test is close to a fixed, one-time cost: write the flow, wire the locators, run codegen if you want a head start. Maintaining it is a recurring cost that shows up every time the DOM underneath it changes, for as long as the test lives, and it scales with suite size rather than with how well the test was originally written. A perfectly written test on a page that gets redesigned every sprint costs more over a year than a sloppy one on a page nobody touches.
This next part is genuinely illustrative, not a benchmark: if a mid-size Playwright suite spends even a few maintenance hours a week on selector repair, that's not new coverage, it's just keeping existing tests green, and it adds up to dozens of hours a quarter that never appear as a line item anywhere, because they're spread across whoever happened to be on call when CI went red. The number that actually matters isn't "how many hours," which varies wildly by team and app and should never be taken from a blog post as gospel, it's the ratio: for your suite, does maintenance time grow faster or slower than the value of the coverage. On a churny UI, it grows faster, and that's the real argument for changing the approach, not raw execution speed.
What changes when an agent reads the page instead of a locator file
BrowserBash is a free, open-source (Apache-2.0) CLI that replaces the locator with a plain-English objective. You describe the goal, an AI agent drives a real Chrome browser, observes the current page, decides the next action, and returns a pass/fail verdict plus any values you asked it to extract. There's no Page Object file, because there's nothing to store: the agent reads labels, roles, and visible text on this run and finds the match, the same way a human tester would ignore the banner's button because the objective said "the real Add to cart button in the product card," not a CSS path.
npm install -g browserbash-cli
browserbash run "Open the product page, dismiss any promotional banner if one is showing, add the featured product to the cart, and verify the cart shows 1 item" \
--headless
Two engines sit behind that command. stagehand is the default, MIT-licensed, built by Browserbase. builtin is an in-repo Anthropic tool-use loop that additionally captures a Playwright trace when you record a run, so the debugging artifact you already know how to read still shows up. Model choice is yours: BrowserBash resolves a free local Ollama model first if one is installed, then falls back to ANTHROPIC_API_KEY, then OPENAI_API_KEY, then an OpenRouter key, so you can run the whole thing at zero API cost before ever reaching for a hosted model.
The honest tradeoff is the one you'd expect: a Playwright click is a direct protocol command measured in milliseconds, an agent step includes a model inference round trip measured in seconds, and two runs of the same objective can take a slightly different number of steps because the model plans at run time instead of replaying a fixed script. For an 800-test regression wall running on every commit, that's a real cost, and Playwright is still the right tool for the job. For the tests that keep breaking because a banner shipped, the calculus looks different.
Making sure "passed" still means something
An agent told only to "complete the checkout" can report success without checking the one thing you actually cared about, and that's a real risk, not a hypothetical one. It's the honest reason "AI testing" alone isn't a slogan worth trusting on its face. The fix is the same discipline good Playwright tests already use: explicit, checkable assertions instead of vibes.
BrowserBash's *_test.md format supports a version: 2 frontmatter flag that turns on per-step execution, where a plain-English Verify line matching a known pattern compiles to a real Playwright check, not a model's opinion. Verify the URL contains 'order-confirmation' and Verify the 'Place order' button is visible both run deterministically; a Verify line that doesn't match a known pattern still runs, but is flagged so you can tell it was the agent's judgment call rather than a hard check.
---
version: 2
---
# Checkout smoke test
- Open {{base_url}}/cart
- Click the "Proceed to checkout" button
- Fill in the shipping form with {{name}} and {{address}}, then submit
- Verify the URL contains 'order-confirmation'
- Verify the text 'Order confirmed' is visible
One limitation worth stating plainly instead of burying: version: 2 files currently run on the builtin engine only, and it switches automatically even if you have stagehand configured as your default. The plain-English steps in that mode also need an Anthropic-speaking model rather than a local Ollama or OpenRouter model directly. That's a real constraint today, not a footnote. More on the assertion grammar and both engines is on the BrowserBash features page.
You do not have to hand-convert the whole suite
The obvious objection to any of this is switching cost: nobody wants to hand-rewrite two years of Playwright specs to try a different approach. You don't have to start from a blank page. browserbash import reads existing *.spec.ts / *.test.ts files and generates plain-English *_test.md drafts from them:
browserbash import tests/checkout.spec.ts --out-dir .browserbash/imported
It also writes an IMPORT-REPORT.md next to the generated tests, flagging the lines it couldn't confidently translate so a human reviews exactly those, instead of the whole file line by line. That's a realistic migration path: point it at your highest-churn spec files first, not your entire suite, review what it produces, and run the result locally before it goes anywhere near CI.
Where Playwright still wins
None of this makes Playwright the wrong tool for most of what it already does well. Keep Playwright for the large, stable regression wall where sub-second-per-test budgets matter, for anything needing pixel-precise interaction or low-level network interception, for flows where a bit-identical execution trace matters for compliance or debugging, and for pages your team owns and rarely restructures. Playwright's built-in sharding across workers is also still the right answer for a suite with hundreds of tests that all need to finish before a merge gate opens.
Reach for a plain-English agent on the handful of flows that break for reasons unrelated to the product: pages under heavy A/B testing or frequent redesign, third-party checkout or SSO pages you don't control, and the smoke and journey tests you want a product manager or a manual QA engineer to read and approve in a pull request without knowing what a locator is. The realistic pattern is both, in the same pipeline, gated by the same exit code, not a rewrite of a suite that already works.
FAQ
Does Playwright's auto-waiting reduce test maintenance cost?
It reduces timing-related flakiness, the race conditions that come from acting on an element before it's ready, but it does nothing for structural change. Auto-waiting patiently waits for a locator to become actionable; it does not help when the locator itself no longer matches the right element because a class was renamed, a component was restructured, or a new element sharing the same role or text was added to the page.
What is the biggest hidden cost in a Playwright suite?
Usually it isn't one dramatic failure, it's the steady, distributed cost of triage: deciding whether a red build is a real defect or a broken locator, then patching Page Object files, data-testid conventions, and codegen output every time the frontend changes shape. It rarely appears as a line item because it's spread across whoever is on call when CI goes red, which is exactly why teams underestimate it until someone finally totals it up.
Is an AI browser agent slower than Playwright?
Per action, yes. A Playwright click is a direct protocol command in milliseconds; an agent step includes a model inference round trip, typically seconds. For a large regression suite run on every commit, that difference matters and Playwright is the better tool. For the smaller set of tests that mostly fail due to selector churn rather than real defects, the maintenance time saved usually outweighs the added run time.
Can an AI browser agent give a false pass?
Yes, if the objective is vague, such as "complete the checkout," and nothing verifies the specific outcome you actually care about. The mitigation is explicit checks: BrowserBash's Verify steps in a version: 2 test file compile to real, deterministic Playwright assertions instead of relying on the model's judgment, so a pass means the condition actually held, not that the agent felt finished.
Do I have to rewrite my Playwright suite to try this?
No. browserbash import converts existing *.spec.ts files into plain-English *_test.md drafts and flags the lines it couldn't confidently translate in an IMPORT-REPORT.md, so you review those specifically rather than starting from scratch. The realistic path is converting the handful of specs that break most often for selector reasons and leaving the stable ones exactly where they are.
Does BrowserBash replace Playwright entirely?
No, and it isn't positioned that way. Playwright remains the right choice for large, stable, deterministic regression suites where speed and exact repeatability matter most. BrowserBash targets the tests that cost the most to maintain over time: high-churn UIs, third-party pages you don't control, and smoke or journey coverage that benefits from being readable by people who don't write locators.