Choosing model browser testing decisions are less about brand loyalty and more about failure modes. A browser agent has to read a real page, decide the next action, recover from small UI surprises, and produce a verdict you can trust. The cheapest model is not cheap if it gets stuck. The most accurate hosted model is not the right default if your data should never leave the machine. The fastest model is not useful if it confidently clicks the wrong thing.
BrowserBash is built around that tradeoff. It is a free, open-source Apache-2.0 natural-language browser automation CLI by The Testing Academy. It is Ollama-FIRST, so it defaults to free local models, no API keys, and nothing leaving your machine when you stay local. It can also use Anthropic Claude, OpenAI, and OpenRouter when you bring keys. The CLI auto-resolves local Ollama first, then ANTHROPIC_API_KEY, then OPENAI_API_KEY, then OpenRouter.
This guide gives you a practical model selection framework for BrowserBash and similar agentic browser testing. We will cover local versus hosted models, small-model flakiness, latency, cost, privacy, cheap-model execution routing, deterministic assertions, and when to split a suite by difficulty. No fabricated benchmark table, no fake universal winner. The right model depends on the flow you are validating and the risk of being wrong.
Choosing model browser testing criteria
A browser testing model has three jobs. It must understand the objective, map page observations to useful actions, and know when enough evidence exists for a verdict. Cost, latency, and accuracy all matter because each job stresses a different part of the system. Accuracy decides whether the agent can finish the run. Latency decides whether developers will tolerate it in the loop. Cost decides whether CI can run it often.
The first criterion is task complexity. A short flow with stable labels, such as "open the pricing page and verify the Free and Pro plan headings are visible," can often work with a smaller or cheaper model. A long flow through auth, dashboard navigation, form validation, and role-specific UI needs stronger reasoning. Very small local models around 8B and under can be flaky on long multi-step objectives, so do not judge the category by a tiny model failing a hard workflow.
The second criterion is data sensitivity. If the run touches private staging data, internal admin pages, or regulated user records, local Ollama may be the default even if it is slower. BrowserBash's local-first story is important here: no API keys are needed, and nothing leaves your machine when you stay local. Hosted models can be appropriate, but that is a conscious provider decision.
The third criterion is determinism. The more you move assertions into deterministic Verify lines, the less the model has to judge. That can let you use a cheaper model for navigation while Playwright-backed assertions handle evidence. Model selection is not isolated from test design.
BrowserBash model resolution in plain English
BrowserBash starts local. If Ollama is available, it defaults to local models. If not, it can auto-resolve to Anthropic through ANTHROPIC_API_KEY, then OpenAI through OPENAI_API_KEY, then OpenRouter. It also supports OpenRouter and Anthropic Claude explicitly when you bring your own key. This model story is designed for practical adoption: try it free and local, then move to a hosted provider only when the run needs it.
That order matters because many teams want an AI browser tool without immediately creating another SaaS dependency. You can install with npm install -g browserbash-cli, run browserbash, and keep the first experiments on your machine. If the local model is good enough for your smoke suite, you have a low-cost validation layer. If a hard flow needs more reasoning, route that test differently.
The engine also matters. BrowserBash has Stagehand as the default engine and a builtin Anthropic tool-use loop in the repo. The builtin engine is required for LambdaTest and BrowserStack providers, and testmd v2 currently drives the builtin engine. That means v2 needs ANTHROPIC_API_KEY or an ANTHROPIC_BASE_URL compatible gateway today. It does not yet run directly on Ollama or OpenRouter.
This caveat is important when choosing model browser testing architecture. If you need deterministic API and Verify steps from testmd v2 right now, plan for the builtin engine requirement. If your priority is local Ollama, use v1 behavior or direct objectives where that fits, and keep an eye on the project for future engine support. The GitHub repository is the right place to inspect current behavior.
Local Ollama: privacy and cost control
Local Ollama is the best starting point when privacy, independence, and direct cost control matter. BrowserBash defaults to it because local models let you validate browser behavior without sending page observations to a hosted API. That fits internal apps, staging environments, and exploratory adoption by teams that do not want procurement friction before they learn whether agentic browser testing helps.
The tradeoff is hardware and model quality. A local 8B-class model may handle simple flows, but it can be flaky on longer multi-step objectives. It may miss a subtle button, loop on a modal, or judge success too early. A mid-size local model such as Qwen3 or a Llama 3.3 70B-class setup is the more realistic sweet spot for hard flows, assuming you have the machine to run it with acceptable latency.
Latency is often the hidden cost. A local model that is technically free but slow enough to break developer flow may not be the right model for pre-merge CI. You can still use it for nightly validation or security-sensitive checks. Split the suite by feedback need. Fast smoke checks can use the fastest reliable model. Deeper flows can run later or on stronger hardware.
browserbash run "Open https://staging.example.com/login, sign in with the saved profile, and verify the account dashboard is visible" --auth staging-user --agent
The local path works best with clear objectives and short critical paths. Do not ask a small model to perform vague exploratory QA across an entire application. Give it one outcome, one page family, and deterministic checks where possible.
Hosted models: accuracy and managed latency
Hosted models are the better fit when hard flows need stronger reasoning, faster response, or less local infrastructure. BrowserBash supports Anthropic Claude and OpenRouter with your own key, and it can use OpenAI via environment detection. A hosted model may finish a complex run in fewer attempts, which can make it cheaper overall despite API cost.
The honest downside is data movement and provider cost. Page observations and task context may leave your machine depending on provider configuration. For public sites, demo apps, or non-sensitive staging data, that may be acceptable. For private customer data, regulated workflows, or unreleased features, you need a policy decision before routing tests through hosted models.
Hosted models also change failure analysis. If a run fails, look at BrowserBash's structured output first: status, summary, final_state, assertions, cost_usd, and duration_ms. Do not assume a stronger model makes every failure a product bug. Browser automation can still hit auth setup issues, timing problems, unexpected modals, or ambiguous objectives.
Use hosted models selectively. A capable hosted model is a good choice for onboarding flows, role-heavy dashboards, complex checkout paths, or test generation from recorded behavior. It may be overkill for "verify the docs page title" checks. Choosing model browser testing means matching model strength to the economic value of the result.
Cheap-model execution routing
BrowserBash supports cheap-model routing through --model-exec: plan on a strong model, execute on a cheap one. This is useful because not every step of a browser run requires the same reasoning level. Planning may need the stronger model to interpret the objective and decide strategy. Execution may be a sequence of simpler actions that a cheaper model can handle.
This pattern is not a free lunch. If the cheap execution model misreads the page, you can lose the savings in reruns. Start with a narrow suite and compare outcomes. Look at pass rate, duration, cost_usd, and whether failures are product failures or agent failures. The point is to route based on evidence, not optimism.
browserbash run "Create a trial workspace, invite a viewer, and verify the invitation appears as pending" --model-exec cheaper-model --agent
Cheap-model execution pairs well with replay cache. Once a green run records actions, repeat runs can replay with zero model calls. That makes the execution model less relevant for stable paths. The model matters again when the UI changes or replay needs help.
If your suite has three tiers, route them differently. Tier one: deterministic Playwright or BrowserBash replay-heavy checks. Tier two: local or cheap hosted model for simple agent flows. Tier three: stronger hosted or mid-size local model for difficult workflows. This is more useful than arguing for one universal model.
Deterministic assertions reduce model pressure
Model selection gets easier when the model is not responsible for every verdict. BrowserBash 1.5.0 introduced deterministic Verify assertions in testmd v2. Supported Verify steps compile to real Playwright checks: URL contains, title is or contains, text visible, named button, link, or heading visible, element counts, and stored value equals. If a Verify line falls outside the grammar, it still runs with agent judgement and is flagged judged: true.
This matters for accuracy. A model can navigate to the dashboard, but a Playwright check can verify the exact heading or stored value. A model can apply a coupon, but an assertion can confirm the discount text is visible. When the pass condition is deterministic, you can choose a model for navigation skill rather than asking it to be the final judge of truth.
---
version: 2
---
GET https://staging.example.com/api/users/me
Expect status 200, store $.email as 'email'
Open the account settings page
Verify text '{email}' visible
Verify 'Save changes' button visible
Testmd v2 has the engine caveat mentioned earlier: it currently needs the builtin engine with Anthropic or a compatible gateway. Still, the design principle applies everywhere. Push facts into deterministic assertions when the grammar supports them. Use the agent for the messy human-like parts of the flow.
The BrowserBash tutorials and learn hub contain practical examples of objective phrasing. Good objectives reduce model pressure because the model has fewer interpretations to choose from.
Compare model choices with a real matrix
A useful model evaluation is small, repeatable, and honest. Pick a set of flows that represent your product. Run each flow with the candidate model. Record whether it passed, whether the failure was a product bug or agent behavior, how long it took, whether replay worked on the second run, and whether the cost estimate was present. Do not compare models only on one happy path.
| Model path | Best fit | Watch out for |
|---|---|---|
| Local small model | Short, stable checks with low sensitivity to occasional retries | Flaky behavior on long multi-step objectives |
| Local mid-size model | Private staging flows that need stronger reasoning | Hardware, latency, and operational complexity |
| Hosted capable model | Complex workflows, faster feedback, hard UI reasoning | API cost and data movement policy |
| Cheap execution route | Suites where planning is hard but execution is routine | Savings vanish if execution mistakes cause reruns |
| Replay-heavy suite | Stable green checks after first successful run | Page randomness can force the agent back in |
The matrix should include your product's pain points. If your UI has many modals, include a modal flow. If auth is fragile, include saved login reuse. If responsive behavior matters, use --viewport or --matrix-viewport. BrowserBash 1.5.0 labels viewport matrix results in events, JUnit, and results, so the data stays understandable.
Do not hide failures during evaluation. A failed test is useful evidence. In BrowserBash MCP, a failed validation is a successful tool call that returns a failed verdict. The host agent should read the verdict, not treat tool completion as pass.
MCP and AI coding agents
Choosing model browser testing setup changes again when BrowserBash is called by another AI agent. BrowserBash 1.5.0 added an MCP server through browserbash mcp. It exposes run_objective, run_test_file, and run_suite over stdio. Each returns structured verdict JSON with status, summary, final_state, assertions, cost_usd, and duration_ms. BrowserBash is listed on the official MCP Registry as io.github.PramodDutta/browserbash.
browserbash mcp
claude mcp add browserbash -- browserbash mcp
In this setup, model choice affects two layers: the host coding agent and the browser-driving agent. Avoid stacking expensive models by accident. If the host agent is already strong, you may still want BrowserBash to run a local or cheaper model for simple validation. For hard flows, the BrowserBash side may need stronger reasoning regardless of the host.
Structured verdicts keep the layers clean. The host agent should inspect status and assertions. It should not parse prose or reinterpret page text. This separation is useful for cost and for security, especially when validating changes against pages that may contain untrusted content.
The BrowserBash features page covers the product surface, and the npm package is the fastest way to install the CLI for local experiments.
Who should choose which model path
Choose local Ollama if you are early in adoption, privacy-sensitive, cost-sensitive, or validating internal apps where API calls are hard to approve. Use a mid-size local model for serious flows when hardware allows. Keep objectives short and make assertions deterministic where possible.
Choose hosted models if your flows are complex, your local hardware is weak, or your team values faster feedback enough to pay for it. This is often the right choice for critical onboarding, checkout, role management, and flows that change often during product development. Bring your own key and monitor cost_usd.
Choose cheap-model execution when you have already measured a stronger model's planning value and a cheaper model's execution reliability. It is a tuning knob, not a default guarantee. Use replay cache and budgets around it.
Choose deterministic scripts instead of agentic BrowserBash when the exact selector path is known, extremely stable, and cheaper to maintain than a natural-language objective. BrowserBash is an open-source validation layer for AI agents. It is not a reason to delete every Playwright test.
Model evaluation should include recovery behavior, not only happy-path completion. Real web apps have cookie banners, slow spinners, disabled buttons, validation errors, duplicate labels, and role-specific states. A model that succeeds only when the page is perfectly clean may look good in a demo and still waste time in CI. BrowserBash makes this visible because you can inspect final_state, assertions, duration, and the event stream rather than relying on a polished summary.
One practical pattern is to tag tests by model difficulty in the file name or folder structure. Simple read-only checks can use the default local path. Workflows with multiple decisions can run on a stronger model or with cheap-model execution routing. Tests with valuable deterministic claims should move to testmd v2 when the builtin engine requirement fits your environment. This gives the team a routing map instead of a debate every time a run fails.
Do not forget the human side of latency. A five-minute validation may be fine for nightly runs but unacceptable after every AI-generated code patch. A slower local model can still be the right choice for sensitive internal data, while a hosted model can be the right choice for a public marketing flow that needs quick feedback. The mature answer is usually a portfolio: local where privacy dominates, hosted where speed and accuracy dominate, and deterministic checks wherever the assertion can be expressed without a model.
Finally, record the model choice in the test documentation or CI command instead of leaving it implicit. Future maintainers need to know whether a test was designed for local privacy, hosted accuracy, cheap execution, or replay-heavy steady state. When a run starts failing, that context prevents random model switching and keeps debugging focused on the page, the objective, the assertion, or the provider boundary.
A last sanity check is to keep one deliberately simple browser test in every model trial. If the candidate model cannot complete a short, stable, read-only flow, it is not ready for harder validation. If it passes the simple flow but fails the complex one, you have useful routing evidence rather than a vague model complaint.
FAQ
What is the best model for BrowserBash?
There is no single best model for every BrowserBash run. Start local with Ollama for privacy and low direct cost, then use a mid-size local model or capable hosted model for harder flows. Judge the model by pass quality, latency, reruns, and cost on your actual tests.
Are 8B local models good enough for browser testing?
They can work for short and stable objectives, but very small local models around 8B and under can be flaky on long multi-step browser tasks. For hard workflows, a Qwen3 or Llama 3.3 70B-class local model is a more realistic sweet spot. Hosted models may be better when latency and accuracy matter more than local-only execution.
Can BrowserBash use OpenRouter or Anthropic Claude?
Yes. BrowserBash supports OpenRouter and Anthropic Claude when you bring your own key. Its auto-resolution starts with local Ollama, then Anthropic, then OpenAI, then OpenRouter, so local use remains the default path.
Does testmd v2 work with local Ollama?
Not directly today. Testmd v2 currently drives the builtin engine and needs ANTHROPIC_API_KEY or an ANTHROPIC_BASE_URL compatible gateway. Ollama remains useful for other BrowserBash runs, but v2 has that current engine requirement.
Install with npm install -g browserbash-cli, run a small model matrix on your own flows, and keep an optional account path open through sign up if you want the cloud dashboard later.