If you have a ChatGPT Plus subscription and a website to check, the temptation is obvious: open agent mode, type "go to my staging site, sign in, add a product to the cart, and tell me if checkout works," and let it run. Using ChatGPT agent mode for testing feels like the future of QA — no selectors, no flaky locators, just plain English describing what a user would do. And for a one-off sanity check, it genuinely can pull this off. The agent drives a real browser, reads the page like a person, clicks the right buttons, and reports back in prose. I have watched it walk a checkout flow end to end and catch a broken coupon field along the way.
But "it worked once in a chat window" and "I can rely on this for QA" are very different claims. This article is the real-talk version: where ChatGPT agent mode wins for testing, exactly where it breaks (CAPTCHA, real authentication, and the fact that there is no API to script it), and where a committable, scriptable CLI that emits machine-readable output takes over. I work on BrowserBash, so the last section is the vendor talking — I have tried to keep everything before it honest, including the spots where ChatGPT is the better tool for the job.
What ChatGPT agent mode actually is
ChatGPT agent mode is OpenAI's agentic browsing feature, the descendant of what launched as Operator in early 2025 and folded into ChatGPT later that year. Inside a chat, you ask the agent to do something on the web, and it spins up a virtual browser it controls — navigating, clicking, typing, reading screens, filling forms, and working with files. It reasons about the task, takes actions step by step, and asks for your permission before anything consequential like a purchase or a form submission. You can interrupt it, take over the browser, or stop the run at any point.
For testing, the appeal is the same thing that makes it appealing for booking flights or filling out forms: you describe intent, not implementation. There is no page.locator(), no waiting on selectors, no maintaining a page-object layer. The model looks at the rendered page and figures out what to do. When a button moves or a class name changes, a traditional Selenium or Playwright script snaps; the agent usually just adapts because it is reading the page the way a human tester would.
That generalization is real and it is the headline strength. It is also the source of every limitation below, because a chat-bound, vision-and-reasoning agent is optimized for "help one person finish one task," not "run the same verification a thousand times and tell a pipeline pass or fail."
Where ChatGPT agent mode wins for testing
Let me be specific about the jobs it does well, because dismissing it wholesale would be dishonest.
Exploratory smoke checks. "Open the homepage, click through the top nav, and tell me if anything 404s or looks broken." The agent is good at this kind of open-ended poke. It will narrate what it sees, flag the obvious breakage, and sometimes notice things you did not ask about — a misaligned banner, a console-level error surfaced in the UI, a CTA that goes nowhere.
One-off "does this even work" verification. Before a demo, you want to know the signup form on production accepts a test email and lands on the dashboard. Typing that into agent mode and watching it run is faster than writing a script for a check you will run once.
Reading and summarizing dynamic content. Agent mode is strong at extraction framed as a question: "go to this pricing page and tell me the price of each tier and whether the annual toggle changes them." It reads the live page and answers in structured prose.
Steps a human would describe vaguely. "Find the contact form, fill it with believable test data, and confirm you get a thank-you message." Loose, human-shaped instructions are exactly where the reasoning shines and where a brittle script would need a dozen explicit waits.
Repro hunting. When a bug report is fuzzy ("checkout sometimes fails on mobile"), having an agent try the flow a few different ways and describe what it hits can shorten the loop to a reproduction.
If your QA need is "I, a human, want to verify something now and read a sentence about the result," ChatGPT agent mode is a legitimately good tool. The trouble starts when you need that verification to be repeatable, gated, and machine-readable.
Where it breaks: CAPTCHA
The first wall most testers hit is CAPTCHA, and it is a hard wall. Any site fronted by Google reCAPTCHA, Cloudflare Turnstile, hCaptcha, or a similar challenge tends to stop the agent cold. This is not a bug — it is the entire point of those systems. They exist to detect and block automated, non-human traffic, and a model driving a browser is exactly what they are tuned to catch. Plenty of real-world e-commerce, banking, and account-security flows sit behind one, which means a big slice of the flows you most want to test are off-limits.
There is a well-publicized wrinkle here worth being precise about. In 2025, security researchers at SPLX demonstrated that a prompt-injection trick — framing a CAPTCHA as a "fake test" the agent had already agreed to solve — could get ChatGPT agent mode to click through certain reCAPTCHA-style and even some image challenges. Text-based ones fell more easily than image ones. That made headlines, and it tells you two things. First, the default, intended behavior is that the agent will not solve CAPTCHAs — you have to manipulate it into doing so, which is against policy and not a strategy you should build QA on. Second, even the researchers found it inconsistent, especially on image challenges.
For practical testing the takeaway is simple: if your flow has a CAPTCHA, plan for the agent to stop there and hand control back to you. That is fine for a supervised one-off where you solve the challenge yourself. It is fatal for an unattended, scheduled check.
A cleaner path for your own properties is to put a known test bypass on non-production environments — a header, a query flag, or a reCAPTCHA test key that always passes — so automation never sees a live challenge. That is a sound practice regardless of which tool drives the browser, and it sidesteps the whole argument. But it is environment plumbing you have to set up; out of the box, CAPTCHA is where agent mode's reach ends.
Where it breaks: real authentication
The second wall is login. Agent mode can type a username and password into a form — that part works. The problem is everything around real-world authentication.
Credentials in a chat box. To have the agent log in, you have to give it credentials. Pasting a real password into a ChatGPT conversation is a non-starter for most teams' security posture, and you should not do it for anything that matters. Agent mode supports a "take over" handoff where you type sensitive info into the browser yourself, which is the right pattern for a supervised session — but it also means a human has to be sitting there for every run.
Multi-factor authentication. Push prompts, TOTP codes, SMS one-time passwords, and hardware keys are designed to require the actual account owner. The agent cannot pull a code off your phone. It can pause and ask you to do it, which again pins a human to every run.
Session persistence. The way you avoid logging in on every single test in a real suite is by reusing a saved authenticated session — storage state, cookies, a token. Agent mode does not give you a clean, scriptable hook to inject "here is a logged-in session, start from there." Each conversation is its own context, and there is no committable artifact you carry between runs. So you are either logging in fresh every time or babysitting a handoff.
For a single supervised check, "I'll type my own password when it asks" is workable. For an automated login-flow suite that runs on every deploy, having to manually authenticate each time defeats the purpose. If login-flow testing is your specific need, the trade-offs are worth reading about on their own — we go deeper in our AI login flow testing guide.
Where it breaks: no API, no script, no CI
This is the limitation that matters most for anyone past hobby use, and it gets the least attention. ChatGPT agent mode lives inside the ChatGPT interface. You drive it by typing into a chat. There is no public API for agent mode itself — no endpoint you call, no flag you pass, no way to invoke a run from a script and read a structured result back.
That single fact rules out almost everything a QA engineer actually wants from automation:
- You cannot put it in CI. There is no command for a GitHub Actions or Jenkins step to call. A pipeline cannot type into a chat window and read prose back.
- You cannot version your tests. The "test" is a sentence you typed in a conversation. There is no file in your repo, no diff in code review, no history of what changed.
- You cannot parse the result reliably. The output is natural-language prose written for a human. A pipeline needs a clean pass/fail signal and an exit code, not a paragraph it has to scrape with a regex.
- You cannot run it unattended at scale. Plus-tier agent usage is metered — on the order of a few dozen agent messages a month at the $20 plan (limits as of 2026, check your current plan) — and every run is interactive. Higher tiers raise the ceiling, but the interactive, non-scriptable nature does not change.
- Runs are not deterministic or reproducible across a team. Your colleague cannot check out your test and run the identical thing. They have to retype the prompt into their own chat.
To be fair and precise: OpenAI does offer programmable building blocks elsewhere — the Responses API and the Agents SDK — that let developers build agentic, tool-using workflows in code. Those are real and capable. But they are not "ChatGPT agent mode." They are a different product surface where you write and host the orchestration yourself. If what you wanted was the convenience of agent mode plus the ability to script it, that convenience does not survive the move to the SDK; you are now building an agent, not using one.
So the honest framing is: agent mode is a great cockpit for a human pilot. It is not an autopilot you can wire into a pipeline.
A side-by-side on the testing job
Here is the comparison the way I would put it to a teammate deciding what to reach for.
| Capability | ChatGPT agent mode | A scriptable NDJSON CLI (e.g. BrowserBash) |
|---|---|---|
| Plain-English objectives, no selectors | Yes | Yes |
| Self-adapts when the UI changes | Yes | Yes (Stagehand engine, self-healing) |
| Runs from a shell / CI step | No (chat UI only) | Yes (browserbash run "...") |
| Machine-readable output + exit codes | No (prose) | Yes (--agent NDJSON, exit 0/1/2/3) |
| Committable, versioned tests | No | Yes (*_test.md files in git) |
| Reuse a saved auth session in code | No clean hook | Yes, via your own setup steps |
| Handles CAPTCHA unattended | No (stops/handoff) | No — same wall; use test bypass on staging |
| MFA without a human present | No | No — same constraint; use seeded sessions |
| Runs fully local / $0 model bill | No (hosted, metered) | Yes (Ollama-first, nothing leaves the machine) |
| Cost model | Plus/Pro subscription, metered agent messages | Free, open-source; local models are $0 |
Two honest notes on this table. CAPTCHA and MFA are not problems BrowserBash magically solves either — those are properties of the site, not the driver. Any automation hits them, and the real answer in both cases is environment setup (test bypasses, seeded sessions) rather than a tool that "beats" the challenge. Where the CLI genuinely pulls ahead is everywhere automation infrastructure matters: scripting, CI, version control, parseable output, and cost.
What "scriptable" buys you in practice
BrowserBash is a free, open-source (Apache-2.0) command-line tool from The Testing Academy that takes the same plain-English idea — describe the objective, let an AI agent drive a real Chrome browser — and makes it a first-class CLI citizen instead of a chat feature. You install it once:
npm install -g browserbash-cli
Then a one-off check looks like the agent-mode equivalent, except it runs in your terminal and you can pipe, schedule, and commit it:
browserbash run "Go to https://shop.example.com, add the first product to the cart, \
proceed to checkout, and confirm the order summary shows the correct total"
The model that interprets that English is Ollama-first by default. With the auto setting, BrowserBash looks for a local Ollama instance and uses it — free, no API keys, and nothing leaves your machine, which means a guaranteed $0 model bill and no data going to a third party. If you do not have Ollama, it falls back to an ANTHROPIC_API_KEY (Claude) or an OPENAI_API_KEY (GPT-4.1), and if neither is present it tells you how to fix it. One honest caveat: very small local models (8B and under) get flaky on long multi-step objectives. The sweet spot is a mid-size local model in the Qwen3 or Llama 3.3 70B class, or a capable hosted model for the genuinely hard flows.
Output a pipeline can actually read
The piece that makes this CI-ready is --agent. Add it and BrowserBash emits NDJSON — one JSON object per line — instead of prose:
browserbash run "Sign in with the seeded test account and confirm the dashboard loads" --agent
You get a stream of step events like {"type":"step","step":1,"status":"passed","action":"navigate","remark":"..."} and a terminal {"type":"run_end","status":"passed","summary":"...","final_state":{...},"duration_ms":...}. Exit codes follow the obvious contract: 0 passed, 1 failed, 2 error, 3 timeout. A GitHub Actions or Jenkins step can branch on that exit code without parsing a single sentence of English. This is the exact thing agent mode cannot give you, and it is why a CLI fits the AI browser tests in a Jenkins pipeline story while a chat window does not.
Tests you can commit and review
For anything you will run more than once, BrowserBash supports markdown tests — *_test.md files where each list item is a step. They live in your repo, go through code review, and support {{variables}} templating and @import composition so you can share a login flow across suites. Secret-marked variables are masked as ***** in every log line, which is the answer to "I am not pasting a password into a chat." After each run it writes a human-readable Result.md. You run a suite with:
browserbash testmd run ./checkout_test.md
That is the structural difference in one line: agent mode's test is a sentence in a conversation that vanishes when you close the tab; a markdown test is an artifact your team owns, diffs, and trusts.
Local-first, with optional dashboards
Every run is kept on disk at ~/.browserbash/runs (secrets masked, capped at the last 200), so you have history without any cloud. If you want a UI, browserbash dashboard opens a fully local dashboard at localhost:4477 — no account, nothing uploaded. There is also an opt-in cloud dashboard if you run browserbash connect and then add --upload to a run; without that flag, nothing leaves your machine. For debugging a flaky flow, --record captures a screenshot plus a .webm session video (and, on the builtin engine, a Playwright trace) so you can watch exactly what the agent saw.
When to choose which
I am not going to pretend a CLI replaces agent mode for every job. They overlap on the "plain English, no selectors" promise and diverge sharply on everything else.
Reach for ChatGPT agent mode when the task is a one-off, a human is in the loop anyway, and the value is a quick answer rather than a repeatable check. Exploratory testing, "does this work right now" before a demo, reading a live page and summarizing it, or hunting a vague repro — agent mode is fast and pleasant for all of these. If you already pay for ChatGPT and the flow has no CAPTCHA or MFA in the way, it is often the shortest path to an answer. It is also genuinely better than a CLI at open-ended, unscripted exploration where you do not know in advance what you are looking for.
Reach for a scriptable CLI when the verification needs to repeat, gate a deploy, or live in version control. Anything that runs in CI, anything you want a clean pass/fail and exit code from, anything where the test should be a committed artifact your team reviews, and anything where you want runs to be free and fully local. Regression suites, smoke checks on every merge, scheduled monitors — these are CLI shaped, and a chat window cannot do them.
The two are not really competitors so much as different stages. People often prototype a check in agent mode because it is the fastest way to see if the idea holds, then move the ones worth keeping into committed, scriptable tests once they want them to run unattended. If you want to see that progression with real flows, the tutorials and learn sections walk through it, and the case studies show end-to-end examples.
A realistic workflow that uses both
Here is how I would actually combine them on a real project, no tribalism.
Start in agent mode when a flow is new and you are still figuring out what "correct" even means. Type the objective, watch it run, let it surprise you with edge cases you had not considered. This is exploration, and the reasoning agent is great at it.
Once you know what you want to verify, write it down as a BrowserBash markdown test. Now it is a file. It goes in the repo next to the feature code. A reviewer can read the steps in the PR. The next engineer runs the identical thing. For environments with auth, seed a test account and reuse its session rather than logging in live; for CAPTCHA, flip on the staging bypass so automation never meets a challenge.
Wire the suite into CI with --agent so the pipeline reads NDJSON and exit codes, not prose. Keep runs on local models for the cheap, high-volume checks so your model bill stays at zero, and reserve a capable hosted model for the few genuinely hard, long flows where a small local model would wobble. Use --record when something goes flaky so you have video and a trace to look at.
That gives you the best of both: agent mode's speed for discovery, and a committed, scriptable CLI for everything that has to be trustworthy and repeatable. You are not paying per interactive message for checks that run on every commit, and you are not retyping prompts into a chat to reproduce a teammate's result.
The bottom line
Can ChatGPT agent mode test your website? Yes — and well, for the right shape of task. It is a strong tool for supervised, exploratory, one-off verification where a human is reading the answer. Using ChatGPT agent mode for testing falls apart the moment you need the check to be unattended, repeatable, gated in CI, or owned as code. CAPTCHA stops it. Real authentication forces a human handoff. And the lack of an API means there is no way to script it, version it, or read a machine-parseable result.
None of that is a knock on OpenAI — agent mode was built to help a person finish a task, and it does that. It just was not built to be QA infrastructure. When you need automation that composes, fails fast, integrates with CI, and produces a stable contract a pipeline can trust, that is a CLI's job. BrowserBash gives you the same no-selectors, plain-English experience, plus NDJSON output, committable tests, local-first $0 runs, and exit codes a build server understands. Compare the trade-offs against your own stack on the pricing page (the CLI is free and open source either way).
FAQ
Can ChatGPT agent mode run automated website tests in CI?
No. Agent mode only runs inside the ChatGPT chat interface and has no public API, so there is no command a CI system like GitHub Actions or Jenkins can call, and no machine-readable result to branch on. For pipeline use you need a scriptable tool that emits structured output and exit codes, such as a CLI with an NDJSON agent mode.
Does ChatGPT agent mode solve CAPTCHA when testing a site?
By default, no — it is intended to stop at CAPTCHA challenges and hand control back to you. Security researchers showed it could be tricked into solving some challenges via prompt injection, but that behavior is against policy, inconsistent on image challenges, and not something to build QA on. The reliable fix for your own sites is a CAPTCHA test bypass on non-production environments so automation never sees a live challenge.
How does ChatGPT agent mode handle login and MFA during testing?
It can type a username and password into a form, but pasting real credentials into a chat is a security risk and it cannot retrieve MFA codes from your phone or hardware key. For sensitive logins it pauses and asks you to take over the browser, which means a human has to be present for every run. That handoff is fine for a supervised one-off but defeats unattended, scheduled testing.
What is a scriptable alternative to ChatGPT agent mode for QA?
BrowserBash is a free, open-source CLI that drives a real Chrome browser from plain-English objectives, like agent mode, but adds the automation infrastructure agent mode lacks. It runs from a shell or CI step, emits NDJSON with pass/fail exit codes via the --agent flag, supports committable markdown tests, and runs fully local on Ollama for a $0 model bill. That makes it suited to repeatable, version-controlled, pipeline-friendly testing.
Ready to script what agent mode can only do by hand? Install it with npm install -g browserbash-cli and start a free run — no account required. Create an optional account anytime at browserbash.com/sign-up.