Automate Competitor Monitoring for Web Changes

The morning a rival quietly drops their starter plan from $49 to $29, you want to know within the hour, not three weeks later when a sales rep loses a deal and asks why. That is the whole reason teams want to automate competitor monitoring: pricing pages, feature lists, and landing-page copy change constantly, and a human checking ten competitor URLs by hand every Monday is both expensive and unreliable. You will miss the change that matters and waste time noting the ones that do not.

I have built competitor-watching pipelines the traditional way — a fleet of Puppeteer scripts, a cron job, a database of yesterday's HTML, and a diff function that pages someone when bytes change. It works, until it does not. The selectors rot. The diff fires on a rotating CSRF token and wakes you at 2 a.m. for nothing. This article is an honest look at a different approach: describing what to watch in plain English and letting an AI agent read the live page, using BrowserBash and its CI exit codes to signal a real diff. I will name where custom scrapers are still the right call, and I will not pretend the agent approach is free of trade-offs.

What "automate competitor monitoring" really covers

The phrase gets used for three different jobs, and teams that conflate them buy the wrong tool.

The first is uptime and availability monitoring — is the competitor's site up, how fast does it load, did their checkout break. That is a job for Pingdom, Checkly, or a synthetic-monitoring service, and an AI browser agent is overkill for it.

The second is content and change detection — did the price on the pricing page move, did a feature appear in or vanish from the comparison table, did the hero headline shift from "developer-first" to "enterprise-ready." This is the heart of competitive intelligence, and it is what this article is about. You are not checking whether a page works; you are checking whether its meaning changed.

The third is market and data aggregation — pulling thousands of SKUs and prices across a whole category into a warehouse for analysis. That is large-scale scraping, and at that volume a managed data-as-a-service vendor or a purpose-built scraping stack usually wins on cost per row.

When growth and product teams say they want to automate competitor monitoring, they almost always mean the second job. You have a short, important list of competitors. You care about a handful of pages each. You want to know fast when something on those pages changes in a way a human would notice, and you want to ignore the churn that does not matter. That last part — ignoring noise — is exactly where the old scraper approach falls down.

Why hand-rolled scraper monitoring keeps paging you for nothing

A traditional change-detection scraper does something deceptively simple. It fetches a page, extracts a region, stores it, and on the next run compares the new extraction to the stored one. If they differ, it alerts. The trouble lives in every word of that sentence.

The selector binds you to today's markup. You write something like div.pricing-card:nth-child(2) .price-amount to grab the Pro plan price. That works until the competitor renames the class in a refactor, adds a promo banner that shifts the nth-child index, or wraps the price in a new span for an A/B test. Now your monitor either throws, returns null, or — worst case — silently grabs the wrong number and reports a "change" that is really just your selector pointing at the wrong cell.

Naive whole-page diffing drowns in noise. If you diff the raw HTML or even the rendered text, you fire on everything: a rotating testimonial, a "12 people viewing this" counter, a CSRF token, a cache-busting query string in an image URL, today's date in the footer. Every one of those is a false positive, and after the third 2 a.m. page for a spinning carousel, your team mutes the alert. A muted alert is worse than no alert, because now you believe you are covered and you are not.

You maintain the noise filter forever. The usual fix is a growing pile of regexes and DOM-pruning rules that strip the dynamic bits before diffing. That filter is code. It needs tests. It breaks when the competitor restructures. You are now maintaining two brittle things — the extractor and the noise filter — across every competitor you watch, and competitors do not coordinate their redesigns with your release calendar.

None of this means custom scrapers are bad. For high-volume, stable, structured sources they are excellent, and I will say so plainly later. But for "watch these twelve marketing pages and tell me when something a human cares about changes," the maintenance tax is brutal and the signal-to-noise ratio is poor.

The plain-English approach: describe the check, not the DOM

The alternative is to stop describing where a value lives and start describing what you want to know. With BrowserBash, a free, open-source, Apache-2.0 CLI from The Testing Academy, you write a plain-English objective and an AI agent drives a real Chrome browser step by step — no selectors, no page objects. It reads the rendered page the way a person would, finds the thing you asked about, and returns a verdict plus structured results.

For competitor monitoring, the shift is from this brittle instruction:

Extract div.pricing-card:nth-child(2) .price-amount and alert if it changed.

to this intent:

Open the competitor's pricing page, find the Pro plan, and confirm its monthly price is still $49. Report FAIL if the price differs.

The second version survives a class rename. It survives a layout shuffle. It survives a promo banner shifting the card order, because the agent finds "the Pro plan" by reading the page, not by counting children. And it is inherently noise-resistant in the way that matters: you told it to look at the Pro plan price, so a spinning testimonial or a rotating "X people viewing" counter is simply not in scope. The agent does not diff the whole page; it answers the specific question you asked.

Here is what that looks like as a command:

browserbash run "Go to https://competitor.example.com/pricing. Find the Pro plan. \
Confirm its monthly price is exactly \$49 and that the plan still includes 'Unlimited projects'. \
Report PASS only if both are true; report FAIL with the current values if either changed." \
--headless --record

The --record flag captures a screenshot and a full .webm session video via ffmpeg, so when the run reports a change you have visual proof of what the page looked like at that moment — useful for a Slack message to the product team that does not want to take your word for it. --headless keeps it quiet for a scheduled job.

A key honesty point: the model matters here. BrowserBash is Ollama-first and defaults to free local models, so you can run this with a guaranteed $0 model bill and nothing leaving your machine. But very small local models (roughly 8B and under) get flaky on long multi-step objectives. For a tight, single-page price check a small model is often fine; for a five-page sweep that logs in, navigates a comparison table, and reasons about feature presence, you want a mid-size local model (Qwen3 or a Llama 3.3 70B-class model) or a capable hosted model. That is a real constraint, not a footnote, and it should shape how you split your monitoring objectives.

Exit codes are the whole point in CI

The reason this fits competitor monitoring so naturally is that BrowserBash was built to run in CI and report machine-readable results, not prose. Run any objective with --agent and it emits NDJSON — one JSON event per line on stdout — and it sets a process exit code you can branch on:

That four-value contract is exactly what a monitoring job needs. Exit 0 means "nothing to see, stay quiet." Exit 1 means "a thing you care about changed — go look." And critically, exit 2 and 3 are distinct from 1, so an infrastructure hiccup does not masquerade as a competitor changing their price. With a raw HTML diff, a network blip and a real price change look identical; here they do not.

A scheduled GitHub Actions or GitLab job becomes trivial:

#!/usr/bin/env bash
# competitor-price-watch.sh — exits non-zero when a watched price changes
browserbash run "Open https://competitor.example.com/pricing, find the Pro plan, \
confirm the monthly price is \$49. FAIL if it is anything else." \
--agent --headless > run.ndjson
code=$?

case $code in
  0) echo "No price change." ;;
  1) echo "PRICE CHANGED — alerting #competitive-intel"; ./notify_slack.sh run.ndjson ;;
  2|3) echo "Monitor run failed (code $code) — infra issue, not a competitor change"; exit "$code" ;;
esac

Because the exit code carries the meaning, your alerting logic stays dumb and reliable. No prose parsing, no scraping the agent's text output, no fragile grep. The NDJSON in run.ndjson is there when you want the structured detail — the current value the agent read, the steps it took — but the branch decision is a single integer. AI coding agents that orchestrate these runs lean on the same contract, which is why the agent-mode docs treat NDJSON and exit codes as first-class.

Committable monitoring checks with Markdown tests

A one-off command is fine for an experiment, but real competitor monitoring is a portfolio: ten competitors, three or four pages each, owned by different people on the growth and product teams. You want those checks in version control, reviewable in a pull request, and parameterized so adding a competitor is a one-line change.

BrowserBash Markdown tests handle this. A *_test.md file is a committable, human-readable test where each list item is a step. It supports {{variables}} templating and @import composition so you can share a common "check a pricing page" recipe across competitors, and any variable marked as a secret is masked as ***** in every log line — handy when a check needs to log into a competitor's free trial with a throwaway account you would rather not leak into CI logs.

A pricing_watch_test.md might read:

# Pricing watch: {{competitor}}

- Go to {{url}}
- Find the {{plan}} plan in the pricing table
- Confirm the monthly price shown is {{expected_price}}
- Confirm the plan still lists the feature "{{must_have_feature}}"
- Report PASS only if the price and the feature both match; otherwise FAIL and state the current value

You run it with the variables for each competitor, and BrowserBash writes a human-readable Result.md after every run that a product manager can read without touching a terminal:

browserbash testmd run ./pricing_watch_test.md \
  --var competitor="Rival A" \
  --var url="https://rival-a.example.com/pricing" \
  --var plan="Pro" \
  --var expected_price="\$49/mo" \
  --var must_have_feature="SSO included" \
  --secret trial_password

Now adding a competitor is a new set of variables, not a new scraper. The check reads like documentation, the diff in your pull request is legible to a non-engineer, and the Result.md artifact is something you can paste into a competitive-intel doc. When a check flips to FAIL, the Result.md tells you the old expectation and the new value the agent read, which is the entire payload your team needs to decide whether to react.

Composing copy and feature checks

Pricing is the obvious target, but the same pattern watches messaging and features. A copy-positioning check:

Open the competitor's homepage. Read the main hero headline. Report FAIL if it no longer contains the word "developer" or "engineer."

A feature-table check:

Open the competitor's comparison page. Confirm the row "SAML SSO" is still marked as available only on the Enterprise plan, not on Pro. FAIL if SSO moved to a cheaper tier.

That second one is genuinely hard for a selector scraper and easy to express in English, because "is SSO available on Pro now" is a question about meaning, not a fixed cell coordinate. When a competitor moves an enterprise feature down-market, that is a pricing-strategy signal your product team wants the same day — and the agent reasons about the table the way a human reading it would.

Where the browser runs, and recording the evidence

By default BrowserBash drives your own local Chrome, which keeps everything on your machine and costs nothing. But scheduled monitoring usually wants to run somewhere reliable and unattended, and one --provider flag moves the same English objective to a different browser backend without changing the check:

So a check you prototyped locally can run in the cloud on a schedule by adding, say, --provider lambdatest — the objective text does not change. The two engines (stagehand, the MIT default by Browserbase, and builtin, an in-repo Anthropic tool-use loop) both honor the same objectives; the builtin engine additionally captures a Playwright trace you can open in the trace viewer, which is gold when you want to see exactly how the agent read a tricky pricing table.

Evidence is not optional for competitive intelligence. When you tell the VP of product that Rival A cut prices, "the agent said so" is weaker than a screenshot and a video. The --record flag gives you both on any engine, and if you opt in to the free cloud dashboard — strictly opt-in via browserbash connect plus --upload — you get run history, video recordings, and per-run replay you can link in a Slack thread. Free uploaded runs are kept 15 days, which comfortably covers a "what did their pricing look like last Tuesday" question. If you would rather keep everything local, browserbash dashboard gives you a free, fully local dashboard with no upload at all. No account is needed to run any of this; the dashboard is a convenience, not a gate.

BrowserBash versus building custom Puppeteer monitors

Let me be even-handed, because both approaches are legitimate and the honest answer is "it depends on the page and the volume."

The dividing line is honest and clear. If you are watching tens of thousands of product pages in a stable e-commerce category where the markup rarely changes, a purpose-built scraper amortizes its setup cost and wins on cost per page — build the Puppeteer fleet. If you are watching a curated list of competitor pricing, feature, and positioning pages that get redesigned on someone else's schedule, the agent approach removes the maintenance tax that makes scraper monitoring rot, and the exit-code contract drops cleanly into CI. The comparison-focused posts on the blog work through more of these trade-offs if you want the longer version.

There is also a hybrid worth naming: use cheap, deterministic HTTP checks for the values that live in a stable JSON API or never move, and reserve the AI agent for the pages where layout churn and meaning-level questions make selectors miserable. You do not have to pick one religion.

When to automate competitor monitoring this way — and when not to

Choose the plain-English, agent-driven approach to automate competitor monitoring when your watchlist is small-to-medium and high-value, when the pages you track get redesigned often, when the questions you ask are about meaning ("did SSO move to a cheaper tier") rather than a fixed coordinate, and when you want a free, local, no-account way to gate a monitoring job in CI on a clean exit code. It shines precisely where scrapers suffer: layout volatility and noisy pages.

Reach for a custom scraper or a managed data vendor when you need to monitor thousands of pages where per-run model cost dominates, when the source is a stable structured API or feed, or when you need bit-exact, fully deterministic reproducibility for compliance or legal reasons. And keep your dumb checks dumb: if a competitor publishes prices in a public JSON endpoint, a one-line curl and jq comparison beats any browser agent — do not reach for AI when a string equality check will do.

One more caution I owe you: be respectful and lawful. Read the target site's terms of service, do not hammer pages at a rate that looks like an attack, and treat any login-gated monitoring carefully — use accounts you are entitled to use. None of that is specific to BrowserBash; it is the baseline ethics of competitive monitoring, and the tool does not absolve you of it. Worth weighing your needs against the pricing page and a real case study before you wire this into a daily job.

A realistic multi-page competitor sweep

To make it concrete, here is the kind of objective BrowserBash runs in a single sentence — a small competitor sweep that checks price, a feature, and positioning in one pass, the same way it can run a full store flow (log in, add an item to the cart, complete checkout, verify "Thank you for your order!"):

browserbash run "Go to https://competitor.example.com/pricing. Find the Pro plan and note its \
monthly price. Then open https://competitor.example.com/compare and check whether 'SAML SSO' is \
available on the Pro plan. Then open the homepage and read the main hero headline. \
Report PASS only if the Pro price is \$49, SSO is Enterprise-only, and the headline still \
mentions developers. If any of those changed, report FAIL and state exactly what is different now." \
--agent --headless --record

That single objective covers all three competitive signals — pricing, feature gating, and positioning — and --record leaves you a video and screenshot for the alert. When the competitor reshuffles their pricing cards next month, you do not touch this check. The agent reads the new layout and answers the same three questions. Compare that to re-writing three selectors and a noise filter every time the page moves, and the maintenance math is not close for a watchlist that changes this often. The source, issues, and more examples live in the GitHub repo, and the npm package page has install and version details.

FAQ

How do you automate competitor monitoring without writing scrapers?

You describe what to check in plain English and hand it to an AI agent that drives a real browser, reads the live page, and answers your question. There are no CSS selectors or XPaths to write or maintain. Because the agent re-reads the page on every run, it adapts when the competitor redesigns instead of breaking the way a selector-bound scraper would, and you scope each check to a specific value so dynamic page noise does not trigger false alerts.

How does BrowserBash signal a competitor change in CI?

Run any objective with the --agent flag and BrowserBash sets a process exit code: 0 means no change, 1 means a diff in the value you watch, 2 means the run errored, and 3 means it timed out. Your CI job branches on that integer, so a real price change (exit 1) is cleanly distinct from an infrastructure hiccup (exit 2 or 3). It also emits NDJSON with the structured detail, so you get the current value the agent read without parsing any prose.

Is an AI agent better than a Puppeteer scraper for tracking competitor prices?

It depends on scale and stability. For a focused watchlist of pricing, feature, and copy pages that get redesigned often, the agent wins because it adapts to layout changes and asks meaning-level questions a selector cannot. For monitoring tens of thousands of stable, structured pages, a purpose-built Puppeteer scraper is cheaper per page once built, so that remains the better fit at high volume.

Does monitoring competitor pages with BrowserBash cost anything?

Running locally costs nothing. BrowserBash is free and open-source, defaults to local Ollama models so you can guarantee a $0 model bill, and needs no account to run. The honest caveat is that very small local models can be flaky on long multi-step sweeps, so for a five-page objective you may want a mid-size local model or a capable hosted one, and a hosted provider for the browser if you run unattended in the cloud.

Ready to automate competitor monitoring the resilient way? Install with npm install -g browserbash-cli and write your first plain-English price-watch check in minutes. No account is needed to run locally — though if you want run history and video replay for your competitive-intel evidence, the optional free dashboard is one sign-up away.