Scrape job listings with AI

If you have ever tried to scrape job listings at scale, you know the job is rarely about the data and almost always about the page. The role title, salary band, location, posting date, and apply link are sitting right there in front of you. Getting them into a tidy spreadsheet is the part that breaks. Selectors drift overnight, a board ships a new card layout, infinite scroll hides half the results, and your carefully tuned XPath turns into a pile of null values. This guide walks through a different approach: using an AI agent that drives a real browser, reads each posting the way a human reader would, and hands you back structured rows you can drop straight into a CSV or database.

The tool I'll use throughout is BrowserBash, a free, open-source CLI that takes a plain-English objective and executes it in a real Chrome browser. No selectors, no page objects, no maintenance treadmill. You describe what a row looks like, and the agent extracts it. I'll be honest about where this shines, where a classic scraper still wins, and where the legal and ethical lines sit, because scraping job boards is a domain where those lines matter.

Why scraping job listings is harder than it looks

A job board looks like a simple list. Under the hood it is one of the more hostile surfaces on the public web. Three forces make it tough.

First, the markup is unstable by design. Big boards A/B test their result cards constantly. The salary that lived inside a <span class="metadata-salary"> last week now lives in a flex container with a hashed class name. Any scraper keyed to those classes silently returns empty fields, and you often don't notice until your dataset is already polluted.

Second, the data is loaded late and lazily. Most modern boards render results with client-side JavaScript, paginate through infinite scroll, and pull salary or company detail from a separate API call after the card is already on screen. A plain HTTP fetch of the page source gives you a skeleton with no jobs in it. You need a browser that actually executes the page.

Third, anti-bot defenses are real and getting stricter. Indeed runs behind Cloudflare and starts issuing CAPTCHAs when request rates climb past roughly one per second. LinkedIn applies aggressive rate limiting and account-level flagging; community testing in 2026 found that several popular LinkedIn job scrapers got accounts throttled or flagged after only a few dozen saved jobs. None of this is a reason to give up on scraping job listings, but it does mean your tooling has to behave like a careful human, not a firehose.

An AI browser agent addresses the first two problems head-on. It reads rendered content, it can scroll and wait, and it interprets meaning instead of matching a brittle DOM path. It does not magically defeat the third problem — no honest tool does — so we'll cover rate limits and politeness as first-class concerns later.

How AI changes the extraction model

Traditional scraping is selector-first. You inspect the page, write a locator for each field, and the script blindly grabs whatever sits at that location. The logic lives in the selectors, and the selectors live at the mercy of the front-end team.

An AI agent flips that. The logic lives in your description of the data. You tell the agent, in English, that each row should contain a job title, a company, a location, a salary range if shown, a posting date, an employment type, and the apply URL. The agent looks at the rendered page, decides which on-screen text corresponds to each of those fields, and returns them as structured values. When the board reshuffles its layout, the visual meaning of "the company name" doesn't change, so your extraction keeps working even though every class name moved.

With BrowserBash specifically, you write an objective and the agent drives Chrome step by step, returning a verdict plus a set of structured extracted values. Under the hood the default engine is Stagehand (MIT-licensed, built by Browserbase), which exposes act, extract, and observe primitives and is self-healing against minor DOM changes. There is also a builtin engine — an in-repo Anthropic tool-use loop driving Playwright — that gets used automatically when you target grid providers like LambdaTest or BrowserStack. You can switch with --engine stagehand|builtin, but for scraping job listings the default Stagehand engine is the right call.

The honest caveat here is about model capability. BrowserBash is Ollama-first and will happily run entirely on a local model for a $0 model bill, but very small local models (8B parameters and under) get flaky on long, multi-step extraction objectives. They lose track of which fields they've filled and start hallucinating salaries. The sweet spot is a mid-size local model in the Qwen3 or Llama 3.3 70B class, or a capable hosted model for the harder boards. I'll show both paths.

Installing BrowserBash and running your first extraction

Setup is one command. You need Node 18 or newer and Chrome installed for the local provider.

npm install -g browserbash-cli
browserbash run "Go to https://news.ycombinator.com/jobs and extract every job posting as a row with the company name, the role, the location if listed, and the link to the posting. Return the rows as structured JSON."

The Hacker News jobs page is a friendly first target: it's a flat list, it's public, and it has no anti-bot wall, so it's perfect for confirming your install works before you point the agent at something tougher. The agent opens Chrome, reads the page, and prints a verdict along with the structured rows it pulled.

On the model side, you don't have to configure anything. The default model is auto, which resolves in order: a local Ollama install becomes ollama/<model> (free, no keys, nothing leaves your machine); otherwise an ANTHROPIC_API_KEY becomes claude-opus-4-8; otherwise an OPENAI_API_KEY becomes openai/gpt-4.1; otherwise you get a clear error telling you how to fix it. For a serious scraping run I'd pin the model explicitly so results are reproducible:

# Free and fully local — good for friendly boards
browserbash run "Extract all job rows from https://news.ycombinator.com/jobs as JSON with company, role, location, link" --model ollama/qwen3

# Capable hosted model — better for messy, JS-heavy boards
browserbash run "Extract all job rows from https://news.ycombinator.com/jobs as JSON with company, role, location, link" --model claude-opus-4-8

No account is needed to run any of this. If you want a place to browse past runs, browserbash dashboard opens a fully local dashboard on localhost:4477 with nothing uploaded anywhere.

Designing your objective so the rows come back clean

The quality of your extracted rows is mostly decided by how you phrase the objective. A vague objective produces vague data. A precise one produces a clean schema. Treat the English description like a column spec.

A few patterns that consistently improve results when you scrape job listings:

Name every field explicitly. Don't say "extract the job details." Say "for each posting, capture: title, company, location, salary range (or null if not shown), employment type, posted date, and apply URL." Listing the fields gives the agent an exact shape to fill and makes missing data show up as an honest null instead of a silently dropped column.

Tell it what to do with absent data. Job boards are inconsistent. Some cards show salary, most don't. If you say "salary range, or null if the posting doesn't list one," you get a predictable column instead of the agent guessing or skipping the field.

Bound the scope. "Extract the first 25 postings" or "extract every posting visible without logging in" keeps the run short and predictable. Open-ended "get all jobs" invites the agent to chase infinite scroll forever.

Ask for a canonical format. Dates are the worst offender. "Normalize the posted date to YYYY-MM-DD" turns "3 days ago," "Posted Tuesday," and "Jun 9" into one consistent column. Same idea for salary: "express salary as a numeric annual range in USD where possible."

Here is the kind of objective that produces genuinely usable rows:

browserbash run "Open https://news.ycombinator.com/jobs. For each job posting on the page, extract a row with these exact fields: company, role, location (or null), remote (true/false/null), and url. Skip ads and non-job links. Return a JSON array of row objects." --model claude-opus-4-8 --record

The --record flag captures a screenshot plus a .webm session video through bundled ffmpeg, and on the builtin engine it also writes a Playwright trace. That recording is invaluable when a run returns fewer rows than you expected — you can watch exactly what the agent saw and where it stopped.

Getting structured rows out of agent mode

Running one objective and reading the output by eye is fine for a spot check. For anything you want to schedule or pipe into a database, use --agent, which switches the output to NDJSON: one JSON object per line, no prose to parse.

browserbash run "Extract all job postings from https://news.ycombinator.com/jobs as rows with company, role, location, url" --agent --model claude-opus-4-8

You get a stream of progress events shaped like {"type":"step","step":1,"status":"passed","action":"navigate","remark":"..."}, and a terminal event shaped like {"type":"run_end","status":"passed","summary":"...","final_state":{...},"duration_ms":...}. Your extracted rows arrive in that final state object. Exit codes are machine-friendly too: 0 for passed, 1 for failed, 2 for error, 3 for timeout, so a wrapping script knows whether to trust the output without reading a word of English.

This is the mode that makes scraping job listings reproducible. A small shell or Node wrapper can run the objective, read the last NDJSON line, pull final_state, and append the rows to a CSV. Because the output contract is fixed, you're not regex-matching prose — you're consuming structured JSON, which is exactly what you wanted from a scraper in the first place. The BrowserBash tutorials walk through wiring agent mode into a CI job if you want a worked example.

Committing your scraper as a markdown test

There's a second way to run BrowserBash that's underrated for recurring scrape jobs: markdown tests. A *_test.md file is a committable, human-readable script where each list item is a step. You run it with browserbash testmd run ./jobs_test.md, and it writes a Result.md after each run.

For a scraping pipeline this is genuinely useful. You can keep the extraction logic in version control, review changes in a pull request, and template the target with {{variables}} so the same file scrapes different boards or search queries. If your objective needs a login token or an API key, mark that variable as a secret and it gets masked as ***** in every log line and in the on-disk run store. You can also compose files with @import, so a shared "navigate and accept cookies" preamble lives in one place.

A markdown test for job scraping might list steps like: open the board, dismiss the cookie banner, scroll until 30 cards are loaded, then extract each card into a row with the agreed fields. Because it's prose, a non-engineer on the team can read it and understand exactly what data is being collected — which, for compliance reasons we'll get to, is a feature, not a nicety. The learn section covers the markdown test format in depth.

AI agent versus classic scraper: an honest comparison

AI extraction is not always the right answer. Sometimes a hand-written scraper is faster, cheaper, and more reliable. Here's a straight comparison so you can choose well.

The pattern I'd actually recommend: use an AI agent when the board changes often, when you're scraping a long tail of smaller career sites that each have a different layout, or when you simply don't want to maintain selectors. Reach for a classic Playwright scraper when you're hitting one stable board at high volume and the layout rarely moves — the per-page speed and zero token cost win there. Reach for a commercial scraping API when the wall is the problem, not the parsing — when you genuinely need rotating residential proxies and managed CAPTCHA solving against the hardest targets. Those vendors' exact proxy pools and detection logic are not fully publicly specified, and pricing shifts, so verify current plans before you commit.

BrowserBash deliberately does not bundle proxy rotation or CAPTCHA solving. That's an honest boundary: it drives your real Chrome, so it inherits your IP and your session, and it's at its best on public, accessible pages rather than aggressively defended ones. If your target is a Cloudflare-hardened board at scale, a dedicated scraping API is the better fit, and I'd say so plainly.

Where it runs: providers, scale, and recording

By default the browser runs locally in your own Chrome (--provider local). For most job scraping that's exactly what you want — it's free and the data never leaves your machine. But BrowserBash can point the same objective at other places the browser lives, which matters once you scale.

You can attach to any DevTools endpoint with --provider cdp --cdp-endpoint ws://..., which is handy if you already run a browser farm or a stealth Chrome behind a proxy. You can run on Browserbase's cloud browsers (--provider browserbase, with BROWSERBASE_API_KEY and BROWSERBASE_PROJECT_ID). And you can run on cross-browser grids like LambdaTest (--provider lambdatest) or BrowserStack (--provider browserstack), which automatically switch to the builtin engine. For pure data extraction you rarely need the grids, but the option is there if you want to scrape from a specific region or browser fingerprint.

Whatever provider you choose, the --headless flag runs Chrome without a visible window for unattended jobs, and --timeout <seconds> caps how long a single objective can run so a stuck infinite-scroll page can't hang your pipeline. Combine those with --agent and you have a scraper that's ready for a cron job.

The legal and ethical line you must not cross

This deserves its own section because scraping job listings sits in a genuinely contested space, and a tool that drives a real browser does not absolve you of responsibility.

The headline from U.S. case law: in hiQ Labs v. LinkedIn, courts found that scraping data that is publicly available — visible without logging in — does not by itself violate the Computer Fraud and Abuse Act. That is not a green light. Scraping public LinkedIn data still breaches LinkedIn's Terms of Service, and bypassing a technical barrier such as a login wall, a CAPTCHA, or an IP block can pull you back under CFAA exposure. The line that keeps you safe is roughly: public pages, no circumvention, no overload.

Practical rules I'd hold to when you scrape job listings:

• Stay on public pages. If the data requires an account to view, you're past the line that case law protects, and you're squarely into ToS-violation territory.
• Don't defeat anti-bot measures. BrowserBash gives you no CAPTCHA solver and no proxy rotation, and that's fine — if a board is actively blocking you, the answer is to slow down or stop, not to break through.
• Respect rate limits. Indeed starts challenging you past about one request per second. Build pauses into your runs. A polite scraper that collects 200 rows over ten minutes beats an aggressive one that gets your IP blocked in thirty seconds.
• Honor robots.txt and the site's stated terms as a baseline of good faith, even where they aren't strictly law.
• Mind personal data. Job postings can contain recruiter names and emails. If you're operating in the EU or UK, GDPR applies to that personal data regardless of how "public" it was. Collect only what you need.

None of this is legal advice — rules vary by jurisdiction and change over time, and you should check current law for your situation as of 2026. But the engineering posture that keeps you on the right side of it is simple: behave like a respectful human visitor, because that's exactly what an AI browser agent is emulating anyway.

A realistic end-to-end workflow

Putting it together, here's how a sane recurring job-scraping pipeline looks with this tool.

Start by validating the objective interactively against one page, with --record on, so you can watch the agent and confirm every field comes back correctly. Tune the English until the rows are clean — adjust your null-handling, lock your date format, scope the count. Once the objective is solid, move it into a committable markdown test so it lives in version control and your team can review what's being collected. Then switch to --agent and --headless for the scheduled run, wrap it in a script that reads the NDJSON final_state and appends rows to your store, and lean on the exit codes to decide whether each run succeeded.

For volume, stay polite: scrape one board per run, add a real delay between runs, and keep counts modest per pass. Every run is automatically saved on disk at ~/.browserbash/runs with secrets masked and a cap of 200 entries, so you have an audit trail of exactly what you collected and when. If you later want a shared view of runs across a team, the optional cloud dashboard exists — browserbash connect --key bb_... and then --upload per run — but it is strictly opt-in. Without --upload, nothing leaves your machine, which for scraped employment data is usually the posture you want.

If you're weighing whether this approach fits a specific board, the case studies and the pricing page (the CLI itself is free and Apache-2.0) are good next stops, and the full blog has more extraction walkthroughs.

FAQ

Is it legal to scrape job listings?

Scraping publicly visible job listings — pages you can see without logging in — has been treated by U.S. courts as not violating the Computer Fraud and Abuse Act, following the hiQ v. LinkedIn case. That said, it can still breach a site's Terms of Service, and circumventing login walls, CAPTCHAs, or IP blocks can create real legal exposure. If the postings include personal data and you operate under GDPR, those rules apply regardless of how public the page is. This is not legal advice; check current law for your jurisdiction.

Can I scrape job listings without writing selectors or code?

Yes. With an AI browser agent you describe the row you want in plain English — the fields, how to handle missing values, the date format — and the agent reads the rendered page and returns structured rows. There are no XPath or CSS selectors to write or maintain, which is the main reason this approach survives layout changes that break traditional scrapers. You only write code if you want a wrapper around the NDJSON output.

Will an AI scraper get past Cloudflare or LinkedIn's anti-bot walls?

No, and you should be wary of any tool that claims it reliably does. BrowserBash drives your own real Chrome and ships no CAPTCHA solver or proxy rotation, so it inherits your IP and session and is best on public, accessible pages. For aggressively defended boards at scale, a dedicated scraping API with managed proxies is the better fit. The safe and durable approach is to scrape public data politely and slowly rather than trying to break through defenses.

How do I get the scraped jobs into a CSV or database?

Run with the --agent flag, which outputs NDJSON — one JSON object per line with progress events and a final run_end event whose final_state holds your extracted rows. A short script reads that last line, pulls the rows, and appends them to a CSV or inserts them into a database, and the process exit codes tell it whether the run passed. Because the output contract is fixed, you never have to parse prose to get your structured data.

Get started

Scraping job listings doesn't have to mean a fragile pile of selectors you babysit forever. Describe the row you want, let an AI agent read the page like a person, and collect clean structured data — for free, on your own machine, with nothing leaving it unless you choose.

npm install -g browserbash-cli

BrowserBash is free and open-source, and no account is required to run it. If you'd like the optional free cloud dashboard later, you can sign up here — but it's entirely optional.