Browser Automation Agent
An agent that drives a real browser to finish web tasks that have no API.
Overview
An automation that controls a real browser to do work you can't get at through an API: filling forms, pulling data from behind a login, running the same web task over and over. It reads the page like a person, works out the next step, and recovers when the site changes on it.
The problem
A lot of business software has no API. The only way to automate it is the browser, and the usual approach, a script full of hard-coded selectors, breaks the moment the site changes its markup.
So teams either do the same web task by hand forever, or they keep a pile of fragile scripts alive that need babysitting every week. I wanted automation that adapts instead of shattering.
The solution
I paired plain browser control with a reasoning layer. The system grabs the state of the page, decides the next action from what it's trying to do rather than a fixed selector, and checks the result before moving on. The deterministic steps stay fast, and the model only gets involved when the page is ambiguous.
Every run is easy to follow after the fact. Each step, screenshot, and decision is logged, so a failure is something you can actually debug, and a rerun is safe to repeat.
Architecture
A controller drives Puppeteer through a planned set of steps. A reasoning fallback handles the pages that don't match, and a check gates each step. Everything gets logged with screenshots.
- Actions based on intent, not brittle hard-coded selectors
- The model only reasons on ambiguous pages; the rest is deterministic
- Each step gets checked before the next one runs
- A full screenshot and action trace for every run
Screens & diagrams
Key features
Heals itself
When a selector fails, it reasons about the page to find the right element instead of falling over.
Checks its work
Confirms each action landed against the expected page state before it moves on.
Full run traces
Screenshots and a decision log for every step, so failures are easy to read.
Safe to rerun
Completed steps get detected and skipped, so a retry doesn't redo work.
Implementation highlights
- Kept the model out of the hot path. Deterministic control handles about 90 percent of steps, and reasoning is the fallback, which keeps runs fast and cheap.
- Built a screenshot-and-trace layer that turned opaque failures into something you can debug at a glance.
- Added a worker queue with backoff, so long jobs survive restarts and rate limits.
- Put the runtime in a container so headless runs are consistent and repeatable.
Challenges solved
Sites that change on you
Finding elements by intent, with a reasoning fallback, kept flows alive through markup changes that would break a selector script.
Cost and speed
Only calling the model when the page is unclear kept most steps deterministic, which is both faster and cheaper.
Debugging the headless
A headless failure tells you nothing on its own, so the screenshot trace was what made every run reproducible.
Technologies
Future improvements
- Visual diffing to spot and adapt to layout changes early
- A recorder that turns a manual walkthrough into a reusable task
- Running many jobs in parallel for high volume