Hey everyone, Alex here from Coding with Alex. If you’ve spent any time building web scrapers, automated testing suites, or browser-based RPA (Robotic Process Automation) tools lately, you’ve likely hit a wall. Tools like Puppeteer, Playwright, and Selenium are fantastic when you know exactly what the DOM looks like. But the moment you try to hook them up to an LLM to build an autonomous "browser agent" that can navigate the web on its own, things get incredibly messy, slow, and expensive.
Traditional LLM-based web agents usually work by grabbing the entire DOM tree, converting it to markdown or a stripped-down HTML string, and shoving that massive blob of text into the context window of a frontier model like GPT-4. The model then has to figure out which element to click, write a selector, and pass it back. This approach is plagued by three massive bottlenecks: high latency, eye-watering API costs, and fragile selectors that break the moment a dynamic single-page app (SPA) updates its state.
That’s why the developer community on Hacker News has been buzzing about Jev Ultrafast, an open-source browser agent architecture that leverages a dynamic, indexed action space. Today, we’re going to tear down how this architecture works, why "indexed action spaces" are the future of web automation, and how you can implement these concepts in your own developer workflow.
The Core Problem: The Context Window Tax and Action Space Bloat
Before we look at the solution, we need to understand why current AI browser agents are so painfully slow. Let's look at a standard agent loop:
[Browser Session]
│
▼ (Extract DOM: 150kb of HTML text)
[LLM Context Window]
│
▼ (Reasoning: "I need to click the search bar")
[Generate Selector: #search-input-dynamic-39df]
│
▼ (Execute via Playwright)
[Click Element]In this loop, there are two major issues:
- The DOM is noisy: A typical modern web page contains thousands of nodes. 95% of them are non-interactive divs, spans, and style wrappers that the LLM doesn't need to know about. Shoveling this into the context window is like asking someone to find a specific word in a dictionary by reading the entire book cover-to-cover every time.
- The Action Space is infinite: If you tell an LLM it can write "any JavaScript" or "any CSS selector" to interact with a page, its search space of possible actions is infinitely large. This leads to hallucinations, syntax errors, and failed selector matches.
What is a "Dynamic, Indexed Action Space"?
The breakthrough in Jev Ultrafast is the shift from an open-ended action space to a dynamic, indexed action space. Instead of asking the AI to inspect the DOM and invent its own way to click a button, the system preprocessing step does the heavy lifting:
- It filters the DOM in real-time to extract only interactive elements (buttons, inputs, links, dropdowns).
- It assigns a temporary, unique numerical index to each interactive element (e.g.,
[1],[2],[3]). - It visually superimposes these index badges onto the viewport (or injects them cleanly into a highly minimized representation of the page).
- The LLM’s action space is restricted to a simple set of commands like
click(index),type(index, text), orscroll().
Because the action space is restricted to these indexed primitives, the LLM doesn't need to write complex CSS selectors. It just says: click(4). If the page layout changes dynamically, the indexer re-runs, updates the indices, and hands the new state back to the agent. This results in execution speeds that are orders of magnitude faster (hence the name "Ultrafast").
Under the Hood: Building a Mini-Indexed Agent
Let's look at how we can implement a basic version of this architecture using Node.js and Playwright. We want to write a script that injects indices into the active web page and extracts a clean, minimal "action map" for our LLM.
Step 1: The Interactive Element Filter
First, we need a client-side script that runs inside the browser context to identify interactive elements, assign them IDs, and return their positions and metadata. Here is a robust snippet to achieve this:
// clientSideIndexer.js
function getInteractiveElements() {
const interactiveSelectors = [
'a[href]', 'button', 'input', 'select', 'textarea',
'[role="button"]', '[role="link"]', '[tabindex]:not([tabindex="-1"])'
];
const elements = document.querySelectorAll(interactiveSelectors.join(','));
const actionMap = [];
let index = 0;
elements.forEach((el) => {
// Filter out hidden elements
const rect = el.getBoundingClientRect();
if (rect.width === 0 || rect.height === 0 || window.getComputedStyle(el).display === 'none') {
return;
}
index++;
// Add a visual badge to the DOM for visual/multimodal models
const badge = document.createElement('div');
badge.innerText = index;
badge.style.position = 'absolute';
badge.style.backgroundColor = 'red';
badge.style.color = 'white';
badge.style.fontSize = '10px';
badge.style.fontWeight = 'bold';
badge.style.padding = '2px 5px';
badge.style.borderRadius = '3px';
badge.style.zIndex = '100000';
badge.style.top = `${window.scrollY + rect.top}px`;
badge.style.left = `${window.scrollX + rect.left}px`;
badge.style.pointerEvents = 'none'; // Ensure it doesn't block clicks
document.body.appendChild(badge);
// Map internal index to the actual DOM element for execution
el.setAttribute('data-agent-index', index);
actionMap.push({
index: index,
tagName: el.tagName.toLowerCase(),
type: el.type || null,
text: el.innerText?.trim().substring(0, 30) || el.placeholder || '',
role: el.getAttribute('role') || 'generic'
});
});
return actionMap;
}Step 2: Orchestrating the Agent Loop
Now, let’s wrap this in a Playwright script. This controller will launch the browser, run our indexer, generate a highly compact state description, and present it to an LLM. Notice how tiny the input token size becomes!
import { chromium } from 'playwright';
async function runAgentStep(page, instruction) {
// 1. Inject indexer and get current action map
const actionMap = await page.evaluate(getInteractiveElements);
// 2. Format the action map into a highly condensed string
const formattedState = actionMap.map(item => {
return `[${item.index}] <${item.tagName} type="${item.type || ''}"> "${item.text}"`;
}).join('\n');
console.log("--- CURRENT ACCESSIBLE ACTION SPACE ---");
console.log(formattedState);
console.log("---------------------------------------");
// 3. Construct a highly optimized prompt
const prompt = `
You are an autonomous browser agent. Your goal is to: "${instruction}"
Current Page State:
${formattedState}
To interact, output ONLY one of the following commands:
- CLICK [index]
- TYPE [index] "[text]"
- FINISH
Your next action:`;
// Here you would call your LLM of choice (e.g., Anthropic Claude or OpenAI GPT-4o Mini)
// const response = await callLLM(prompt);
// For demonstration, let's mock the LLM choosing to click a "Login" button at index 3
const mockLLMResponse = 'CLICK 3';
// 4. Parse action and execute via native Playwright selectors targeting our stable index
const match = mockLLMResponse.match(/(CLICK|TYPE)\s+(\d+)(?:\s+"(.*)")?/);
if (match) {
const [_, action, targetIndex, payload] = match;
const selector = `[data-agent-index="${targetIndex}"]`;
if (action === 'CLICK') {
console.log(`Executing: Clicking element [${targetIndex}]`);
await page.click(selector);
} else if (action === 'TYPE') {
console.log(`Executing: Typing "${payload}" into [${targetIndex}]`);
await page.fill(selector, payload);
}
}
}Why This Method is a Game-Changer for Developers
If you've built agents using raw DOM trees, the code above should make you incredibly excited. Let's break down the mathematical and practical benefits of using this dynamic, indexed action space approach:
1. Drastic Reduction in Input Token Costs
A raw HTML DOM of a site like GitHub or Amazon can easily span 100,000 tokens. Using LLM APIs (even cost-effective ones like GPT-4o-mini), running a multi-step agent loop can cost several dollars per run. By filtering the page down to an indexed action map of just 30-50 interactive elements, you reduce the token footprint by up to 98%, cutting down operating costs to pennies.
2. Eliminating CSS Selector Fragility
Modern frontend frameworks (React, Vue, Tailwind CSS) generate randomized, dynamic class names (e.g., class="flex items-center justify-between p-4 md:p-6 bg-slate-900_34dfa"). If an LLM tries to generate selectors targeting these classes, they will inevitably break when the application code compiles. By mapping everything to a stable, runtime-generated integer index (e.g., [data-agent-index="4"]), we decouple the LLM's reasoning engine from the underlying styling and build toolchain.
3. Extreme Latency Improvements
Processing 100k tokens takes time—often 10 to 15 seconds of TTFT (Time to First Token) from standard LLM APIs. Processing 500 tokens takes milliseconds. By reducing the size of the prompt, the agent's reaction time transitions from a slow, batch-like feel to a snappy, near-real-time user experience.
Taking It Further: Multimodal Vision Agents
One of the coolest aspects of the Jev Ultrafast philosophy is how it integrates with multimodal models. Instead of sending a text description of the indices to the LLM, you can take a screenshot of the browser window with the red index badges visually painted over the UI elements, and pass that image directly to a vision model like Claude 3.5 Sonnet or GPT-4o.
The vision model looks at the labeled image, instantly locates the visual boundaries of the page, finds the button it needs, spots the big red "[12]" label sitting on top of it, and outputs the JSON: {"action": "click", "target": 12}. This entirely bypasses the need to translate the accessibility tree to text, providing the ultimate "What You See Is What You Execute" interaction loop.
Summary & Call to Action
The era of treating web browsers as giant text dumps for AI models is coming to an end. Projects like Jev Ultrafast are demonstrating that clever engineering of the action space—pre-filtering, indexing, and runtime mapping—is the key to making web agents viable for production environments.
If you're building automation tools, stop trying to feed raw HTML into your LLM wrappers. Experiment with client-side indexing. Try injecting state badges into your test runners, and let your AI agents act on clean, structured, integer-indexed maps.
What do you think? Have you tried building browser agents with LLMs? What are the biggest hurdles you've run into with dynamic SPA pages? Let's discuss in the comments below!
Until next time, happy coding!
— Alex