Hey everyone, welcome back to another post on Coding with Alex! If you’ve spent any time in a modern CI/CD pipeline, you know the absolute dread of seeing a red build artifact at 4:45 PM on a Friday. You open the logs, bracing yourself for a critical regression, only to find... a flaky test. Again.
Some minor CSS change shifted an element by three pixels, or a dynamic ID changed from button-qa-302 to button-qa-305, and your entire deployment pipeline ground to a screeching halt. We’ve spent years trying to solve this with better selectors, Page Object Models, and arbitrary timeouts, but the truth is: traditional End-to-End (E2E) testing is brittle by design because it relies on static assumptions about a dynamic UI.
Today, we are going to look past the marketing fluff of "AI-driven testing" and build something practical. We’ll explore how to leverage Large Language Models (LLMs) and modern browser automation tools like Playwright to create a self-healing test engine. By the end of this post, you'll see how to write tests that don't just break when the DOM changes, but actively reason about the page to find the right elements and repair themselves on the fly.
The Flaky Selector Problem
Before we dive into the code, let's look at why our current testing paradigms fail. Consider a standard login button. In your test suite, you might target it using one of these common strategies:
// Exhibit A: The fragile XPath
await page.locator('xpath=//*[@id="root"]/div/div[2]/form/button').click();
// Exhibit B: The slightly better class selector
await page.locator('.btn-primary-active').click();
// Exhibit C: The modern data-testid
await page.getByTestId('submit-login-form').click();While Exhibit C is the industry best practice, it still fails in real-world scenarios. What happens when a marketing team installs an A/B testing tool that overrides the button? What if your frontend team migrates from React to Tailwind UI, and the element structure changes entirely?
When these selectors fail, the test runner throws its hands up and crashes. But how does a human tester handle this? A human doesn't care if the ID changed. They look at the page, see a blue button that says "Sign In" near the password field, and click it. AI-driven testing aims to bring this human-like semantic understanding to our automated runners.
The Architecture of a Self-Healing Test Runner
To make a test self-healing, we need to introduce a fallback mechanism. If our standard, fast locator (like a data-testid) fails, instead of throwing an error immediately, we trigger an "AI Repair" lifecycle. Here is how the flow works:
Traditional Flow: [Run Test] ──> [Selector Fails] ──> [Build Crashes ❌]
Self-Healing Flow: [Run Test] ──> [Selector Fails] ──> [Capture DOM Snapshot & Screenshot] ──> [LLM Analyzes Page] ──> [LLM Resolves New Selector] ──> [Execute Action] ──> [Log Healed Selector for Devs ✅]
By using this hybrid approach, we keep our tests incredibly fast (using standard selectors 99% of the time) but highly resilient (using the LLM only when things break).
Step-by-Step: Implementing Self-Healing in Playwright
Let's build a lightweight self-healing wrapper around Playwright using TypeScript and the OpenAI API. We will write a custom helper function that attempts to click an element, and if it fails, falls back to our AI locator.
Step 1: Setting up the LLM Client
First, we need to create a utility that sends the state of our page to an LLM. Sending the entire raw HTML of a modern web page to an LLM is expensive, slow, and often exceeds token limits. Instead, we must extract a simplified representation of the DOM—focusing only on interactive elements (buttons, inputs, links, text areas).
import { Page } from '@playwright/test';
import { OpenAI } from 'openai';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
// Extract a clean, minimal DOM tree of interactive elements
async function getInteractiveDOM(page: Page): Promise<string> {
return await page.evaluate(() => {
const interactiveElements = document.querySelectorAll('button, input, a, select, [role="button"]');
const nodes = Array.from(interactiveElements).map((el, index) => {
return {
index,
tagName: el.tagName.toLowerCase(),
id: el.id || undefined,
className: el.className || undefined,
text: el.textContent?.trim().substring(0, 50) || undefined,
placeholder: (el as HTMLInputElement).placeholder || undefined,
type: (el as HTMLInputElement).type || undefined,
role: el.getAttribute('role') || undefined,
'data-testid': el.getAttribute('data-testid') || undefined
};
});
return JSON.stringify(nodes, null, 2);
});
}Step 2: Asking the AI for the Correct Element
Now, we create the reasoning engine. When a selector fails, we pass the simplified DOM JSON and a natural language description of what we wanted to achieve (e.g., "click the submit button") to the LLM. We'll enforce a JSON response containing the most likely index of the element we want.
interface AIResolution {
selectedIndex: number;
confidence: number;
reasoning: string;
}
async function resolveElementWithAI(
page: Page,
userIntent: string,
domSnapshot: string
): Promise<number> {
const prompt = `
You are an automated QA engineer assistant.
We tried to perform the following action: "${userIntent}", but our selector failed.
Here is a simplified JSON snapshot of the interactive elements on the current page:
\`\`\`json
${domSnapshot}
\`\`\`
Analyze the elements and identify which index best matches the user's intent.
Return your response strictly as a JSON object with this structure:
{
"selectedIndex": number,
"confidence": number (between 0 and 1),
"reasoning": "short explanation of why you chose this element"
}
`;
const response = await openai.chat.completions.create({
model: 'gpt-4o-mini', // Fast and highly accurate for structured JSON tasks
messages: [{ role: 'user', content: prompt }],
response_format: { type: 'json_object' }
});
const result: AIResolution = JSON.parse(response.choices[0].message.content || '{}');
if (result.confidence < 0.7) {
throw new Error(`AI confidence too low: ${result.reasoning}`);
}
return result.selectedIndex;
}Step 3: Creating the Self-Healing Wrapper
Now we glue these parts together into a robust smartClick function. If a standard Playwright action fails, we catch the error, trigger our AI repair system, execute the action using the AI's resolved element, and log a warning to the developer telling them exactly what selector needs to be updated in the codebase.
export async function smartClick(page: Page, fallbackSelector: string, userIntent: string) {
try {
// Attempt normal execution first (fast path)
await page.locator(fallbackSelector).click({ timeout: 5000 });
} catch (error) {
console.warn(`⚠️ Selector "${fallbackSelector}" failed. Attempting self-healing recovery...`);
try {
// 1. Capture state
const domSnapshot = await getInteractiveDOM(page);
// 2. Consult the LLM oracle
const targetIndex = await resolveElementWithAI(page, userIntent, domSnapshot);
// 3. Resolve element in browser and click it
await page.evaluate((index) => {
const interactiveElements = document.querySelectorAll('button, input, a, select, [role="button"]');
const element = interactiveElements[index] as HTMLElement;
if (element) {
element.style.outline = '3px solid #ff007f'; // Visual debug cue
element.click();
} else {
throw new Error('Element resolved by AI no longer exists in DOM.');
}
}, targetIndex);
console.log(`✅ Recovery successful! Suggestion: Update "${fallbackSelector}" to match AI-resolved element properties.`);
} catch (recoveryError) {
// If AI healing fails, throw original error so the pipeline alerts us
console.error('❌ Self-healing failed.');
throw error;
}
}
}Putting It to the Test
How does this look in a real test? Let's say we have an e-commerce checkout page. In our test code, we write:
import { test } from '@playwright/test';
import { smartClick } from './utils/selfHealing';
test('User can complete checkout flow', async ({ page }) => {
await page.goto('https://sysseder-shop.com/cart');
// This class was renamed to .btn-checkout-v2 in production,
// but our test still points to .btn-checkout-submit!
await smartClick(
page,
'.btn-checkout-submit',
'Click the button to proceed to the secure checkout page'
);
});Under a normal Playwright execution, this test dies at step one. With our self-healing wrapper, the runner catches the error, looks at the DOM, sees a button with class .btn-checkout-v2 and text "Proceed to Checkout", clicks it, and keeps the build green while alerting you in the CI logs to fix the selector. That is a massive productivity boost!
The Trade-offs: When NOT to use AI in Testing
As developers, we must always maintain a healthy level of skepticism. AI-driven testing is incredibly powerful, but it isn’t a silver bullet. Here are the trade-offs you need to consider before integrating this into your production pipelines:
- Latency: Traditional locator checks take milliseconds. A call to an LLM API takes between 500ms and 2 seconds. You should only use AI as a fallback, never as your primary selector mechanism.
- Cost: Running LLM API queries on every test step in a large test suite will quickly blow up your cloud bill. Using self-healing only on failure mitigates this cost significantly.
- False Positives: If your application is genuinely broken (e.g., a button is missing entirely), the AI might try to be "too helpful" and click an unrelated button, masking a legitimate bug. Setting strict confidence thresholds (like our
confidence > 0.7check) is crucial to avoid this.
Wrapping Up: The Future of QA is Hybrid
AI-driven testing isn’t about replacing developers or QA engineers; it’s about removing the mundane, repetitive toil of maintaining fragile pipelines. By implementing hybrid strategies like the self-healing wrapper we built today, we can get the best of both worlds: the screaming-fast speed of native browser automation coupled with the resilient, semantic reasoning of machine learning.
Are you currently using any AI tools in your testing workflows? Have you experimented with writing custom LLM wrappers for your dev tools? Let me know in the comments below, or hit me up on Twitter/X at @sysseder!
Until next time, keep your builds green and your code clean!