Agentic QA: letting an LLM drive the regression suite

An agent running a test suite is not a robot QA engineer. It is a very fast, very literal colleague who will happily assert the wrong thing forever. The value shows up only once you decide what it is allowed to conclude on its own.

I gave the harness three jobs: run the suite, cluster the failures, and write a reproduction for each cluster. Anything past that — deciding whether a failure is a product bug or a test bug — still comes back to me with evidence attached.

What held

Clustering was the surprise. Twelve red tests routinely collapse into two causes, and reading two write-ups instead of twelve changed how quickly I could act on a red build.

The assertion helper below is the whole contract the agent is allowed to write against:

test('logs in with a valid session', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill(user.email);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { level: 1 })).toHaveText('Dashboard');
});

Guardrails I keep

1. The agent never edits an assertion — only the setup around it.
2. Every filed bug carries a failing trace, or it does not get filed.
3. A cluster with no reproduction is escalated, not retried.

Six months in, the suite is slower to change and much faster to trust. That trade has been worth it on every release.