Playwright checks that keep AI fixes from breaking the store

Step 1 of 6

My backlog was about 400 static-analysis findings, and an agent could fix them faster than anyone on the team. But "the linter is happy" and "checkout still works" are different claims. One of those findings is why the suite exists: a "cleaned-up" price calculation that passed every linter and left the checkout total wrong. It was caught late, and that was the moment I decided a test would catch the next one.

So before I let Claude Code loose on the rest, I built a small Playwright suite that answers one question after every commit: does the Adobe Commerce (Magento 2) store still work? Setup takes about an hour.

TL;DR

  • A fix that passes the linter can still break checkout. One "cleaned-up" price calculation left the total wrong.
  • Five journey tests cover it: home, search, add to cart, checkout to the payment step, admin login.
  • A screenshot comparison catches .phtml layout breaks; a listener fails the test on JS errors and 5xx responses.
  • The agent runs phpstan and Playwright after every commit, reverts on failure, and may never update screenshot baselines.

You’ll need

  • A local or staging store with sample data
  • Node.js 20 or newer
  • A list of the customer journeys that make money: search, product page, cart, checkout, login

Install Playwright inside the project

Keep the checks in the same repository as the code, so the agent can run them without extra setup.

npm init playwright@latest -- --quiet --lang=ts --browser=chromium

Test journeys, not pages

A useful check follows a customer, not a URL. Start with five: home loads, search finds a known product, the product page adds to cart, checkout reaches the payment step, and an admin can log in.

import { test, expect } from '@playwright/test';

test('guest can reach the payment step', async ({ page }) => {
  await page.goto('/catalogsearch/result/?q=bag');
  await page.getByRole('link', { name: /Joust Duffle Bag/ }).first().click();
  await page.getByRole('button', { name: 'Add to Cart' }).click();
  await expect(page.getByRole('alert')).toContainText('You added');
  await page.goto('/checkout/');
  await expect(page.getByText('Shipping Address')).toBeVisible({ timeout: 20_000 });
});

The payment step has a failure of its own: why a payment redirect can log the customer out.

Tip: Select by role and visible text, not CSS classes. Refactors rename classes far more often than they change what a customer reads.

Snapshot the templates

Plenty of "safe" fixes touch .phtml files. A screenshot comparison catches the layout break a functional assertion walks straight past.

test('product page looks the same', async ({ page }) => {
  await page.goto('/joust-duffle-bag.html');
  await expect(page).toHaveScreenshot('pdp.png', { maxDiffPixelRatio: 0.01, fullPage: true });
});

Record the baseline once with --update-snapshots on a build you trust.

Fail on errors the page hides

Broken JavaScript often leaves a page that looks fine and does nothing. Fail the test when the browser throws, or when any request returns a 5xx. Collect the problems in a listener and assert at the end; an error thrown inside the listener itself doesn't reliably fail the test.

let problems: string[] = [];

test.beforeEach(async ({ page }) => {
  problems = [];
  page.on('pageerror', (err) => problems.push(`JS error: ${err.message}`));
  page.on('response', (res) => {
    if (res.status() >= 500) problems.push(`${res.status()} on ${res.url()}`);
  });
});

test.afterEach(() => {
  expect(problems).toEqual([]);
});

Put the suite inside the agent's loop

Spell out the definition of done in CLAUDE.md. A finding isn't fixed until the checks pass:

## Definition of done for static-analysis fixes
- Fix one rule at a time, in small commits.
- After each commit run: vendor/bin/phpstan analyse && npx playwright test
- If a Playwright test fails, revert the commit and report the finding as "needs human".
- Never update screenshot baselines yourself.

Tip: The last line is the one that matters. An agent allowed to update baselines can "fix" any visual regression by accepting it.

On a legacy codebase, give phpstan a baseline file of the errors it already has, so only new errors fail the check.

Run it on every pull request

CI runs the same suite against a fresh build with sample data and uploads the trace when something fails. npx playwright show-trace then replays exactly what broke, step by step, which settles "real regression or flaky test?" in under a minute.

Done!

Every AI-assisted fix now has to prove the store still works. Next: App Builder for Commerce developers, from Episode 0.

Written by Vishwas Bhatnagar

Technical Architect. Adobe Commerce, marketplaces and AI-assisted engineering.

More about me →

More