Regression suites that stay green

Every automated suite starts green. The interesting question is what it looks like in eighteen months, and for most teams the answer is: mostly green, occasionally red for reasons nobody investigates, and re-run until it passes.

At that point the suite has stopped being a safety net. It is a delay.

A flaky test is a broken test

The single most damaging habit a team can pick up is re-running a failed build to see if it passes the second time. It works, which is the problem. It works often enough that it becomes the reflex, and the reflex is indistinguishable from ignoring a real failure.

If a red build does not stop the release, the suite is decoration.

So the rule has to be absolute: a test that fails intermittently is quarantined the same day. Not fixed the same day. Quarantined. Move it out of the blocking suite, open a ticket, and keep the signal from the tests that still mean something.

Where flakiness actually comes from

In our experience it is almost never the test framework. It is four things, in this order:

• Timing. A fixed sleep(2000) that passes on a fast machine and fails in CI under load.
• Shared state. Two tests that both write to the same user, and pass only in the order they happen to run.
• Real races in the product. The most valuable kind. The test is not flaky; the app is, and the test found it.
• The environment. A dependency that is up 99% of the time, which at 400 test runs a week is a red build every other day.

The third one is worth dwelling on. A test that fails one time in twenty may be reporting a genuine race condition that hits one user in twenty. Quarantining it and moving on throws away the finding. Quarantine, then investigate. The ticket is the point, not the quiet.

Wait for state, never for time

Most timing flakiness disappears with one change: stop waiting for durations, start waiting for conditions.

// flaky: passes on a fast machine, fails in CI under load
await page.click("#save");
await sleep(2000);
expect(await page.textContent("#status")).toBe("Saved");

// stable: waits for the thing you actually care about
await page.click("#save");
await page.waitForSelector("#status:has-text('Saved')", { timeout: 10_000 });

The second version is faster on a fast machine and more reliable on a slow one, which is the rare change that costs nothing. A generous timeout is not a slow test. It is an upper bound that is almost never reached.

Keep the blocking suite small

The last piece is scope discipline. Not every test deserves to block a deploy.

Split the suite in two. A small blocking set covers the flows where a failure means real money or real data loss: sign-up, checkout, permissions, anything that writes. It runs on every pull request, it takes under ten minutes, and it is allowed zero known-flaky tests. Everything else runs on a schedule, reports into the team channel, and blocks nothing.

Teams resist this because it feels like lowering the bar. It is the opposite: it is the only way a red build keeps meaning "stop". A suite that everyone trusts and that covers eighty percent of the risk is worth more than a suite that covers everything and that nobody believes.

If you want to see the shape of the reports this produces, how we work has the real artifacts from live engagements.