When to automate a test, and when not to

Most teams automate the wrong tests first. They pick the ones that are easy to automate, which are rarely the ones worth automating, and eighteen months later they own a slow suite that covers the settings page and misses checkout.

The decision is not "can this be automated". Almost anything can. The decision is whether it should be.

The cost is not the writing

A test costs you three times. Once when it is written, once every time the feature changes and it has to be updated, and once every time it fails for a reason that is not a bug and somebody has to find out why.

The first cost is the small one. Teams estimate the first cost, commit to the suite, and then pay the other two for years.

An automated test is not an asset. It is a subscription.

Four questions

Ask them in order. A case has to pass all four.

1. Will this run more than ten times? Below that, a person is cheaper. Above it, the machine wins and keeps winning.
2. Is the outcome unambiguous? A test that asserts "the layout looks right" cannot be automated. A test that asserts the order total equals 4200 can.
3. Is the thing it tests stable? Automating a flow that is being redesigned next sprint means writing the test twice. Wait.
4. Does the failure mean something? If a red result would be investigated by nobody, the test is not covering risk. It is producing noise.

What stays manual, on purpose

Some checks are cheaper and better with a person, and there is no shame in a checklist:

• Anything visual. Spacing, contrast, a logo that stretches on a narrow screen.
• Anything that needs judgement, like whether an error message actually helps.
• One-time checks after a migration, where the answer matters once and never again.
• New features that are still moving. Test them by hand until the shape settles.

A worked example

Here is the same check written both ways. The manual version is a line in a checklist. The automated version is worth writing because it will run on every pull request for two years:

// worth automating: runs on every deploy, unambiguous, covers money
test("annual plan applies the yearly discount", async () => {
  const cart = await addPlan("pro", { billing: "annual" });
  expect(cart.total).toBe(4200);
});

The rule that produced it is the first question, not the third. It runs on every deploy, so it crosses ten runs in a week.

Review the suite twice a year

Automation decisions expire. A flow that changed every sprint two years ago is stable now and should be covered. A test written for a feature that was removed is still running, still slow, and still green.

Put an hour in the calendar every six months and delete things. A suite nobody prunes grows until its runtime is the reason people stop trusting it, which is the same failure as a suite full of flaky tests, arriving more slowly.