Four famous software failures and what they teach us about QA

When the US stock market opened on 1 August 2012, Knight Capital's trading system started sending orders it was never meant to send. In about 45 minutes it traded more than 397 million shares. Knight lost over $460 million, according to the SEC decision.

Software now runs inside banking, healthcare, transport, and communication. When it fails there, services stop, companies lose money, and people can get hurt.

This article looks at four failures: Knight Capital, CrowdStrike, Ariane 5, and Therac-25. It draws on the SEC decision, CrowdStrike's own root cause analysis, the Ariane 5 inquiry board report, and Leveson and Turner's paper on Therac-25. It keeps what those sources found separate from our reading of them, and it ends with what QA teams can do about it.

Knight Capital missed one server

Knight was rolling out new software for its trading system. The new code was installed on only seven of its eight servers. The eighth still held an old function called Power Peg, and the new release reused a flag that switched that old function back on. When trading opened, the eighth server started generating orders without properly tracking the ones already filled.

The SEC found that Knight had no written procedure requiring a second person to check the deployment, so the missing update went unnoticed. Before the market opened, its systems also sent 97 automated error messages. Nobody treated them as alerts. The SEC also found no effective automated controls to stop abnormal trading once it started.

CrowdStrike's tests did not cover the trigger

In July 2024, a faulty content update crashed Windows computers and disrupted businesses in many industries. According to CrowdStrike's own analysis, the new template defined 21 input fields, but the code that fed the content interpreter supplied only 20 values. The tests did not cover that condition, and the validation process missed it before the update went out.

One limit applies here: CrowdStrike wrote this report about its own failure, so it is not an independent finding.

Ariane 5 and Therac-25 trusted what had worked before

Both of these older cases involved software carried over from an earlier system.

• Ariane 5. The inquiry board found that the rocket broke apart shortly after its 1996 launch. Software reused from Ariane 4 hit a numerical conversion error under flight conditions it was not built for, and it had not been properly checked against the new rocket.
• Therac-25. Leveson and Turner studied six accidents between 1985 and 1987 in which patients received massive radiation overdoses. The machine reused software from earlier models but dropped the hardware safety interlocks those models had. They point to flaws in the software design, weak safety mechanisms, and too much confidence in the software's reliability.

What we take from these four cases

This section is our reading of the reports. None of them states it as a finding.

In each case, the software met a condition its tests or reviews had not covered. Ariane 5 flew a different trajectory than the one its reused code was built for. CrowdStrike's tests did not include the input that triggered the crash.

Passing tests under familiar conditions says little about what happens once the environment changes.

When software is reused, updated, or moved to a new setting, the assumptions behind the old tests need to be tested again.

Coverage was only part of the problem. Knight released new code without confirming it reached every server. Therac-25 relied on software to do a job that hardware had done before. QA covers more than testing. It also covers how changes are released, how a system reacts to unexpected behavior, and whether any safeguard works without depending on the software being right.

Four steps follow from this:

1. Test high-risk scenarios first, instead of relying on expected user behavior.
2. Run regression and integration tests whenever the software or its environment changes.
3. After every deployment, check what is actually running in production, and make sure someone owns each automated warning.
4. Set limits that stop a failure from spreading.

A useful first check for a QA team is to list the assumptions behind its existing tests and mark which ones have changed since the tests were written. In the Ariane 5 and CrowdStrike cases, that check might have pointed to the gap before release.

Knight's loss started with one server that nobody checked. The 97 error messages sent before the market opened gave a second chance to catch it. Each of the four cases had a point like this, where a check could have stopped the failure and either did not exist or was ignored.

QA teams cannot prevent every defect, but they can decide which of these checks are in place before the next release.

References

• Securities and Exchange Commission. (2013). Order instituting administrative and cease-and-desist proceedings against Knight Capital Americas LLC. Release No. 70694.
• CrowdStrike. (2024). Channel File 291 incident root cause analysis.
• Inquiry Board. (1996). Ariane 5 Flight 501 failure. European Space Agency and CNES.
• Leveson, N., & Turner, C. (1993). An investigation of the Therac-25 accidents. IEEE Computer, 26(7), 18–41.