A taxonomy of failure, not a list of anecdotes

Most training on AI failure is a collection of stories. Stories are memorable and useless for planning, because the next failure will not be one of them. What you need is a taxonomy: a small number of categories that differ in cause, in who notices, and in what control addresses them. Six will do.

1. Fabrication

The system produces content that is fluent, specific and untrue: a statutory article that does not exist, a figure with no source, a case that was never decided. This is not a malfunction. A generative model selects plausible continuations, and plausible wrong detail is an expected output of that process.

Who notices: only someone who checks against a source. Control: verification of every specific claim, and a rule that the system supplies material rather than recalling it.

2. Distribution shift

The model was right about the world it was trained on and the world moved. Performance degrades without any change to the model, and without any error message, because producing an output on an unfamiliar input is all the model knows how to do.

Who notices: nobody, unless performance is monitored per segment over time. Control: post-market monitoring under Article 72 of the EU AI Act, based on a documented plan, with thresholds defined in advance as Article 9 requires.

3. Bias

Performance differs systematically between groups, usually because the training data represented them unequally or recorded a historical practice as though it were a fact about the world. An aggregate accuracy figure will not show it.

Who notices: the affected group, usually long before the organisation. Control: the data governance duties in Article 10, which require examination for bias and measures to detect, prevent and mitigate it, and per-group measurement rather than totals. Where a bias audit is mandated, as under NYC Local Law 144, note that the duty is to calculate and publish selection rates, scoring rates and impact ratios, not to reach a particular figure.

4. Specification failure

The system does exactly what it was asked and what it was asked was wrong. A model optimised to predict which claims will be disputed is not a model that identifies which claims deserve scrutiny, and the two diverge quietly. This failure is invisible in every technical metric, because by the metric the system is performing well.

Who notices: someone who compares the objective to the purpose, which is nobody's job unless it is made someone's job. Control: stating the intended purpose precisely and revisiting it, which is also what determines classification under the Act.

5. Adversarial failure

Someone makes the system fail deliberately. Prompt injection places instructions inside material the system will read; data is manipulated to move an outcome; inputs are crafted to be misclassified. Article 15 of the EU AI Act requires high-risk systems to be resilient to attempts to alter their use, outputs or performance. The OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026 name the concrete forms.

Who notices: nobody, by design; a successful attack looks like normal operation. Control: treating all retrieved and user-supplied content as untrusted, and logging sufficient to reconstruct a sequence after the fact.

6. Oversight failure

The system was wrong, a person reviewed it, and the person agreed. This is the failure that makes the other five consequential, and it has a name in the Act: Article 14(4) requires those exercising oversight to remain aware of automation bias, the tendency to over-rely on output from an automated system. Approving at volume is where it lives. A reviewer given ninety seconds and a confident summary will approve, and the control that was counted on in the risk assessment quietly contributes nothing.

Who notices: an auditor, much later. Control: oversight designed around a realistic unit of review, with someone who has the competence, training and authority the Act requires, and a rate at which real review is possible.

Why the taxonomy is worth the effort

Each category is caught by a different control, and none of the controls catches the others. Monitoring will never detect fabrication. Verification will never detect drift. A fairness test will not reveal that you measured the wrong objective. When someone says a system has been tested, the useful question is which of the six the test addressed, and the usual honest answer is one.

As of 24 September 2026.