What an accuracy number actually claims

Every model arrives with a number. Ninety-four per cent accurate. An area under the curve of 0.89. A vendor slide with one figure in large type. The number is usually true and almost always misunderstood, and the misunderstanding is the single most common way an organisation ends up relying on something it should not.

Where the number comes from

Before training, the available data is split. Most of it trains the model. A held-back portion, the test set, is kept away from training entirely and used once at the end to measure performance. Measuring on data the model was trained on would be marking your own homework: a model can memorise, and a memorised answer is not a prediction.

So the number means something specific. It is the rate at which the model was right about a sample of past cases drawn from the same source as the cases it learned from. That is a claim about the past, and it is a claim about a particular past, in a particular place.

Why it stops being true

The claim holds only while the cases the model meets resemble the cases in that sample. That resemblance decays for reasons that have nothing to do with the model, which is why nobody notices.

The population changes: you open in a new canton, a new product attracts a different sort of customer, a competitor leaves the market and their customers arrive. The process changes: a form is redesigned and a field that used to be free text becomes a dropdown, so a signal the model leaned on flattens. The world changes: prices move, a regulation alters what people claim for, a pandemic rewrites a year of behaviour. Or the system is pointed at a group it never saw, which is the most abrupt version of the same thing.

The model will not complain. It produces confident outputs on inputs unlike anything in its training data, because producing an output is all it does. Degradation is silent by construction, and the original accuracy figure keeps appearing in decks long after it stopped describing anything.

What the number never told you

An aggregate figure hides its own distribution. A model that is 94 per cent accurate overall can be 97 per cent accurate on the large group that dominates the test set and substantially worse on a small one, and the aggregate will not move enough to show it. This is why Article 10 of the EU AI Act asks for data that is sufficiently representative and examined for bias rather than merely plentiful, and why it directs attention to the specific geographical, contextual, behavioural or functional setting in which the system will be used.

An aggregate figure also says nothing about which errors it is making. For most consequential uses, the two directions of error are not equally bad. Wrongly fast-tracking a fraudulent claim and wrongly declining an honest one are both errors, and they land on different people with different consequences. A single accuracy figure treats them as interchangeable. They never are.

What the law asks you to do about it

The Act treats the number as a starting point rather than a result. Article 9 requires risk management to be a continuous iterative process across the lifecycle, with testing against metrics and probabilistic thresholds defined in advance. Defined in advance is the operative phrase: you decide what good enough means before you see the result, which is what stops the threshold being drawn around whatever the model happened to achieve.

Article 15 requires accuracy, robustness and cybersecurity appropriate to the system. Article 72 requires a documented post-market monitoring system, proportionate, actively and systematically collecting and analysing performance data across the system's lifetime, based on a plan. Article 13(3) requires the provider's instructions for use to state the system's capabilities and performance limitations, including the accuracy metrics it was validated against. When you buy, that is the document to ask for, and the metrics in it are the ones you are entitled to hold the vendor to.

The financial sector reaches the same place by another route. FINMA Guidance 08/2024 names robustness and reliability as one of its four themes and expects testing before deployment and continuous monitoring afterwards, and it is explicit that responsibility remains with the institution even where the AI is bought in or outsourced. The US interagency model risk guidance SR 26-2, which superseded SR 11-7 in April 2026, carries forward validation covering conceptual soundness, ongoing monitoring and outcomes analysis.

Different instruments, one instruction: the number is a hypothesis about the future, and monitoring is the experiment that keeps testing it.

As of 24 September 2026.