Three doors bias comes in through

Bias in an AI system is not usually a moral failing on anyone's part. It is an accounting outcome. A model reproduces the structure of what it was shown, and what it was shown was assembled by people under deadline, for a purpose that was not fairness. Knowing which door the bias came in through tells you which fix has any chance of working.

Door one: the data

A training set is a sample of the world, and every sample was drawn by some process. If loan applications from one region were historically discouraged before they were ever filed, the training set contains few of them, and the model has learned almost nothing about that region except that it is unusual. If a bank's records of fraud contain only the fraud that was caught, the model learns to find the kind of fraud that investigators already knew how to find.

This is why the EU AI Act, in Article 10, asks for more than a file of rows. For high-risk systems it requires you to record where the data came from and what it was originally collected for, the assumptions you made, and its availability, quantity and suitability. It requires training, validation and testing data to be relevant, sufficiently representative, and to the best extent possible free of errors and complete, with appropriate statistical properties. It also requires the data to be examined for possible biases, with measures to detect, prevent and mitigate them. Article 10(4) adds that the data must account for the geographical, contextual, behavioural or functional setting the system will be used in. A dataset that is fine for Zurich is a different dataset once the product ships in Lisbon.

There is a trap here that catches well-meaning teams. To test whether a model treats groups differently, you generally need to know who is in which group, and that information is usually special category data. Article 10(5) permits processing special categories of personal data for the purpose of bias detection and correction, but only where it is strictly necessary and subject to safeguards. The permission is narrow and conditional. It is not a licence to collect ethnicity across the customer base because fairness sounds like a good reason.

Door two: the labels

The second door is quieter and does more damage. A model does not learn the thing you care about. It learns the thing you wrote down.

You care about whether a claim is fraudulent. What you have recorded is whether an investigator opened a file. You care about whether a candidate will do the job well. What you have recorded is whether their manager rated them highly in a performance cycle. You care about creditworthiness. What you have recorded is who defaulted among the people who were granted credit, which excludes everyone who was refused.

In each case the recorded label is a human judgement wearing the costume of a fact. Whatever tendencies were present in those judgements are now the target the model is optimising towards, and they arrive with the authority of a number. Article 10(2) specifically directs attention at preparation operations including annotation and labelling, which is the Act's way of saying that the labelling step is part of the system and has to be documented like one.

Door three: the objective

The third door is the one nobody notices, because it is a design decision rather than an accident. Every model is trained to maximise something, and that something is almost always aggregate accuracy across the whole population.

Aggregate accuracy is indifferent to distribution. A model that is right 95 per cent of the time overall can be right 98 per cent of the time for the largest group and 71 per cent of the time for a small one, and the optimiser will prefer that arrangement over an even one if it scores a fraction higher overall. Nobody chose to serve the small group badly. The objective function chose, and it chose because it was never told to care.

The remedy is not a better optimiser. It is to decide, before training, which errors matter and for whom, and to test against that. Article 9(6) and (7) require high-risk systems to be tested against metrics and probabilistic thresholds defined in advance. Written afterwards, a threshold is a description of the result you got. Written beforehand, it is a commitment you can fail.

As of 24 September 2026.