Training, inference, and the gap between them

A machine learning model is built once and used many times. Almost every unpleasant surprise in production lives in the gap between those two moments, so it is worth being precise about what happens in each.

Training

Training is the process that produces the model. You supply a body of examples, each one an input paired with the outcome you want predicted: a claim and whether it settled without dispute, a photograph and what is in it, a sentence and its translation. An optimisation procedure then adjusts a large collection of numbers, the model's parameters, until the predictions it makes on those examples are as close as the procedure can get them to the recorded outcomes.

Two things follow immediately. The first is that nobody writes the rule. The rule is discovered, and it exists only as those numbers. The second is that whatever pattern is present in the examples is the pattern the model will learn, including patterns nobody intended and nobody noticed. If past claims handlers were slower with claims written in imperfect German, the model learns that hesitancy as a signal, because from the inside there is no difference between a pattern that reflects the world and a pattern that reflects a habit.

This is why the EU AI Act puts data governance duties where it does. Article 10 requires, for high-risk systems, that training, validation and testing data be relevant, sufficiently representative and to the best extent possible free of errors and complete, and that the data be examined for possible biases with measures in place to detect, prevent and mitigate them. It also requires you to record the design choices, where the data came from and what it was originally collected for, and what preparation was done to it: annotation, labelling, cleaning, enrichment, aggregation. Those requirements exist because the training set is the specification. There is no other place to look.

Inference

Inference is every use of the model afterwards. An input arrives, the fixed parameters turn it into an output, and nothing about the model changes. That is the ordinary case. Some systems are retrained on new data periodically, which is a fresh training run producing a new model, not the same model improving.

The important property of inference is that the model has no access to its own training. It cannot tell you which examples shaped a particular answer. It cannot show its working, because there is no working of the kind a person would recognise, only an arithmetic path through parameters. When someone asks why the model declined this application, the honest first answer is that the model does not contain a reason; it contains a number, and any reason offered afterwards was reconstructed by a separate method.

Why a model cannot be asked to forget

This is the consequence people find hardest, and it matters because it collides with rights that do exist.

Training does not store examples. It distributes their influence across the parameters, mixed with the influence of every other example. Deleting a record from the training database therefore does nothing to the model that was already trained on it. The record is gone; the influence is not, because the influence is no longer in a form that can be located and removed. The only reliable way to produce a model that was not shaped by a given example is to train again without it, which is expensive and produces a different model with different behaviour throughout.

Deletion rights under data protection law are not suspended by this fact. What changes is when you have to think about them. A deletion request is cheap to honour against a database and very hard to honour against a trained model, so the decision that matters is the one made before training: what goes in, on what basis, and what you have committed to be able to undo. The European Data Protection Board addressed personal data in AI models in Opinion 28/2024, which is the place to start when the question is live.

What to take away

Training is where the behaviour is fixed and where the evidence has to be created, because it cannot be recovered later. Inference is where the behaviour is used and where it can only be observed, not explained from the inside. Governance that arrives after training is governance of a thing whose most important decisions have already been made.

As of 24 September 2026.