Worked example: the model that was right until the form changed

A health insurer trains a model to predict which reimbursement requests will need manual review. The training data is four years of processed requests. Performance on the held-back test set is strong, the business case is signed off, and the model goes live routing roughly two thirds of requests straight through.

Fourteen months later an internal audit finds that manual review volumes have quietly risen, complaint handling times have doubled for one product line, and nobody can say when it started. Here is the sequence, with the point at which each step could have been caught.

Month 0: the model is honest

At launch the model is doing exactly what was measured. The test set was drawn from the same four years, the population has not moved, and the accuracy figure describes reality. Nothing is wrong yet. What is missing is the paperwork that would let anyone tell later: a record of what the model was trained on, what setting it was validated for, and what threshold would count as unacceptable. Article 9 of the EU AI Act asks for metrics and probabilistic thresholds defined in advance precisely so that this month is the month they are written down.

Month 3: the form is redesigned

A digital team replaces the free-text treatment description with a structured dropdown. The change is an improvement for customers and was never discussed with anyone who owned the model.

The model had learned to lean heavily on the length and wording of the free-text field, because in the training data a long, detailed description correlated with a complicated case. That signal now arrives as one of twelve fixed values. The input has not become invalid; it has become uninformative. The model still produces a confident score for every request, because producing a score is all it does.

This is the cheapest possible catch and the one most often missed, because it is not a model change at all. It is a change to something the model depends on. An inventory entry that lists the fields a system consumes turns this into a routine impact check rather than an archaeology exercise.

Month 7: a new product launches

The insurer starts selling a dental add-on. Dental requests look nothing like anything in four years of training data. The model has never seen this distribution and has no way of signalling that fact.

The outputs it produces for dental requests are not random; they are worse than random in a specific way, because the model maps the unfamiliar input onto the nearest familiar pattern and reports the usual confidence. This is where failures cluster: not in the middle of the training distribution but at its edges and outside it.

Month 11: the aggregate still looks fine

The quarterly dashboard shows overall straight-through accuracy down by about two points. Dental is a small fraction of volume, so its much worse performance is diluted into an aggregate that nobody escalates. The small group is failing and the big number is calm. This is the ordinary arithmetic of averages, and it is why performance has to be monitored per segment rather than only in total.

Month 14: the audit

Three findings, and note that none of them is that the model was bad.

No post-market monitoring plan. Article 72 requires a documented system, proportionate, actively and systematically collecting and analysing performance data across the system's lifetime. Reading a single aggregate on a quarterly slide is not that.

No threshold set in advance. Because no one defined unacceptable before launch, every degraded number could be discussed rather than acted on.

No change control linking the system to its inputs. The form redesign and the product launch were both approved through processes that had no reason to know a model depended on them.

What would have changed the outcome

Not a better model. Three cheap artefacts written at launch: an inventory entry naming the fields and populations the model was validated for; a monitoring plan with per-segment metrics and a threshold that triggers a named person; and a line in the change process that asks whether any AI system consumes a field being altered. Each of them is dull, and each of them would have moved the discovery from month 14 to month 3.

Note also what the insurer owes the people affected. Where a request is refused by a solely automated decision with a significant effect, Article 21 of the revised Swiss Federal Act on Data Protection requires that the person be told the decision was automated, be allowed to state their position, and be allowed to request review by a natural person. Fourteen months of silently degraded performance is fourteen months of those reviews being answered with a number nobody had checked.

As of 24 September 2026.