Worked example: an AI-assisted governance function

A governance team of three supports about ninety AI systems. They are behind. They introduce AI assistance in four places over a quarter. Two of the four are sound, one is defensible with a change, and one quietly destroys the value of the thing it speeds up.

Use one: searching vendor documentation

They load vendor documentation into a retrieval tool and ask it to locate statements about accuracy, limitations and intended purpose, which are among the things Article 13(3) requires instructions for use to contain for a high-risk system.

Assessment: sound. The tool points at passages; a person reads them and decides whether they satisfy the requirement. If the retrieval misses something, the failure mode is a gap someone can find, not a false assurance. Output is verifiable against a source that exists, and checking is cheap.

One caution. A retrieval tool reporting that a vendor document contains no statement of limitations is a finding to confirm by hand before it goes to the vendor, because absence is exactly what retrieval is worst at establishing.

Use two: drafting risk assessments

They give the model a system description and ask for a first-draft risk assessment. It produces three pages: risks, likelihood, impact, mitigations. The analyst edits lightly and files it.

Assessment: this is the one that hollows out. The document is fluent and complete, and indistinguishable from one a careful analyst wrote. But the assessment was supposed to be evidence that someone thought about what could go wrong with this specific system. It is now evidence that someone asked for a risk assessment.

The second-order effect is worse than the first. Three months later the analyst cannot remember which of the risks they identified and which the model supplied, so they cannot say which ones they actually believe. The document has lost its ability to inform its own author.

The change that fixes it. Invert the order. The analyst lists the risks first, in rough notes, from their own knowledge of the system. Then the model is asked what is missing and to format the result. The judgement is produced by a person and the drafting is not, which is the split that preserves what the artefact is for.

Use three: classifying systems against Annex III

They build a classifier: system description in, high-risk or not high-risk out, with a confidence score.

Assessment: defensible only with two changes. This is opacity governing opacity. The classifier infers, nobody can state its rule, and it will be least reliable on unusual systems, which are precisely the ones where classification is difficult and consequential. A steady stream of not-high-risk outputs reads as reassurance and may be evidence that the classifier does not detect the pattern.

What makes it defensible. First, treat it as a triage step that can escalate but never clear: a high-risk output routes to a human, and a not-high-risk output also routes to a human for anything touching employment, credit, insurance pricing, education, essential services or the other Annex III areas. It may only reduce the queue, never close a case. Second, the classifier goes into the register with its own owner, its own oversight and its own statement of what it is bad at. A model that performs a safety function is not outside governance.

Use four: monitoring model outputs for anomalies

Assessment: sound, with the reporting framed correctly. A monitoring model is a detector, and the useful discipline is to report what it found rather than that it found nothing. Green means the monitor did not flag anything, which is a weaker statement than nothing is wrong, and the two must not be allowed to merge in a board pack. Correlated blind spots are the specific risk: if the monitor and the monitored system rest on similar models, the monitor is least likely to catch the error precisely when the system is most likely to make it. That is a reason to complement it with sampling by people, not a reason to abandon it.

What the quarter shows

Three of the four uses survive with adjustment, and the team does get faster. The one that had to change is the one that felt most valuable, which is the usual shape of this problem: the more a governance task feels like paperwork, the more likely it is that the paperwork was the thinking.

As of 24 September 2026.