Where the risk concentrates, and where it does not

Checking everything is not a policy. It is a wish, and it collapses the first week someone is busy. A policy that survives has to tell people which sentences to check, which means being honest that most of an output does not need checking at all.

The five high-risk shapes

Anything with a number in it. Figures, percentages, dates, durations, thresholds, counts. Numbers are exactly the kind of detail that is generated rather than retrieved, and a wrong number looks identical to a right one. This includes numbers the model has copied from a source you supplied, because transcription errors happen in the summarising step.

Anything with a citation in it. Case names, article numbers, standard clauses, document titles, URLs, author names. Fabricated references are the signature failure of this technology because references have a strong surface form the model can imitate perfectly. Article numbers are a particular trap: the instrument is real, the number is plausible, and the content attributed to it belongs somewhere else or nowhere.

Anything about a specific named entity. A person, a company, a product, a case. General claims about a category are often reliable; specific claims about one member of it are where the gaps are.

Anything recent. A model's material has a cut-off, and around the edge of it the behaviour is not a clean refusal but a confident answer built from how things were. Regulatory status is the sharpest example. A rule that was proposed when the material was gathered will be described as if it were in force.

Anything negative. "There is no requirement to do X." "No court has held Y." Absence claims cannot be supported by a process that generates plausible continuations, because the absence of something in the training material is indistinguishable from the model not having seen it. Treat every confident negative as unverified.

Where the risk genuinely is low

It is as important to say this, because a policy that treats all output as suspect gets ignored wholesale.

Transformation of text you supplied is low risk: rewriting your paragraph in plainer language, changing the tone of your email, turning your bullet points into prose. The content came from you, and you can see whether it survived. Structure and scaffolding are low risk: a document outline, a list of questions to consider, a draft agenda. Nothing there is a factual assertion. Brainstorming is low risk, because you are going to select from the output rather than ship it. Code that you run and test is lower risk than it looks, because running it is a check, though this is only true if you actually run it.

The pattern is simple. Risk is high where the output contains a claim about the world that you cannot see the basis for. Risk is low where the output is a rearrangement of something you brought, or where a separate mechanism will catch the error.

Why this has to be written down

Left to individual judgement, the boundary moves with how tired people are. Written into a team norm, it becomes a thing colleagues can remind each other of without it being a criticism.

The EU AI Act's deployer duties point the same way. Article 26 requires a deployer to assign human oversight to people with the necessary competence, training and authority, and to monitor operation and suspend use where a risk is identified. Competence here is not a general disposition to be careful. It is knowing that a case citation and a rewritten paragraph carry entirely different risk, and acting accordingly. Article 4 makes the same point at the level of the organisation: AI literacy proportionate to role and context, an obligation of means rather than result.

As of 24 September 2026.