Why confident fabrication is not a bug to be patched

People wait for the version that stops making things up. It is worth understanding why that framing misleads, because it shapes how you supervise the tool today.

What the system is doing

A language model produces text one piece at a time, each piece chosen according to what tends to follow in the material it was trained on. It is a machine for continuing text plausibly. That is the whole job description.

Notice what is absent. There is no separate store of facts that the model consults and reports from. There is no step where a claim is checked before it is emitted. The sentence "the court decided in favour of the claimant in 2019" and the sentence "the court decided in favour of the claimant in 2021" are produced by the identical process, and the process has no way to prefer the true one except that true statements were more common in the training material. Where they were not more common, or where the question falls in a gap, the machinery continues doing exactly what it does and produces something that reads correctly.

This is why the word hallucination is a slightly unfortunate one. It suggests a malfunction, an episode, something the system does when it goes wrong. What actually happens is the system working normally on an input where plausibility and truth come apart. The fluent correct answer and the fluent wrong answer are the same event from the inside.

Why confidence never drops

The most dangerous property is not that the output is wrong. Colleagues are wrong. It is that the output carries no signal of being wrong.

A human expert who does not know something usually hedges. The hedge is a social behaviour learned alongside the knowledge, and it tracks the person's actual uncertainty. A language model also produces hedges, and produces them well, but the hedging is itself generated text. It reflects how hedged the training material tended to be on topics that look like this one. It does not reflect whether this particular output is supported.

So a fabricated citation arrives in the same register as a correct one. The formatting is right. The court sounds right. The year is plausible. Everything a reader normally uses as a cue for reliability is present, because those cues are surface features of text and surface features of text are precisely what the system has mastered.

Retrieval helps and does not solve

The common mitigation is to give the model real documents to work from, retrieving relevant passages and asking it to answer using them. This genuinely reduces fabrication, and you should prefer tools that do it. It does not eliminate it, for three reasons worth knowing.

The retrieval can fetch the wrong passages, in which case the model answers confidently from irrelevant material. The retrieval can fetch nothing useful, and the model will usually answer anyway from its general tendencies rather than declining. And even with correct passages in front of it, the model can summarise them in a way that shifts a qualification, drops a condition, or merges two provisions into one that does not exist. The last failure is the hardest to catch, because the answer is anchored in real sources and the error is in the joinery.

What this means for how you use it

The EU AI Act's human oversight requirements in Article 14 read differently once you accept this. They ask that the person overseeing a system understands its capacities and limitations, remains aware of automation bias, can correctly interpret the output, and can decide not to use it or to disregard, override or reverse it. Automation bias is named explicitly: the tendency to over-trust a machine output precisely because it came from a machine.

Fluency is what makes automation bias worse here than in any previous class of system. A spreadsheet that returns a wrong number looks like a number. A model that returns a wrong paragraph looks like expertise.

So the working posture is not suspicion of every output, which is unsustainable, nor trust, which is unsafe. It is knowing which parts of an output carry the risk. That is the next lesson.

As of 24 September 2026.