Predicting the next piece of text, and everything that follows from it

A large language model does one thing. Given a stretch of text, it produces a probability distribution over what comes next, picks from it, appends the result, and repeats. The unit it works in is a token, roughly a word or a fragment of one. Everything a chat assistant appears to do is that loop, run until it decides to stop.

This is not a simplification offered to save you from the mathematics. It is the mechanism, and nearly every governance question about these systems becomes easy once you hold it steady.

What the loop explains

Why the output is plausible rather than true. The model is selecting text that fits the pattern of text it learned from. Fitting the pattern and being correct coincide often, because most text in the world is more or less accurate about the things it describes. They are nonetheless separate properties, and nothing in the loop checks the second one. There is no step at which a claim is looked up, compared against a record, or marked unverified.

Why fluency tells you nothing. Fluency is what the model optimises. A confident, well-structured, grammatically flawless paragraph is the expected output whether or not its content is right, so the signals a reader normally uses to gauge reliability have been detached from reliability. A human expert who writes smoothly about a subject usually knows it. That inference is safe with people and unsafe here, and the habit is hard to switch off because it has served you well your whole life.

Why the same question gives different answers. Selection from a distribution involves an element of chance. Ask twice, get two answers, both fluent, possibly contradicting. This is normal operation rather than a fault, and it is why "I checked it and it was right" does not establish that it will be right next time.

Why a justification is not a reason. Ask a model why it recommended something and it generates text that fits the pattern of an explanation. That text is produced by the same loop that produced the recommendation, not by inspecting it. The result reads like reasoning and is evidence of nothing. If a generated justification is shown to a customer as the reason for a decision, the organisation is presenting a plausible narrative as a record of how the decision was made.

Where the text came from

The distribution the model draws from is shaped by an enormous body of text collected before training. You generally cannot inspect it. This is why the transparency obligations that exist are worth knowing about: California AB 2013, operative 1 January 2026 and reaching back to systems released on or after 1 January 2022, requires developers to publish twelve enumerated disclosures about training data on their website, including the sources or owners, the number and types of data points, whether material is protected by copyright, trademark or patent or is in the public domain, whether it was purchased or licensed, whether personal information is included, what cleaning was applied, the collection period, and whether synthetic data was used.

Those disclosures are a floor and they are useful. What they are not is a guarantee about any particular answer.

What the Act calls these things

The EU AI Act separates the underlying general-purpose AI model from the AI system built on it. GPAI obligations sit with whoever placed the model on the market and have applied since 2 August 2025; the final text of the GPAI Code of Practice arrived on 10 July 2025 and became enforceable with fines from 2 August 2026. Your duties concern the system you built or use.

Two transparency duties under Article 50 apply from 2 August 2026 and were not deferred by the Digital Omnibus. People must be informed they are interacting with an AI system unless that is obvious. And providers of systems generating synthetic audio, image, video or text must mark the output in a machine-readable format, detectable as artificially generated, with a grace period for systems already on the market before 2 August 2026 that ends 2 December 2026.

Note the shape of that second duty carefully, because it is widely misread. Machine-readable marking is a provider duty. It does not discharge a deployer's separate disclosure obligation: the Commission's guidance on the deepfake disclosure in Article 50(4) requires a label a person can see or hear and understand without a detection tool. Two duties, two audiences, and satisfying one leaves the other exactly where it was.

As of 24 September 2026.