Worked example: the prompt that became a policy

A complaints team at a bank has a backlog. An analyst writes a prompt that reads a complaint and returns a category, an urgency rating and a suggested first response. It works well. She shares it. Within two months it is pasted into every complaint handler's workflow and roughly nine hundred complaints a month pass through it.

Nobody approved this. Nobody would say they had implemented an AI system. They would say they had shared a useful prompt.

Reading the prompt as a specification

The original text, lightly abbreviated, was: "You are an expert complaints handler with twenty years of experience. Read the complaint below and tell me how serious it is and what we should say. Be thorough but concise. Take a deep breath and think step by step."

Against the four elements, this is thin. The task is two operations joined by "and", one of which ("how serious") is undefined. There is no context beyond the complaint itself: no definition of the bank's urgency bands, no statement of which regulatory categories matter. There are two constraints, "thorough" and "concise", which contradict. There is no output format, so handlers get prose and re-key it. And there is nothing at all about what to do when the complaint is ambiguous, which is the case that actually needs handling.

Rewritten as a specification, it names the operation, supplies the bank's four urgency definitions verbatim, requires the output as three labelled fields, and adds one line that changes more than the rest combined: "If the complaint does not contain enough information to assign an urgency, return NEEDS REVIEW and state what is missing. Do not infer."

What went wrong before the rewrite

Three things, all traceable.

Urgency drifted. Without the bank's definitions, the system applied its own, which were plausible and not the bank's. Complaints mentioning financial hardship were rated lower than the bank's own policy would rate them.

The suggested response was sometimes sent. Under pressure, handlers pasted it. It is generated text, and its confident tone concealed the cases where it had filled a gap in the complaint with a reasonable-sounding assumption.

A complaint contained instructions. One complainant, quoting an earlier email from the bank, included text that the system read as instruction rather than content. This is prompt injection, and it did not require an attacker; it required a quoted email. The risk is catalogued in the OWASP Top 10 for Large Language Model Applications 2026, and it arises structurally whenever untrusted text becomes part of the prompt.

The governance status nobody had noticed

By the time nine hundred complaints a month run through one text, that text is a rule the bank applies to its customers. It should have had a version identifier recorded against every complaint, a named owner, a change record and retention of superseded versions. It had none of those, so when the urgency drift was found, the bank could not say when it started or which complaints were affected.

Note what was not the problem. The model was not wrong in any interesting sense. It answered the question it was asked. The question was underspecified, and the underspecification was invisible because the answers were fluent.

Fixing it, in the order it was fixed

• The prompt was rewritten as a specification with the bank's own definitions supplied as context.
• The NEEDS REVIEW instruction was added, and roughly one complaint in nine started coming back unclassified, which was the correct number and had previously been guessed at.
• The suggested response was cut. The system now classifies; humans write. The team decided that generated text addressed to a complainant was not something it wanted to defend.
• A version identifier was recorded against each complaint, and the old prompt was archived rather than deleted.
• Before and after behaviour was compared on eighty closed complaints, against criteria written down before the comparison was run, in the spirit of the requirement in Article 9(7) to test against thresholds defined in advance.

The lesson

A prompt becomes infrastructure at the moment it is shared, not at the moment someone writes a policy about it. Everything the team did after the fact would have taken an afternoon before.

As of 24 September 2026.