Worked example: auditing an oversight arrangement

A health insurer uses a model to triage pre-authorisation requests. The model recommends approve, decline or refer. A clinical reviewer confirms every decline before it is sent. The governance pack describes this as human oversight by qualified clinicians. You have been asked to check whether that description holds.

What the paperwork says

The procedure names the role, requires a registered clinician, and logs a reviewer identity against every declined case. Sampling the log shows no missing signatures. On paper this is a clean arrangement, and the first instinct is to report that it is fine.

What five questions find

Throughput. One reviewer covers the decline queue for a region. The queue runs at roughly two hundred and forty cases a shift; the shift is seven and a half hours. That is under two minutes per case, including the reviewer's other duties. Reading a pre-authorisation file and forming an independent clinical view does not fit in two minutes, and the reviewer says so when asked directly.

Override rate. Four per cent in the first month after deployment, under one per cent now. The trend is presented internally as evidence the model improved. Nothing was retrained in that period, so the model did not change. The reviewer did.

What happens on override. The reviewer selects a reason from a list and writes a free-text note. The note goes into a field nobody has a process for reading. No override has ever been discussed with the modelling team.

Outcome feedback. Reviewers are told when a decline is appealed and reversed, but the notification goes to a shared inbox, not to the individual, and arrives weeks later. No reviewer could name a case of theirs that turned out wrong.

Incentives. The team has a service-level target on time-to-decision. There is no measure of review quality. Speed is measured; care is not.

Mapping the findings to Article 14(4)

Understand capacities and limitations: partially met. Reviewers had a demonstration at launch. They have never been shown where the model is weakest, and the vendor documentation gives no accuracy figures by case type.

Awareness of automation bias: not met, and the override curve is the evidence. A rate falling from four per cent to one per cent against an unchanged model is automation bias with a timestamp.

Correctly interpret output: not met. The interface shows a recommendation and a confidence percentage. Nobody has told reviewers what the confidence number is calibrated against, so it functions as decoration.

Decide to disregard or override: formally present, practically constrained. The right exists; the time, the feedback and the incentive to use it do not.

Intervene or interrupt: not met at system level. There is no mechanism for a reviewer to halt the triage model for a class of cases, only to disagree case by case. When a reviewer noticed a pattern of wrong declines for one procedure code, their only route was email.

Testing Article 26(2)

Competence: met. These are registered clinicians reviewing clinical decisions.

Training: not met. Training covered the interface, not the system's failure modes.

Authority: not met in substance. Individual overrides are permitted; nothing lets a reviewer stop or restrict the system, and no route exists for a reviewer's pattern observation to reach anyone with that power.

The finding and the fix

The finding is not that anyone did anything wrong. It is that the arrangement is described as oversight and cannot function as oversight at the volume it runs at, and that the organisation would have discovered this at any point by dividing cases by hours.

Four changes make it defensible: bring the per-case time above the time the task takes by reducing the scope of what needs confirming; route override notes to the modelling team with a response duty; give each reviewer their own overturned-appeal outcomes; and create a documented path by which a reviewer can suspend the model for a case class, which is what Article 14(4)(e) is asking for. Note that none of these is a training course.

As of 24 September 2026.