Explainability and robustness: thresholds before results

Explainability and robustness are assessed together because they fail together. A system nobody can explain is a system whose failure modes nobody predicted, and a system that has only been tested on the happy path has no explanation worth having for what it does elsewhere.

The rule that governs the whole exercise

Article 9(6) and 9(7) require that high-risk systems be tested against metrics and probabilistic thresholds that are defined in advance and appropriate to the intended purpose. That is the single most useful sentence in the Act for a practitioner, because it converts an argument about whether performance is good enough into a documented decision made before anyone knew the answer.

Once results exist, the threshold conversation is contaminated. Everyone in the room now knows the number, and the discussion about what is acceptable will be shaped by what was achieved. Defining thresholds in advance is not bureaucratic ceremony; it is the only way the test carries information.

Article 9 also frames risk management as a continuous, iterative process across the whole lifecycle, covering known and foreseeable risks to health, safety and fundamental rights, risks arising under reasonably foreseeable misuse, and risks identified from post-market data. Article 9(8) requires consideration of the impact on persons under eighteen and other vulnerable groups. In insurance and banking that last point is live wherever products reach young account holders or people under financial distress.

Explanation is not justification

Keep two things apart. An explanation is an account of how the system produced the output. A justification is an account of why the output is right. Feature attributions, counterfactuals and surrogate models give you the first. They do not give you the second, and presenting them as reasons to a customer converts a technical artefact into a representation you may have to defend.

Generative systems make this worse, not better. A model that produces a recommendation and a paragraph of reasoning has produced both by the same process. The paragraph is generated text that reads as a reason. It is not a record of the computation. If your adverse-action notice quotes it, you are asserting something about the system that is not true.

The legal hooks are specific and they are not all in the AI Act. Article 25(2) of the revised Swiss Federal Act on Data Protection gives a data subject the right to information on the logic of an automated decision. Article 21 revFADP lets them state their position and request review by a natural person. Colorado SB 26-189 requires disclosure after an adverse decision within thirty days. Each of these demands something the system must be built to produce, which is a design requirement, not a documentation task.

What Article 13(3) obliges a provider to hand you

The instructions for use must state the provider identity; the characteristics, capabilities and performance limitations including the accuracy metrics the system was validated against; the risks from foreseeable misuse; pre-determined changes; the human oversight measures; the computational and hardware resources, expected lifetime and required maintenance; and how to interpret the logs.

Treat that as a procurement checklist. A vendor who cannot state which accuracy metric the system was validated against has told you something important about their testing. Record the refusal in the third-party assessment rather than leaving the field empty, because an unanswered question is a finding.

Robustness: test the system, not the model

Article 15 requires accuracy, robustness and cybersecurity, including resilience against attempts to alter use, outputs or performance. Most real failures are not model failures. They arise in retrieval, prompt construction, tool calls, upstream data pipelines and integration code, none of which a benchmark exercises.

A robustness record worth keeping states the claimed performance and the metric behind it; the thresholds set in advance; the test set and how it differs from training data; performance for the specific groups the system will be used on; edge cases and out-of-distribution behaviour; adversarial results covering prompt injection including indirect injection through retrieved content, jailbreaks, model extraction and data poisoning; the failure modes found; and whether each is mitigated or accepted as residual risk with reasons. Tester, date and version tested belong on every entry, and the Act expects test logs dated and signed.

Worked example: a bank sets thresholds before building the challenger

A bank plans to replace a rules-based transaction monitoring system with a model. Before any development starts, the model risk function writes the acceptance criteria: minimum detection rate on a held-out set of confirmed cases, a maximum alert volume per analyst per day, a maximum acceptable difference in false positive rate across customer segments, and a stability requirement on the score distribution month to month.

Each threshold carries a rationale. The alert volume ceiling is derived from how long an analyst needs to work an alert properly, which is an oversight constraint rather than a performance one. Article 14(4) requires the oversight person to be able to correctly interpret the output and to remain aware of automation bias; an analyst clearing four hundred alerts a shift is not exercising oversight, whatever the process document says.

The challenger model beats the incumbent on detection rate and fails the segment fairness criterion. Because the criterion was written down in March and the results arrived in September, the conversation is about whether to fix the model or revise the criterion with stated reasons, and both options are visible to the committee. Had the criterion been written in September, there would have been no conversation at all.

The bank also tests the full pipeline, not the model in isolation. The finding that matters is not in the model: a batch job silently drops transactions above a field length, and those transactions were disproportionately from one corporate segment. No model benchmark would ever have surfaced it.

As of 24 September 2026.