What an explanation is worth

Explainability is usually sold as a property of a model. It is better understood as a property of a conversation: an explanation is worth exactly what it lets its recipient do next. Test every explanation against that.

Three people ask for an explanation of the same declined application and want three different things. The customer wants to know what to change. The case handler wants to know whether to trust the score. The regulator wants to know whether the decision was lawfully made. An output that satisfies one of them may be useless to the other two, and organisations regularly congratulate themselves for producing the easiest one.

Local and global

A global explanation describes how the system behaves in general: which inputs carry weight across the population, how performance varies between groups, where the model is known to be weak. A local explanation describes one output: why this application, this applicant, this day.

The two do not substitute for each other, and conflating them is the most common error in practice. Knowing that income is globally the strongest predictor tells the declined applicant nothing, because their income may have had no bearing on their own outcome. Conversely, a local explanation says nothing about whether the system is systematically worse for one group; that question is only answerable in aggregate.

The law asks for both, in different places. Article 13(3) requires instructions for use that state characteristics, capabilities and performance limitations, including the accuracy metrics the system was validated against, and how to interpret the logs. That is global. Article 25(2) of the revised Swiss Federal Act on Data Protection gives the data subject a right of access that includes information on the logic of an automated individual decision. That is asked about a particular decision, so it is local, and it is a right an individual can exercise directly.

Explanations that are technically true and practically worthless

Three patterns recur.

The feature list. "The most influential inputs were tenure, claim count and postcode." This names inputs without stating direction or magnitude. Every one of those could have pushed either way. A recipient cannot act on it.

The confidence number. "The model was 87% confident." A calibration figure is not a reason. It describes the model's own estimate of being right, which is itself a model output and can be wrong in the same way the prediction can.

The process description. "The application was scored by our automated assessment system and reviewed under our standard procedure." This explains the workflow, not the decision. It is what organisations produce when they have nothing else, and it is usually a sign that nobody can reconstruct the reason.

A usable test

An explanation is adequate for a given recipient if that recipient can act on it without being told anything further. For a customer, that means naming something specific and, where the model actually behaves that way, something they could change. For a reviewer, it means enough to disagree: the reviewer must be able to identify a case where the system is likely wrong, which requires knowing where the system is weak. For an auditor, it means enough to reconstruct the decision from records rather than from the current state of a live system.

That last point is underrated. An explanation that can only be produced by re-running the system today is not an audit trail, because the system today is not necessarily the system that made the decision. This is why Article 12's automatic event logging and the six-month retention minimum in Articles 19 and 26(6) are explainability controls as much as security ones.

What the supervisors ask for

FINMA's Guidance 08/2024 of 18 December 2024 on governance and risk management when using artificial intelligence names explainability and transparency as one of its four themes, alongside governance and accountability, robustness and reliability, and non-discrimination. It is guidance rather than a circular and creates no new rules, but it states clearly that responsibility remains with the institution even where the AI is bought in or outsourced. You cannot borrow a vendor's explanation and call it yours if you cannot stand behind it.

The NIST AI Risk Management Framework takes a similar line from the other direction. Its Playbook publishes, for each subcategory, a set of transparency and documentation questions an organisation should be able to answer. The framework is voluntary and has no enforcing body, so its value is the question set rather than any compliance claim. If you cannot answer the questions, saying you follow the framework means nothing.

As of 24 September 2026.