KPIs, KRIs and thresholds set before the results

A monitoring programme fails in one of two ways. Either it measures nothing that would change a decision, or it measures a great deal and nobody agreed in advance what any number would trigger. Both produce dashboards. Neither produces evidence.

The distinction that organises the work

A key performance indicator tells you whether the system is doing its job. A key risk indicator tells you whether the conditions for failure are building. They are different measurements with different owners and different response routes, and conflating them is why monitoring packs run to forty pages and change nothing.

For a claims-triage model, the KPIs are things like the proportion of claims routed straight through, the accuracy of the routing measured against final outcome, and the time to settlement. The KRIs are the proportion of inputs falling outside the training distribution, the override rate by handlers, the distribution shift in a key feature, the rate of cases where the model abstains, and the age of the last fairness test. Notice that every KRI is a leading measure of something going wrong, not a report of it having gone wrong.

Thresholds first, always

Article 9(6) and 9(7) require testing against metrics and probabilistic thresholds defined in advance. The same logic governs live monitoring, for the same reason: a threshold chosen once the trend is visible encodes the trend.

Each indicator needs four things agreed and dated before go-live: the value that triggers investigation, the value that triggers action, who is alerted, and what the action is. The fourth is the one omitted. A threshold with no defined action produces an alert, a conversation, and a decision to watch it, repeatedly, until the incident happens and the alert history becomes evidence that you knew.

Be specific about the action. "Investigate" is not an action. "Suspend straight-through processing for the affected segment and route to manual review pending investigation" is. Article 26(5) expects a deployer that identifies a risk to suspend use and inform the provider, so a pre-agreed suspension route is the control that makes that duty executable at three in the afternoon rather than after a committee meeting.

Oversight metrics are risk metrics

Include the human in the measurement set. Override rate, approval rate, seconds per decision and volume per reviewer per shift are the numbers that reveal whether Article 14 oversight is real. An override rate of zero means either a perfect model or a reviewer who has stopped reviewing, and only the second is likely.

These metrics also make automation bias observable, which the Act names expressly in Article 14(4). Measure them from the start, because the baseline is only available early: once throughput expectations have adjusted to the model, you have lost the comparison.

Aggregate model risk across the estate

SR 26-2, issued 17 April 2026 with OCC 2026-13 and FDIC FIL-15-2026, superseded SR 11-7 and added two emphases worth carrying into the monitoring design: proportionate tiering by materiality, and aggregate model risk across the estate.

The second is a genuine change in what the reporting has to show. Per-model monitoring answers whether a model is behaving. It does not answer whether the firm's total exposure to model failure is rising, whether many models share a common dependency, or whether a single data source feeds enough models that its degradation is systemic. A firm running forty models on one feature store has a concentration that no per-model dashboard displays.

Practically, the aggregate view needs three things the per-model view does not carry: shared dependencies across models, including data sources, feature pipelines and upstream model providers; the materiality tier of each model and the total exposure by tier; and the count of models whose validation, fairness test or documentation is overdue. That last number is the one a board can act on.

Worked example: an insurer sets the thresholds before go-live

An insurer is deploying a triage model that routes motor claims to fast-track, standard or investigation. Before launch, the model risk committee approves a monitoring specification with named owners.

Four KPIs: straight-through rate, agreement between model routing and final outcome, average time to settlement by route, and reopened-case rate for fast-tracked claims. The last one is the KPI that matters, because it detects the failure mode that looks like success: routing too much to fast-track improves every other number.

Six KRIs with thresholds set in March, before any production data exists. Input distribution shift on the top ten features, triggering investigation at a stated divergence and action at a higher one. Handler override rate, with both a ceiling and a floor, because a floor detects a handler population that has stopped looking. Proportion of claims scored with missing feature values. Age of the last fairness test, with action at eleven months, because the NYC Local Law 144 structure the insurer has adopted requires an audit no more than one year old and a deadline reached is a deadline missed. Complaint volume relating to routed claims. And model version drift, meaning the count of days the production model version differs from the validated version, which should be zero and occasionally is not.

Each threshold carries an action and an owner. The straight-through rate exceeding its ceiling triggers automatic reversion to a lower threshold for the affected segment, which is a control the operations team can execute without a meeting.

Six weeks after go-live the input distribution KRI fires on a single feature: repair estimate values have shifted upward across the book. The cause is external, a change in parts pricing, and the model is behaving reasonably. Nothing is suspended. But the investigation happened in week six rather than at the quarterly review, and the record shows a threshold set in March firing in May and being handled. That record is worth more to an auditor than a year of green dashboards.

As of 24 September 2026.