Bias testing: incompatible metrics and a published calculation

Fairness is not one property, and the request to make the model fair cannot be executed as stated. What a practitioner can do is choose a definition, state it in advance, measure against it, and record what followed.

Why you cannot satisfy every fairness metric at once

The common fairness definitions are mutually incompatible except in degenerate cases. Demographic parity asks that the proportion selected be equal across groups. Equalised odds asks that error rates be equal across groups. Calibration asks that a given score mean the same thing across groups. Where base rates genuinely differ between groups, you cannot have all three, and improving one degrades another. This is a mathematical result, not a resourcing problem, and no vendor tool resolves it.

The governance consequence is specific: the choice of metric is a policy decision with distributional consequences, and it belongs to a named accountable person and a dated record, not to whoever configured the fairness dashboard. A model optimising calibration and a model optimising equal error rates will decline different people. Someone has to own that.

What the EU AI Act requires, and what it does not

Article 10(2)(f) requires examination of datasets in view of possible biases likely to affect health, safety or fundamental rights, or to lead to unlawful discrimination. Article 10(2)(g) requires appropriate measures to detect, prevent and mitigate those biases. Article 10(5) permits processing special category personal data for bias detection and correction only where strictly necessary and subject to safeguards.

Read 10(5) carefully. It is an exception for a defined purpose, not a general licence to collect ethnicity data. The practical trap is collecting protected attributes to run a fairness test and then leaving them in a warehouse where every downstream model can reach them. If you invoke 10(5), the record should say what was collected, for which test, where it is held, who can read it, and when it is deleted.

Note also what the Act does not do. It sets no fairness threshold. It requires examination and mitigation, and under Article 9(6) and 9(7) it requires testing against metrics and probabilistic thresholds defined in advance. The threshold is yours to set and to justify.

NYC Local Law 144: the only regime that prescribes the arithmetic

New York City Local Law 144, with rules at 6 RCNY sections 5-300 to 5-304, is the one regime in force anywhere that specifies the actual calculation. Even where it does not bind you, its structure is the most defensible template available.

It requires an independent bias audit no more than one year old, producing three quantities for sex categories, race and ethnicity categories, and the intersectional combinations of the two:

• Selection rate per category: those selected to advance, divided by the number of applicants in that category.
• Scoring rate per category, where the system scores rather than selects: the proportion of applicants in that category scoring above the sample median.
• Impact ratio per category, computed against the most-selected or highest-scoring category.

Categories representing under 2% of the data may be excluded from the impact ratio calculation, with the justification stated. The published summary must carry the audit date, the data source and an explanation of it, the number of individuals assessed per category, the rates, the impact ratios, the number of individuals in an unknown category, and the distribution date of the system. Candidates get notice at least ten business days before use.

0.80 is not a pass mark, and building a gate on it misreads the law twice

The four-fifths figure comes from the EEOC Uniform Guidelines. It is not a Local Law 144 threshold. Local Law 144 requires you to calculate the impact ratio and publish it. It does not require you to clear any particular value.

Teams build a pass/fail gate at 0.80 anyway, and it fails in both directions. A ratio of 0.83 is published as a pass and the disparity goes uninvestigated. A ratio of 0.78 triggers a scramble to adjust the model until the number clears, which is the worst response available: you have tuned to a statistic rather than to a cause, and documented yourself doing it.

The right structure is to publish what the law requires, and separately to hold your own pre-declared internal threshold with a defined action on breach. Two documents, two purposes. Choosing the internal threshold after seeing the results makes the test prove nothing, which is the same failure Article 9(6) is written against.

Worked example: a bank audits a collections-prioritisation model

A retail bank uses a model to prioritise which delinquent accounts a collections agent calls first. It is not an employment tool, so Local Law 144 does not apply. The bank runs the LL144 calculation anyway, because it is the only prescribed method available.

Selection rate is defined as the proportion of accounts placed in the top priority tier. The bank has no ethnicity data, and under Swiss and EU data protection law cannot simply collect it. It records that limitation explicitly rather than leaving the field blank, and tests against the proxies it does hold lawfully, noting that geographic proxies are themselves a hazard: Illinois HB 3773, effective 1 January 2026, prohibits the use of zip code as a proxy in employment decisions, which is a signal about how regulators view proxy variables generally even though that statute governs employment.

The intersectional analysis is where the finding appears. Neither sex nor age shows a material disparity alone; the combination does. Under the pre-declared internal threshold this triggers investigation rather than retuning. The cause turns out to be a feature encoding part-time employment history. The bank records the disparity, the investigation, the cause, the decision, and the date, and keeps the finding even though it chose to accept part of the residual difference with reasons.

That record is the artefact. A clean dashboard with no history of anything ever being found is not evidence of fairness; it is evidence of not looking.

As of 24 September 2026.