Xuran Li
← All publications

Accurate Fairness: Improving Individual Fairness without Trading Accuracy

Xuran Li, Peng Wu, Jing Su
AAAI 2023 · Proceedings of the 37th AAAI Conference on Artificial Intelligence, 37(12), 14312–14320 · Poster
PaperarXivCode

In one sentence. Accuracy and individual fairness are usually evaluated separately, so a model can be accurate but biased, or fair but wrong, without either metric noticing. Accurate fairness rules out both by asking that an individual and every similar counterpart be treated according to the individual's ground truth.

Motivation

Take one loan application and two similar applicants who differ only in gender, whose application should be approved. Only the top-left case below is what we want; the other three are the ways a model can look good on one metric while failing the other.

Accurate Fair is the goal. Accurate Bias is invisible to an accuracy-only evaluation; Inaccurate Fair is invisible to a fairness-only evaluation.
Accurate Fair is the goal. Accurate Bias is invisible to an accuracy-only evaluation; Inaccurate Fair is invisible to a fairness-only evaluation.

The criterion

For an input (x, a) with non-sensitive attributes x, sensitive attribute a and ground truth y, the similar sub-population I(x, a) contains every version of the same person with a different sensitive value. A classifier is accurately fair if every member's prediction stays within a Lipschitz bound of the ground truth:

Taking a′ = a gives accuracy; taking a′ ≠ a gives individual fairness. One inequality carries both requirements.
Taking a′ = a gives accuracy; taking a′ ≠ a gives individual fairness. One inequality carries both requirements.
Accurate fairness sits between accuracy and individual fairness and implies both (Theorem 1); it also implies statistical parity and confusion-matrix group fairness over the union of similar sub-populations (Theorems 2–3).
Accurate fairness sits between accuracy and individual fairness and implies both (Theorem 1); it also implies statistical parity and confusion-matrix group fairness over the union of similar sub-populations (Theorems 2–3).

Siamese fairness in-processing

To improve the accurate fairness of a model, we train it under the accurate-fairness constraints:

Minimise the training loss over all similar sub-populations subject to the accurate-fairness constraints.

By the Karush–Kuhn–Tucker conditions this is equivalent to a max–min problem with one Lagrange multiplier λij ≥ 0 per similar individual:

Max–min Lagrangian form of the accurate-fairness optimisation.

A Siamese network is the natural optimiser: every member of a similar sub-population goes through a copy of the model with shared parameters θ, so the whole sub-population is treated uniformly in each back-propagation step.

Fair augmentation builds the similar sub-population; m parallel copies with shared parameters θ feed the accurate-fairness loss; gradient descent on θ and ascent on λ.
Fair augmentation builds the similar sub-population; m parallel copies with shared parameters θ feed the accurate-fairness loss; gradient descent on θ and ascent on λ.

Fairness confusion matrix

Each prediction is either correct or wrong, and either consistent across the sub-population or not. The four cells give the true fair, true biased, false fair and false biased rates, and three summary metrics:

The fairness confusion matrix.
The fairness confusion matrix.
Fair-precision, fair-recall and fair-F1.
Fair-precision, fair-recall and fair-F1.

Results

Share of each prediction type for FCNN classifiers, averaged over Adult, German Credit and COMPAS. Siamese fairness (SF, SF_3) has the largest true-fair share and almost no biased predictions of either kind.
Share of each prediction type for FCNN classifiers, averaged over Adult, German Credit and COMPAS. Siamese fairness (SF, SF_3) has the largest true-fair share and almost no biased predictions of either kind.

Compared with state-of-the-art individual bias mitigation, Siamese fairness improves individual fairness by 1.02–8.78 %, accuracy by 8.38–13.69 %, true fair rate by 10.09–20.57 % and fair-F1 by 5.43–10.01 % on average, and is the only method in the Fairea win-win region. On a real Ctrip service-recommendation dataset it raises the share of customers served both accurately and fairly from 34.68 % to 62.85 %.

Citation

@inproceedings{li2023accurate,
  title     = {Accurate Fairness: Improving Individual Fairness without Trading Accuracy},
  author    = {Li, Xuran and Wu, Peng and Su, Jing},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {37}, number = {12}, pages = {14312--14320},
  year      = {2023}
}