Xuran Li
← All publications

Perturbation Effects on Robustness and Individual Fairness

Xuran Li, Hao Xue, Peng Wu, Xingjun Ma, Zhen Zhang, Huaming Chen, Flora D. Salim
KDD 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Jeju Island, August 2026 · Oral
DOIarXivCode

In one sentence. A small, meaning-preserving edit can either break a prediction (robustness) or make two similar people get different answers (individual fairness); existing evaluations check the two separately and miss the cases where one breaks while the other holds. Robust Individual Fairness (RIF) is a unified notion that jointly characterizes robustness and individual fairness under semantic-preserving perturbations, and RIFair is a black-box framework whose decoupled substitution strategy exposes the robustness and fairness violations induced by superficial lexical variation.

The robustness–fairness relationship

Robust Fair satisfies RIF. Robust Biased is accepted by robustness-only evaluations, Unrobust Fair by fairness-only evaluations; Unrobust Biased is the worst case.
Robust Fair satisfies RIF. Robust Biased is accepted by robustness-only evaluations, Unrobust Fair by fairness-only evaluations; Unrobust Biased is the worst case.
Real pairs found by RIFair on Bias in Bios and SST-2. Different but equally admissible substitutions on the similar counterparts (talk → discuss / chat) trigger different failure modes.
Real pairs found by RIFair on Bias in Bios and SST-2. Different but equally admissible substitutions on the similar counterparts (talk → discuss / chat) trigger different failure modes.

Semantic equivalence, not geometric proximity

Perturbations and "similar individuals" are usually defined by geometric neighbourhoods (an Lp ball, cosine similarity). In NLP that proxy is unreliable: China → Guangzhou is close in embedding space yet changes the truth conditions of a sentence. We instead ground equivalence in truth-conditional semantics and operationalise it with bidirectional entailment from an NLI model:

S_eq(v, v′) = P(v ⊨ v′) · P(v′ ⊨ v); an edit is admissible iff S_eq(v, v′) ≥ τ_sem.

With this filter we define the two sets RIF quantifies over — the similar instances I(v) (equivalent once the sensitive tokens 𝒜 are masked by M) and the admissible perturbations Δadm (meaning-preserving token substitutions):

Definitions of the similar-instance set I(v) and the admissible perturbation set Δ_adm.

Two assumptions make the ground truth usable as the target: A1 (label invariance) — admissible edits do not change y; A2 (semantic filter) — an edit is admissible only if it passes the bidirectional entailment check.

Robust Individual Fairness (RIF)

With the two sets in place, RIF is one inequality over both of them. A classifier satisfies ε-RIF on an instance v if, for every similar counterpart v′ ∈ I(v) and every admissible edit δ ∈ Δadm, the prediction error on both sides stays within ε of the ground truth:

The RIF definition.
RIF implies robustness and 2ε individual fairness; robustness plus Lipschitz fairness implies only a relaxed RIF.
RIF implies robustness and 2ε individual fairness; robustness plus Lipschitz fairness implies only a relaxed RIF.

RIFair

RIFair builds similar pairs by masking sensitive tokens, constructs a meaning-preserving perturbation space with a Generate–Filter pipeline, ranks tokens by occlusion saliency, and searches decoupled substitutions on the two sides of a pair to minimise a signed objective whose sign pattern selects the target quadrant.

Generate–Filter: Step 1, an LLM proposes raw candidate paraphrases and synonyms; Step 2, an NLI discriminator keeps the semantically imperceptible ones.
Generate–Filter: an LLM proposes raw candidates, an NLI discriminator keeps the semantically imperceptible ones.
The unified objective. +1 minimises a term, −1 maximises it; (α_v, α_v′, β) = (+1, −1, −1) targets Robust Biased, (−1, −1, +1) Unrobust Fair, (−1, +1, −1) Unrobust Biased.
Hyperparameters for the three RIF violations: +1 minimises a term, −1 maximises it.
Hyperparameters for the three RIF violations: +1 minimises a term, −1 maximises it.
Prediction trajectories of a similar pair under the three objectives.
Prediction trajectories of a similar pair under the three objectives.

Results

Resistance of four backbones under fairness, robustness and RIF evaluation. The order of the models changes with the axis.
Resistance of four backbones under fairness, robustness and RIF evaluation. The order of the models changes with the axis.
When RIFair targets one quadrant, robustness stays high under Robust Biased and fairness stays high under Unrobust Fair; RIF is lowest in every scenario.
When RIFair targets one quadrant, robustness stays high under Robust Biased and fairness stays high under Unrobust Fair; RIF is lowest in every scenario.
Retraining with RIFair-generated pairs reduces every failure rate on Bias in Bios.
Retraining with RIFair-generated pairs reduces every failure rate on Bias in Bios.

Decoupled search exposes 2.3× more Robust Biased and 3.0× more Unrobust Biased violations than a coupled attack, and entailment-constrained pairs keep their meaning (Seq = 0.978, LLM-judge 4.60/5) where cosine-constrained ones drift.

Citation

@inproceedings{li2026perturbation,
  title     = {Perturbation Effects on Robustness and Individual Fairness},
  author    = {Li, Xuran and Xue, Hao and Wu, Peng and Ma, Xingjun and Zhang, Zhen and Chen, Huaming and Salim, Flora D.},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26)},
  year      = {2026}, publisher = {ACM}, doi = {10.1145/3770855.3817848}
}