Assessing the Impact of Debiasing in Classification

Fairness and Explainable Artificial Intelligence (XAI) have become more and more intertwined over the years. Their interplay gave birth to two open research areas: Explainable Fairness and Explanation Fairness. It is standard practice to audit machine learning models for bias and check if they behave correctly for every demographic group. However, a poorly investigated aspect concerns the transparency of the debiasing and the impact it has on the model’s behaviour and output. How did the debiasing process affect decisions based on different demographics? Why has an instance belonging to a protected group received a negative outcome? Is it because the debiasing procedure has not been effective? In this paper, we address these research questions by defining a framework for analysing the behaviour of a mitigated model by explaining a classifier trained to discriminate between debiased instances and non-debiased ones. We show experimentally that, thanks to our framework, it is possible to draw useful insights on the debiasing process, and to inspect instances for which the debiased classifier behaves strangely or unexpectedly by looking at the feature importance of the discriminator.