Investigating the Gap between Offline Metrics and Perceived Fairness

Fairness represents a crucial aspect of recommender systems. While past research has focused on improving both item and group fairness, these efforts have relied primarily on offline evaluation methods. However, fairness-aware recommender systems face a critical gap between offline algorithmic evaluation and users' online perceptions. In this paper, we investigate relationships between offline and perceived fairness measurements, and how group item fairness post-hoc interventions impact group user fairness across demographic segments. Our findings reveal misalignments between offline fairness evaluation and users' perceptions of fairness. Moreover, post-hoc interventions to enhance offline group item fairness produced unexpected effects across different user groups, particularly regarding age and education level. This calls for the need to incorporate user studies into the evaluation of algorithms for fairness-aware recommender systems to ensure equitable outcomes. Our source code is publicly available at https://bit.ly/483TssZ.