A Study on User Preferential Choices about Rating Scales

Websites usually offer the same rating scale for all users and all tasks, but users can have very different preferences. In this paper, we study rating scales from the point of view of preferential choices, investigating i) if user preferences for rating scales depend on the object to evaluate, and ii) if user preferences change after they have rated an object repeatedly, gaining a high level of experience with the evaluated object. We first defined a model of rating scales, identifying generic classes based on features like granularity and visual metaphor. Then, we had users choose between three scales, one for each class, for rating two objects with opposite features, first in a condition where users had a low level of experience, and then in a condition where their level of experience was high. Results showed that user choices depend on the evaluated objects, while their level of experience influences their overall preferences, but not their choices when they have to rate a specific object. We conclude with some insights and guidelines for designers of interactive systems.