The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities

• Published in: International journal of electronic commerce 2013-04, Vol.17 (3), p.7-36
• Main Authors: Riedl, Christoph, Blohm, Ivo, Leimeister, Jan Marco, Krcmar, Helmut
• Format: Article
• Language: English
• Subjects: Bootstrap; co-creation; Decision analysis; decision quality; Effectiveness studies; Monte Carlo simulation; Multiple criteria decision making; online innovation community; rating scale; Ratings & rankings; User behavior; User generated content; Websites
• ISSN: 1086-4415; 1557-9301
• DOI: 10.2753/JEC1086-4415170301

Given the rise of the Internet, consumers increasingly engage in co-creating products and services. Whereas most co-creation research deals with various aspects of generating user-generated content, this study addresses designing ratings scales for evaluating such content. In detail, we analyze functional and perceptional aspects of two frequently used rating scales in online innovation communities. Using a multimethod approach, our experiments show that a multicriteria scale leads to higher decision quality of users than a single-criterion scale, that idea elaboration (i.e., idea length) negatively moderates this effect such that the single-criterion rating scale outperforms the multicriteria scale for long ideas, and finally that the multicriteria scale leads to more favorable user attitudes toward the Web site. To ensure robustness of our results, we applied a bootstrap-based Monte Carlo simulation based on our experimental data. We found that around 20 user ratings per idea are sufficient for creating stable idea rankings and that a combination of both rating scales leads to a 63 percent performance improvement over the single-criterion rating scale and 16 percent over the multicriteria rating scale. Our work contributes to co-creation research by offering insights as to how the interaction of the technology being used (i.e., rating scale) and the attributes of the rating object affects two central outcome measures: the effectiveness of the rating in terms of decision quality of its users and the perception of the scale by its users as a predictor of future use.