Characterizing and Predicting the Multifaceted Nature of Quality in Educational Web Resources

• Published in: ACM transactions on interactive intelligent systems 2013-10, Vol.3 (3), p.1-25
• Main Authors: Wetzler, Philipp, Bethard, Steven, Leary, Heather, Butcher, Kirsten, Bahreini, Soheil Danesh, Zhao, Jin, Martin, James H., Sumner, Tamara
• Format: Article
• Language: English
• ISSN: 2160-6455; 2160-6463
• DOI: 10.1145/2533670.2533673

Efficient learning from Web resources can depend on accurately assessing the quality of each resource. We present a methodology for developing computational models of quality that can assist users in assessing Web resources. The methodology consists of four steps: 1) a meta-analysis of previous studies to decompose quality into high-level dimensions and low-level indicators, 2) an expert study to identify the key low-level indicators of quality in the target domain, 3) human annotation to provide a collection of example resources where the presence or absence of quality indicators has been tagged, and 4) training of a machine learning model to predict quality indicators based on content and link features of Web resources. We find that quality is a multifaceted construct, with different aspects that may be important to different users at different times. We show that machine learning models can predict this multifaceted nature of quality, both in the context of aiding curators as they evaluate resources submitted to digital libraries, and in the context of aiding teachers as they develop online educational resources. Finally, we demonstrate how computational models of quality can be provided as a service, and embedded into applications such as Web search.