Quantitative Analysis of Users' Agreement on Open Educational Resources Quality Inside Repositories

Quality assessment inside learning object repositories is normally performed by the community of users that share interest and rate the same resources. At the same time, this strategy is largely disseminated in the most known repositories. In addition, the final presentation of the overall quality o...

Full description

Saved in:
Bibliographic Details
Published in:IEEE-RITA 2023-02, Vol.18 (1), p.2-9
Main Authors: Stein, Mayara Sousa, Cechinel, Cristian, Ramos, Vinicius Faria Culmant
Format: Article
Language:English
Subjects:
Correlation
Correlation coefficients
educational technology
Evaluation
Frequency measurement
Instruments
multimedia databases
online services
Open Educational Resources
Quality assessment
Quantitative analysis
Ratings
Reliability
Repositories
Statistical analysis
User-generated content
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Quality assessment inside learning object repositories is normally performed by the community of users that share interest and rate the same resources. At the same time, this strategy is largely disseminated in the most known repositories. In addition, the final presentation of the overall quality of the resources is normally restricted to the average rating given by the community, thus, hiding the internal distribution of the ratings and the characteristics of the users involved in the evaluation process. The present paper analyzes to which extent different raters tend to agree about the quality of the resources inside the Merlot repository. For that, data were collected from the repository and calculated the Intra-Class Correlation coefficient for 102 pairs of evaluators, as well as the Spearman correlation among the average ratings of a given resource by evaluators coming from the same categories of disciplines. Results point out a high concentration of poor agreement between raters (75% to 85% of the pairs of raters tended to disagree), and no correlation among the average ratings of the resources from the different disciplines. Based on these findings, the authors suggest improvements to the repository interface better presenting the overall quality of the resources.
ISSN:1932-8540
2374-0132
DOI:10.1109/RITA.2023.3250446