Automated summary evaluation with inbuilt rubric method: An alternative to constructed responses and multiple-choice tests assessments

Automated summary evaluation is proposed as an alternative to rubrics and multiple-choice tests in knowledge assessment. Inbuilt rubric is a recent Latent Semantic Analysis (LSA) method that implements rubrics in an artificially-generated semantic space. It was compared with classical LSA's cos...

Full description

Saved in:
Bibliographic Details
Published in:Assessment and evaluation in higher education 2019-10, Vol.44 (7), p.1029-1041
Main Authors: Martínez-Huertas, José Á., Jastrzebska, Olga, Olmos, Ricardo, León, José A.
Format: Article
Language:English
Subjects:
Alternative Assessment
automated summary evalution
Automation
Correlation
Educational attainment
Educational tests & measurements
Effect Size
Foreign Countries
High School Students
inbuilt rubric
Measurement Techniques
Multiple Choice Tests
Reading Comprehension
Reading Tests
rubrics
Scoring Rubrics
Semantic analysis
Semantics
Semiotics
Student Evaluation
summaries
Test Reliability
Test Validity
Undergraduate Students
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Automated summary evaluation is proposed as an alternative to rubrics and multiple-choice tests in knowledge assessment. Inbuilt rubric is a recent Latent Semantic Analysis (LSA) method that implements rubrics in an artificially-generated semantic space. It was compared with classical LSA's cosine-based methods assessing knowledge in a within-subjects design regarding two validation sources: a comparison with the results of rubric scores and multiple-choice tests, and the sensitivity of predicting the academic level of the test-taker. Results showed a higher reliability for inbuilt rubric (from Pearson correlation coefficient .81 to .49) over the classical LSA method (from .61 to .34), and a higher sensitivity using binary logistic regressions and effect sizes to predict academic level. It is concluded that inbuilt rubric has a qualitatively higher reliability and validity than classical LSA methods in a way that is complementary to models based on semantic networks. Thus, it is concluded that new automated summary evaluation approaches such as the inbuilt rubric method can be practical in terms of reliability and efficiency, and, thus, they can offer an affordable and valuable form of knowledge assessment in different educational levels.
ISSN:0260-2938
1469-297X
DOI:10.1080/02602938.2019.1570079