New algorithms assessing short summaries in expository texts using latent semantic analysis

In this study, we compared four expert graders with latent semantic analysis (LSA) to assess short summaries of an expository text. As is well known, there are technical difficulties for LSA to establish a good semantic representation when analyzing short texts. In order to improve the reliability o...

Full description

Saved in:
Bibliographic Details
Published in:Behavior research methods 2009-08, Vol.41 (3), p.944-950
Main Authors: Olmos, Ricardo, León, José A., Jorge-Botana, Guillermo, Escudero, Inmaculada
Format: Article
Language:English
Subjects:
Adolescent
Adult
Algorithms
Behavioral Research - methods
Behavioral Science and Psychology
Cognitive Psychology
Comprehension
Essays
Humans
Linguistics
Models, Statistical
Psychology
Reading
Semantic analysis
Semantics
Statistical analysis
Statistical methods
Students
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this study, we compared four expert graders with latent semantic analysis (LSA) to assess short summaries of an expository text. As is well known, there are technical difficulties for LSA to establish a good semantic representation when analyzing short texts. In order to improve the reliability of LSA relative to human graders, we analyzed three new algorithms by two holistic methods used in previous research (León, Olmos, Escudero, Cañas, & Salmerón, 2006). The three new algorithms were (1) the semantic common network algorithm , an adaptation of an algorithm proposed by W. Kintsch (2001, 2002) with respect to LSA as a dynamic model of semantic representation; (2) a best-dimension reduction measure of the latent semantic space, selecting those dimensions that best contribute to improving the LSA assessment of summaries (Hu, Cai, Wiemer-Hastings, Graesser, & McNamara, 2007); and (3) the Euclidean distance measure , used by Rehder et al. (1998), which incorporates at the same time vector length and the cosine measures. A total of 192 Spanish middle-grade students and 6 experts took part in this study. They read an expository text and produced a short summary. Results showed significantly higher reliability of LSA as a computerized assessment tool for expository text when it used a best-dimension algorithm rather than a standard LSA algorithm. The semantic common network algorithm also showed promising results.
ISSN:1554-351X
1554-3528
DOI:10.3758/BRM.41.3.944