Evaluation of text summaries without human references based on the linear optimization of content metrics using a genetic algorithm

•The proposed evaluation provides a better correlation than state-of-the-art methods.•31 state-of-the-art metrics are combined to generate an optimized evaluation metric.•The proposed evaluation method enables a balanced correlation improvement.•The relevance of evaluation metrics presents a direct...

Full description

Saved in:
Bibliographic Details
Published in:Expert systems with applications 2021-04, Vol.167, p.113827, Article 113827
Main Authors: Rojas-Simón, Jonathan, Ledeneva, Yulia, García-Hernández, René Arnulfo
Format: Article
Language:English
Subjects:
Correlation
Evaluation of text summaries
Genetic algorithm
Genetic algorithms
Jensen-Shannon divergence
Latent semantic analysis
Linear optimization of content metrics
Optimization
ROUGE-C
State-of-the-art reviews
Summaries
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:•The proposed evaluation provides a better correlation than state-of-the-art methods.•31 state-of-the-art metrics are combined to generate an optimized evaluation metric.•The proposed evaluation method enables a balanced correlation improvement.•The relevance of evaluation metrics presents a direct relation of individual correlation.•State-of-the-art metrics with higher correlations improve the resultant final rank. The Evaluation of Text Summaries (ETS) has been a task of constant challenges to the development of Automatic Text Summarization (ATS). Within the ATS task, the ETS is crucial to determine the performance of text summaries. Over the last two decades, the scientific community has used the ROUGE system as a standard package to assess the content of automatic summaries. However, if there are not human-made summaries (called human references), then the evaluation cannot be carried out. For this reason, the different state-of-the-art evaluation methods have been proposed that analyze the summary content using the source documents. Nonetheless, these methods do not highly correlate with human assessment. In this paper, a linear optimization of content-based metrics is proposed using a Genetic Algorithm (GA) to improve the correlation between automatic and manual evaluation. The proposed method combines 31 content metrics based on the evaluation without human references. The results of the linear optimization show correlation improvements concerning other evaluation metrics on DUC01 and DUC02 datasets.
ISSN:0957-4174
1873-6793
DOI:10.1016/j.eswa.2020.113827