Reducing the Loss of Information through Annealing Text Distortion

Compression distances have been widely used in knowledge discovery and data mining. They are parameter-free, widely applicable, and very effective in several domains. However, little has been done to interpret their results or to explain their behavior. In this paper, we take a step toward understan...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering 2011-07, Vol.23 (7), p.1090-1102
Main Authors: Granados, Ana, Cebrian, Manuel, Camacho, David, de Borja Rodriguez, Francisco
Format: Article
Language:English
Subjects:
Algorithms
Annealing
Blocking
Clustering
Clustering algorithms
clustering by compression
Complexity theory
Compressing
Compression algorithms
Data compression
Data mining
Distortion
Distortion measurement
Information analysis
Information distortion
Kolmogorov complexity
normalized compression distance
Studies
Texts
Upper bound
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Compression distances have been widely used in knowledge discovery and data mining. They are parameter-free, widely applicable, and very effective in several domains. However, little has been done to interpret their results or to explain their behavior. In this paper, we take a step toward understanding compression distances by performing an experimental evaluation of the impact of several kinds of information distortion on compression-based text clustering. We show how progressively removing words in such a way that the complexity of a document is slowly reduced helps the compression-based text clustering and improves its accuracy. In fact, we show how the nondistorted text clustering can be improved by means of annealing text distortion. The experimental results shown in this paper are consistent using different data sets, and different compression algorithms belonging to the most important compression families: Lempel-Ziv, Statistical and Block-Sorting.
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2010.173