Real-World Data Difficulty Estimation with the Use of Entropy

In the era of the Internet of Things and big data, we are faced with the management of a flood of information. The complexity and amount of data presented to the decision-maker are enormous, and existing methods often fail to derive nonredundant information quickly. Thus, the selection of the most s...

Full description

Saved in:
Bibliographic Details
Published in:Entropy (Basel, Switzerland) Switzerland), 2021-12, Vol.23 (12), p.1621
Main Authors: Juszczuk, Przemysław, Kozak, Jan, Dziczkowski, Grzegorz, Głowania, Szymon, Jach, Tomasz, Probierz, Barbara
Format: Article
Language:English
Subjects:
Algorithms
Big Data
Classification
Datasets
Decision making
decision table
Entropy
entropy measure
Feature selection
Flood management
Fuzzy sets
Internet of Things
Machine learning
Preprocessing
Real estate
real-world data
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In the era of the Internet of Things and big data, we are faced with the management of a flood of information. The complexity and amount of data presented to the decision-maker are enormous, and existing methods often fail to derive nonredundant information quickly. Thus, the selection of the most satisfactory set of solutions is often a struggle. This article investigates the possibilities of using the entropy measure as an indicator of data difficulty. To do so, we focus on real-world data covering various fields related to markets (the real estate market and financial markets), sports data, fake news data, and more. The problem is twofold: First, since we deal with unprocessed, inconsistent data, it is necessary to perform additional preprocessing. Therefore, the second step of our research is using the entropy-based measure to capture the nonredundant, noncorrelated core information from the data. Research is conducted using well-known algorithms from the classification domain to investigate the quality of solutions derived based on initial preprocessing and the information indicated by the entropy measure. Eventually, the best 25% (in the sense of entropy measure) attributes are selected to perform the whole classification procedure once again, and the results are compared.
ISSN:1099-4300
1099-4300
DOI:10.3390/e23121621