A framework for dependency estimation in heterogeneous data streams

Estimating dependencies from data is a fundamental task of Knowledge Discovery. Identifying the relevant variables leads to a better understanding of data and improves both the runtime and the outcomes of downstream Data Mining tasks. Dependency estimation from static numerical data has received muc...

Full description

Saved in:
Bibliographic Details
Published in:Distributed and parallel databases : an international journal 2021-06, Vol.39 (2), p.415-444
Main Authors: Fouché, Edouard, Mazankiewicz, Alan, Kalinke, Florian, Böhm, Klemens
Format: Article
Language:English
Subjects:
Chi-square test
Computer Science
Data mining
Data Structures
Data transmission
Database Management
Heterogeneity
Information Systems Applications (incl.Internet)
Memory Structures
Operating Systems
Selected Papers from SSDBM 2019
Statistical tests
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Estimating dependencies from data is a fundamental task of Knowledge Discovery. Identifying the relevant variables leads to a better understanding of data and improves both the runtime and the outcomes of downstream Data Mining tasks. Dependency estimation from static numerical data has received much attention. However, real-world data often occurs as heterogeneous data streams: On the one hand, data is collected online and is virtually infinite. On the other hand, the various components of a stream may be of different types, e.g., numerical, ordinal or categorical. For this setting, we propose Monte Carlo Dependency Estimation (MCDE), a framework that quantifies multivariate dependency as the average statistical discrepancy between marginal and conditional distributions, via Monte Carlo simulations. MCDE handles heterogeneity by leveraging three statistical tests: the Mann–Whitney U, the Kolmogorov–Smirnov and the Chi-Squared test. We demonstrate that MCDE goes beyond the state of the art regarding dependency estimation by meeting a broad set of requirements. Finally, we show with a real-world use case that MCDE can discover useful patterns in heterogeneous data streams.
ISSN:0926-8782
1573-7578
DOI:10.1007/s10619-020-07295-x