Efficiently Estimating Mutual Information Between Attributes Across Tables

• Main Authors: Santos, Aecio, Korn, Flip, Freire, Juliana
• Format: Conference Proceeding
• Language: English
• Subjects: Accuracy; Analytical models; Data augmentation; data discovery; Data engineering; Machine learning; Maximum likelihood estimation; mutual information estimation; Smoothing methods
• ISSN: 2375-026X
• DOI: 10.1109/ICDE60146.2024.00022

Relational data augmentation is a powerful technique for enhancing data analytics and improving machine learning models by incorporating columns from external datasets. However, it is challenging to efficiently discover relevant external tables to join with a given input table. Existing approaches rely on data discovery systems to identify "joinable" tables from external sources, typically based on overlap or containment. However, the sheer number of tables obtained from these systems results in irrelevant joins that need to be performed; this can be computationally expensive or even infeasible in practice. We address this limitation by proposing the use of efficient mutual information (MI) estimation for finding relevant joinable tables. We introduce a new sketching method that enables efficient evaluation of relationship discovery queries by estimating MI without materializing the joins and returning a smaller set of tables that are more likely to be relevant. We also demonstrate the effectiveness of our approach at approximating MI in extensive experiments using synthetic and real-world datasets.