Clustering semantically heterogeneous distributed aggregate databases

• Published in: Knowledge and information systems 2014-02, Vol.38 (2), p.331-364
• Main Authors: Zhang, Shuai, McClean, Sally I., Scotney, Bryan W.
• Format: Article
• Language: English
• Subjects: Aggregates; Algorithmics. Computability. Computer arithmetics; Algorithms; Analysis; Applied sciences; Clustering; Computer Science; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Cooperation; Data mining; Data Mining and Knowledge Discovery; Data processing. List processing. Character string processing; Database Management; Exact sciences and technology; Information Storage and Retrieval; Information Systems and Communication Service; Information Systems Applications (incl.Internet); Information systems. Data bases; IT in Business; Knowledge discovery; Memory organisation. Data processing; Regular Paper; Semantic web; Semantics; Software; Studies; Theoretical computing
• ISSN: 0219-1377; 0219-3116
• DOI: 10.1007/s10115-012-0588-4

Databases developed independently in a common open distributed environment may be heterogeneous with respect to both data schema and the embedded semantics. Managing schema and semantic heterogeneities brings considerable challenges to learning from distributed data and to support applications involving cooperation between different organisations. In this paper, we are concerned mainly with heterogeneous databases that hold aggregates on a set of attributes, which are often the result of materialised views of native large-scale distributed databases. A model-based clustering algorithm is proposed to construct a mixture model where each component corresponds to a cluster which is used to capture the contextual heterogeneity among databases from different populations. Schema heterogeneity, which can be recast as incomplete information, is handled within the clustering process using Expectation-Maximisation estimation and integration is carried out within a clustering iteration. Our proposed algorithm resolves the schema heterogeneity as part of the clustering process, thus avoiding transformation of the data into a unified schema. Results of algorithm evaluation on classification, scalability and reliability, using both real and synthetic data, demonstrate that our algorithm can achieve good performance by incorporating all of the information from available heterogeneous data. Our clustering approach has great potential for scalable knowledge discovery from semantically heterogeneous databases and for applications in an open distributed environment, such as the Semantic Web.