Stochastic clustering for organizing distributed information sources

• Published in: IEEE transactions on cybernetics 2004-10, Vol.34 (5), p.2035-2047
• Main Authors: Mei-Ling Shyu, Shu-Ching Chen, Rubin, S.H.
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Cluster Analysis; Clustering; Clustering methods; Clusters; Computer Communication Networks; Cybernetics; Data analysis; Data mining; Database Management Systems; Databases, Factual; Delay; Engineering management; Information retrieval; Information sources; Information Storage and Retrieval - methods; Markov models; Mathematical models; Models, Statistical; Navigation; Organizing; Pattern Recognition, Automated; Queries; Searching; Stochastic Processes; Studies; Systems Integration; User-Computer Interface
• ISSN: 1083-4419; 2168-2267; 1941-0492; 2168-2275
• DOI: 10.1109/TSMCB.2004.833599

The number of information sources and the volumes of data in these information sources have greatly increased, which may be attributed to the ever-increasing complexity of real-world applications. The enormous amount of information available in the information sources in a distributed information-providing environment has created a need to provide users with tools to effectively and efficiently navigate and retrieve information. Queries in such an environment often access information from multiple information sources. This may be attributed to navigational characteristics. Clusters provide a structure for organizing the large number of information sources for efficient browsing, searching, and retrieval. This paper presents a stochastically-based clustering mechanism, called the Markov model mediator (MMM), to group the information sources into a set of useful clusters. Each information source cluster groups those information sources that show similarities in their data access behavior. Information sources within the same cluster are expected to be able to provide most of the required information among themselves for user queries that are closely related with respect to a particular application. This can significantly improve system response time, query performance, and result in an overall improvement in decision support. Empirical studies on real databases are performed and the results demonstrate that our proposed mechanism leads to a better set of clusters in comparison with other clustering methods. This serves to illustrate the effectiveness of our proposed MMM mechanism.