STAIRS: Towards efficient full-text filtering and dissemination in DHT environments

• Published in: The VLDB journal 2011-12, Vol.20 (6), p.793-817
• Main Authors: Rao, Weixiong, Chen, Lei, Fu, Ada Wai-Chee
• Format: Article
• Language: English
• Subjects: Applied sciences; Computer Science; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Data processing. List processing. Character string processing; Database Management; Exact sciences and technology; Information systems. Data bases; Memory organisation. Data processing; Regular Paper; Software
• ISSN: 1066-8888; 0949-877X
• DOI: 10.1007/s00778-011-0224-z

Nowadays “live” content, such as weblog, wikipedia, and news, is ubiquitous in the Internet. Providing users with relevant content in a timely manner becomes a challenging problem. Differing from Web search technologies and RSS feeds/reader applications, this paper envisions a personalized full-text content filtering and dissemination system in a highly distributed environment such as a Distributed Hash Table (DHT) based Peer-to-Peer (P2P) Network. Users subscribe to their interested content by specifying input keywords and thresholds as filters. Then, content is disseminated to those users having interest in it. In the literature, full-text document publishing in DHTs has suffered for a long time from the high cost of forwarding a document to home nodes of all distinct terms. It is aggravated by the fact that a document contains a large number of distinct terms (typically tens or thousands of terms per document). In this paper, we propose a set of novel techniques to overcome such a high forwarding cost by carefully selecting a very small number of meaningful terms (or key features) among candidate terms inside each document. Next, to reduce the average hop count per forwarding, we further prune irrelevant documents during the forwarding path. Experiments based on two real query logs and two real data sets demonstrate the effectiveness of our solution.