STAVIES: a system for information extraction from unknown Web data sources through automatic Web wrapper generation using clustering techniques

• Published in: IEEE transactions on knowledge and data engineering 2005-12, Vol.17 (12), p.1638-1652
• Main Authors: Papadakis, N.K., Skoutas, D., Raftopoulos, K., Varvarigou, T.A.
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Automation; Clustering; Clustering algorithms; Computer science; control theory; systems; Computer Society; Data mining; data source wrappers; Exact sciences and technology; Extraction; generic wrappers; Human; Humans; Index Terms- Automatic wrappers; Information retrieval; Information systems. Data bases; Intelligent agent; intelligent agents on the Web; Memory organisation. Data processing; Repositories; resource discovery; Software; Studies; Technological innovation; Web data extraction; Web mining; Web pages; Web sites; Web structure mining; Websites; World Wide Web
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2005.203

A fully automated wrapper for information extraction from Web pages is presented. The motivation behind such systems lies in the emerging need for going beyond the concept of "human browsing". The World Wide Web is today the main "all kind of information" repository and has been so far very successful in disseminating information to humans. By automating the process of information retrieval, further utilization by targeted applications is enabled. The key idea in our novel system is to exploit the format of the Web pages to discover the underlying structure in order to finally infer and extract pieces of information from the Web page. Our system first identifies the section of the Web page that contains the information to be extracted and then extracts it by using clustering techniques and other tools of statistical origin. STAVIES can operate without human intervention and does not require any training. The main innovation and contribution of the proposed system consists of introducing a signal-wise treatment of the tag structural hierarchy and using hierarchical clustering techniques to segment the Web pages. The importance of such a treatment is significant since it permits abstracting away from the raw tag-manipulating approach. Experimental results and comparisons with other state of the art systems are presented and discussed in the paper, indicating the high performance of the proposed algorithm.