A hybrid approach for extracting informative content from web pages

• Published in: Information processing & management 2013-07, Vol.49 (4), p.928-944
• Main Authors: Uzun, Erdinç, Agun, Hayri Volkan, Yerlikaya, Tarık
• Format: Article
• Language: English
• Subjects: Accessibility; Artificial intelligence; Automatic text analysis; Data mining; Data processing; Decision trees; Exact sciences and technology; Extraction; Feature extraction; Information and communication sciences; Information processing; Information processing and retrieval; Information retrieval; Information retrieval systems. Information and document management system; Information retrieval. Man machine relationship; Information science. Documentation; Learning; Machine learning; Management; Research process. Evaluation; Sciences and techniques of general use; Strings; Studies; Template Detection; Web Cleaning; Web Content Extraction; Web Learning Modeling; Web sites
• ISSN: 0306-4573; 1873-5371
• DOI: 10.1016/j.ipm.2013.02.005

► A hybrid approach for informative web content extraction is presented. ► This approach has two steps that invoke each other. ► In the first step, the learning algorithm infers rules from features based on DOM structure. ► In the second step, the rules are applied to the HTML text to efficiently extract the content. Eliminating noisy information and extracting informative content have become important issues for web mining, search and accessibility. This extraction process can employ automatic techniques and hand-crafted rules. Automatic extraction techniques focus on various machine learning methods, but implementing these techniques increases time complexity of the extraction process. Conversely, extraction through hand-crafted rules is an efficient technique that uses string manipulation functions, but preparing these rules is difficult and cumbersome for users. In this paper, we present a hybrid approach that contains two steps that can invoke each other. The first step discovers informative content using Decision Tree Learning as an appropriate machine learning method and creates rules from the results of this learning method. The second step extracts informative content using rules obtained from the first step. However, if the second step does not return an extraction result, the first step gets invoked. In our experiments, the first step achieves high accuracy with 95.76% in extraction of the informative content. Moreover, 71.92% of the rules can be used in the extraction process, and it is approximately 240 times faster than the first step.