An effective approach to enhancing a focused crawler using Google

In this paper, we share our experience in augmenting a focused crawler of our vertical search engine designed to work with academic slides. The goal of the focused crawler was to collect Microsoft PowerPoint files from academic institutions. A previous approach based on a general web crawler can fai...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of supercomputing 2020-10, Vol.76 (10), p.8175-8192
Main Authors: Lee, Jae-Gil, Bae, Donghwan, Kim, Sansung, Kim, Jungeun, Yi, Mun Yong
Format: Article
Language:English
Subjects:
Compilers
Computer Science
Information retrieval
Interpreters
Processor Architectures
Programming Languages
Search engines
Websites
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we share our experience in augmenting a focused crawler of our vertical search engine designed to work with academic slides. The goal of the focused crawler was to collect Microsoft PowerPoint files from academic institutions. A previous approach based on a general web crawler can fail to collect a sufficient number of files mainly because of the robots exclusion protocol and missing hyperlinks. As a remedy to these problems, we propose a combinatory approach in which the indexing information maintained by a general web search engine such as Google is utilized for target URL list generation through our query generator, further then complemented by our URL extractor and file downloader. Because Google has already crawled billions of web pages, it will be more cost-efficient and potentially effective to systematically retrieve the desired information from Google than to redo crawling from scratch by ourselves. Our focused crawler, which we call SlideCrawler , has been used for our vertical search engine CourseShare since the fall of 2011. The capability of SlideCrawler was verified for the top-500 world wide universities. SlideCrawler collected about one million files from the top-500 universities. Further, the study results show that SlideCrawler outperforms Nutch, collecting 3.7 times more slide files.
ISSN:0920-8542
1573-0484
DOI:10.1007/s11227-019-02787-9