A Complete and Fast Scraping Method for Collecting Tweets

• Main Authors: You, Jaebeom, Lee, Jaekyu, Kwon, Hyuk-Yoon
• Format: Conference Proceeding
• Language: English
• Subjects: Blogs; Crawling; Multiprocessing; Parallel processing; Robots; Social networking (online); Spatial databases; Tor Network; Tweets; Web pages
• ISSN: 2375-9356
• DOI: 10.1109/BigComp51126.2021.00014

In this paper, we propose a scraping method for collecting tweets, which we call DeepScrap. DeepScrap provides the complete scraping for the recent tweets that can be viewed on a specific user's page and crawls with a fast speed that overcomes the limited rates in Twitter APIs. Especially, to improve the crawling speed of DeepScrap, we devise a multiprocessing architecture while assigning different IPs to the multiple processes to follow the robots.txt of Twitter. This allows us to maximize the parallelism of crawling in a machine. We show that DeepScrap can crawl the entire tweets that are crawled by Twitter standard APIs by analyzing the tweets on 97 users. Through extensive experiments, we show that DeepScrap can crawl the entire tweets of 97 users, which amounts to 222,194 tweets while Twitter standard API can crawl only 12,586 tweets of them because of the constraints. We also show that multiprocessing of DeepScrap improves single processing of DeepScrap by 2.97 times to crawl 222,194 tweets for 97 users when four processes are running simultaneously.