A Complete and Fast Scraping Method for Collecting Tweets

In this paper, we propose a scraping method for collecting tweets, which we call DeepScrap. DeepScrap provides the complete scraping for the recent tweets that can be viewed on a specific user's page and crawls with a fast speed that overcomes the limited rates in Twitter APIs. Especially, to i...

Full description

Saved in:
Bibliographic Details
Main Authors: You, Jaebeom, Lee, Jaekyu, Kwon, Hyuk-Yoon
Format: Conference Proceeding
Language:English
Subjects:
Blogs
Crawling
Multiprocessing
Parallel processing
Robots
Social networking (online)
Spatial databases
Tor Network
Tweets
Web pages
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose a scraping method for collecting tweets, which we call DeepScrap. DeepScrap provides the complete scraping for the recent tweets that can be viewed on a specific user's page and crawls with a fast speed that overcomes the limited rates in Twitter APIs. Especially, to improve the crawling speed of DeepScrap, we devise a multiprocessing architecture while assigning different IPs to the multiple processes to follow the robots.txt of Twitter. This allows us to maximize the parallelism of crawling in a machine. We show that DeepScrap can crawl the entire tweets that are crawled by Twitter standard APIs by analyzing the tweets on 97 users. Through extensive experiments, we show that DeepScrap can crawl the entire tweets of 97 users, which amounts to 222,194 tweets while Twitter standard API can crawl only 12,586 tweets of them because of the constraints. We also show that multiprocessing of DeepScrap improves single processing of DeepScrap by 2.97 times to crawl 222,194 tweets for 97 users when four processes are running simultaneously.
ISSN:2375-9356
DOI:10.1109/BigComp51126.2021.00014