Serving Hybrid-Cloud SQL Interactive Queries at Twitter

The demand for data analytics has been consistently increasing in the past years at Twitter. In order to fulfill the requirements and provide a highly scalable and available query experience, a large-scale in-house SQL system is heavily relied on. Recently, we evolved the SQL system into a hybrid-cl...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-07
Main Authors: Tang, Chunxu, Wang, Beinan, Wu, Huijun, Wang, Zhenzhao, Yao, Li, Channapattan, Vrushali, Luo, Zhenxiao, Kabra, Ruchin, Ghosh, Mainak, Navadiya, Nikhil Kantibhai, Mishra, Prachi, Mukhedkar, Prateek, Lu, Anneliese
Format: Article
Language:English
Subjects:
Data centers
Federations
Hybrid systems
Query languages
Query processing
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The demand for data analytics has been consistently increasing in the past years at Twitter. In order to fulfill the requirements and provide a highly scalable and available query experience, a large-scale in-house SQL system is heavily relied on. Recently, we evolved the SQL system into a hybrid-cloud SQL federation system, compliant with Twitter's Partly Cloudy strategy. The hybrid-cloud SQL federation system is capable of processing queries across Twitter's data centers and the public cloud, interacting with around 10PB of data per day. In this paper, the design of the hybrid-cloud SQL federation system is presented, which consists of query, cluster, and storage federations. We identify challenges in a modern SQL system and demonstrate how our system addresses them with some important design decisions. We also conduct qualitative examinations and summarize instructive lessons learned from the development and operation of such a SQL system.
ISSN:2331-8422
DOI:10.48550/arxiv.2207.04199