Packed-Memory Quadtree: A cache-oblivious data structure for visual exploration of streaming spatiotemporal big data

• Published in: Computers & graphics 2018-11, Vol.76, p.117-128
• Main Authors: Toss, Julio, Pahins, Cícero A.L., Raffin, Bruno, Comba, João L.D.
• Format: Article
• Language: English
• Subjects: Computer memory; Computer Science; Data analysis; Data management; Data structures; Data Structures and Algorithms; Data transmission; Datasets; Exploration; Performance indices; Queries; Range queries; Real-time exploration; Spatiotemporal data; Spatiotemporal datasets; Streaming; Visualization
• ISSN: 0097-8493; 1873-7684
• DOI: 10.1016/j.cag.2018.09.005

•A self-organized cache-oblivious data structure for storing and indexing large streaming spatiotemporal datasets.•Algorithms to support real-time visual and range queries over streaming data.•Extensive performance comparison against tried and trusted indexing data structures used by relational and non-relational databases. [Display omitted] The visual analysis of large multidimensional spatiotemporal datasets poses challenging questions regarding storage requirements and query performance. Several data structures have recently been proposed to address these problems that rely on indexes that pre-compute different aggregations from a known-a-priori dataset. Consider now the problem of handling streaming datasets, in which data arrive as one or more continuous data streams. Such datasets introduce challenges to the data structure, which now has to support dynamic updates (insertions/deletions) and rebalancing operations to perform self-reorganizations. In this work, we present the Packed-Memory Quadtree (PMQ), a novel data structure designed to support visual exploration of streaming spatiotemporal datasets. PMQ is cache-oblivious to perform well under different cache configurations. We store streaming data in an internal index that keeps a spatiotemporal ordering over the data following a quadtree representation, with support for real-time insertions and deletions. We validate our data structure under different dynamic scenarios and compare to competing strategies. We demonstrate how PMQ could be used to answer different types of visual spatiotemporal range queries of streaming datasets.