FastBit: interactively searching massive data

As scientific instruments and computer simulations produce more and more data, the task of locating the essential information to gain insight becomes increasingly difficult. FastBit is an efficient software tool to address this challenge. In this article, we present a summary of the key underlying t...

Full description

Saved in:
Bibliographic Details
Published in:Journal of physics. Conference series 2009-07, Vol.180 (1), p.012053
Main Authors: Wu, K, Ahern, S, Bethel, E W, Chen, J, Childs, H, Cormier-Michel, E, Geddes, C, Gu, J, Hagen, H, Hamann, B, Koegler, W, Lauret, J, Meredith, J, Messmer, P, Otoo, E, Perevoztchikov, V, Poskanzer, A, Prabhat, Rübel, O, Shoshani, A, Sim, A, Stockinger, K, Weber, G, Zhang, W-M
Format: Article
Language:English
Subjects:
Data search
Physics
Response time
Simulation
Software
Software development tools
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:As scientific instruments and computer simulations produce more and more data, the task of locating the essential information to gain insight becomes increasingly difficult. FastBit is an efficient software tool to address this challenge. In this article, we present a summary of the key underlying technologies, namely bitmap compression, encoding, and binning. Together these techniques enable FastBit to answer structured (SQL) queries orders of magnitude faster than popular database systems. To illustrate how FastBit is used in applications, we present three examples involving a high-energy physics experiment, a combustion simulation, and an accelerator simulation. In each case, FastBit significantly reduces the response time and enables interactive exploration on terabytes of data.
ISSN:1742-6596
1742-6588
1742-6596
DOI:10.1088/1742-6596/180/1/012053