A framework for building hypercubes using MapReduce

• Published in: Computer physics communications 2014-05, Vol.185 (5), p.1429-1438
• Main Authors: Tapiador, D., O’Mullane, W., Brown, A.G.A., Luri, X., Huedo, E., Osuna, P.
• Format: Article
• Language: English
• Subjects: Clouds; Column-oriented; Computació distribuïda; Computational grids (Computer systems); Computer simulation; Data mining; Data storage; Diagrames; Diagrams; Estadística; Framework; Gaia mission; Hadoop; Histogram; Hypercube; Hypercubes; Infrastructure; MapReduce; Mathematical models; Milky Way Galaxy; Mineria de dades; Statistics; Via Làctia
• ISSN: 0010-4655; 1879-2944
• DOI: 10.1016/j.cpc.2014.02.010

The European Space Agency’s Gaia mission will create the largest and most precise three dimensional chart of our galaxy (the Milky Way), by providing unprecedented position, parallax, proper motion, and radial velocity measurements for about one billion stars. The resulting catalog will be made available to the scientific community and will be analyzed in many different ways, including the production of a variety of statistics. The latter will often entail the generation of multidimensional histograms and hypercubes as part of the precomputed statistics for each data release, or for scientific analysis involving either the final data products or the raw data coming from the satellite instruments. In this paper we present and analyze a generic framework that allows the hypercube generation to be easily done within a MapReduce infrastructure, providing all the advantages of the new Big Data analysis paradigm but without dealing with any specific interface to the lower level distributed system implementation (Hadoop). Furthermore, we show how executing the framework for different data storage model configurations (i.e. row or column oriented) and compression techniques can considerably improve the response time of this type of workload for the currently available simulated data of the mission. In addition, we put forward the advantages and shortcomings of the deployment of the framework on a public cloud provider, benchmark against other popular solutions available (that are not always the best for such ad-hoc applications), and describe some user experiences with the framework, which was employed for a number of dedicated astronomical data analysis techniques workshops.