Distributed Fast Self-Organized Maps for Massive Spectrophotometric Data Analysis

Analyzing huge amounts of data becomes essential in the era of Big Data, where databases are populated with hundreds of Gigabytes that must be processed to extract knowledge. Hence, classical algorithms must be adapted towards distributed computing methodologies that leverage the underlying computat...

Full description

Saved in:
Bibliographic Details
Published in:Sensors (Basel, Switzerland) Switzerland), 2018-05, Vol.18 (5), p.1419
Main Authors: Dafonte, Carlos, Garabato, Daniel, Álvarez, Marco A, Manteiga, Minia
Format: Article
Language:English
Subjects:
Algorithms
Apache Hadoop
Apache Spark
Astrophysics
Big Data
computational astrophysics
Consortia
Data analysis
Data mining
Data processing
Design optimization
distributed computing
Distributed processing
fast self-organized maps
Gaia Mission
Quasars
remote sensing
Self organizing maps
Sensors
Software
Spectrophotometry
Star & galaxy formation
Stars & galaxies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Analyzing huge amounts of data becomes essential in the era of Big Data, where databases are populated with hundreds of Gigabytes that must be processed to extract knowledge. Hence, classical algorithms must be adapted towards distributed computing methodologies that leverage the underlying computational power of these platforms. Here, a parallel, scalable, and optimized design for self-organized maps (SOM) is proposed in order to analyze massive data gathered by the spectrophotometric sensor of the European Space Agency (ESA) Gaia spacecraft, although it could be extrapolated to other domains. The performance comparison between the sequential implementation and the distributed ones based on Apache Hadoop and Apache Spark is an important part of the work, as well as the detailed analysis of the proposed optimizations. Finally, a domain-specific visualization tool to explore astronomical SOMs is presented.
ISSN:1424-8220
1424-8220
DOI:10.3390/s18051419