Architecture of processing and analysis system for big astronomical data

This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that performance of both framewor...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2017-03
Main Authors: Kolosov, Ivan, Gerasimov, Sergey, Meshcheryakov, Alexander
Format: Article
Language:English
Subjects:
Data management
Data processing
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astronomical data processing pipelines.
ISSN:2331-8422