HDM-MC in-Action: A Framework for Big Data Analytics across Multiple Clusters

Big data are increasingly collected and stored in a highly distributed infrastructures due to the development of several emerging technologies including sensor network, cloud computing, IoT and mobile computing among many other emerging technologies. In practice, the majority of existing big data pr...

Full description

Saved in:
Bibliographic Details
Main Authors: Wu, Dongyao, Sakr, Sherif, Zhu, Liming, Lee, Sung, Wu, Huijun
Format: Conference Proceeding
Language:English
Subjects:
Big Data
Data centers
Distributed databases
Distributed Systems
Organizations
Planning
Scheduling
Workflows
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Big data are increasingly collected and stored in a highly distributed infrastructures due to the development of several emerging technologies including sensor network, cloud computing, IoT and mobile computing among many other emerging technologies. In practice, the majority of existing big data processing frameworks (e.g., Hadoop, Spark, Flink) are designed based on the single-cluster setup with the assumptions of centralized management and homogeneous connectivity which makes them sub-optimal and sometimes infeasible to be applied for scenarios that require implementing data analytics jobs on highly distributed data sets (across racks, data centers or multi organizations). We demonstrate HDM-MC, a big data processing framework that is designed to enable the capability of performing large scale data analytics across multi-clusters with minimum extra overhead due to additional scheduling requirements. We describe the architecture and realization of the system using a step-by-step example scenario.
ISSN:2575-8411
DOI:10.1109/ICDCS.2018.00165