SIDR: structure-aware intelligent data routing in Hadoop

The MapReduce framework is being extended for domains quite different from the web applications for which it was designed, including the processing of big structured data, e.g., scientific and financial data. Previous work using MapReduce to process scientific data ignores existing structure when as...

Full description

Saved in:
Bibliographic Details
Main Authors: Buck, Joe, Watkins, Noah, Levin, Greg, Crume, Adam, Ioannidou, Kleoni, Brandt, Scott, Maltzahn, Carlos, Polyzotis, Neoklis, Torres, Aaron
Format: Conference Proceeding
Language:English
Subjects:
Abstracts
Hadoop
Information systems > Data management systems > Database management system engines > Database query processing
Information systems > Information retrieval > Search engine architectures and scalability > Distributed retrieval
Information systems > Information retrieval > Search engine architectures and scalability > Peer-to-peer retrieval
Information systems > Information storage systems > Storage architectures > Distributed storage
MapReduce
Scientific Data
Shape
Software and its engineering > Software organization and properties > Software system structures > Distributed systems organizing principles
Theory of computation > Theory and algorithms for application domains > Database theory > Database query processing and optimization (theory)
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The MapReduce framework is being extended for domains quite different from the web applications for which it was designed, including the processing of big structured data, e.g., scientific and financial data. Previous work using MapReduce to process scientific data ignores existing structure when assigning intermediate data and scheduling tasks. In this paper, we present a method for incorporating knowledge of the structure of scientific data and executing query into the MapReduce communication model. Built in SciHadoop, a version of the Hadoop MapReduce framework for scientific data, SIDR intelligently partitions and routes intermediate data, allowing it to: remove Hadoop's global barrier and execute Reduce tasks prior to all Map tasks completing; minimize intermediate key skew; and produce early, correct results. SIDR executes queries up to 2.5 times faster than Hadoop and 37% faster than SciHadoop; produces initial results with only 6% of the query completed; and produces dense, contiguous output.
ISSN:2167-4329
2167-4337
DOI:10.1145/2503210.2503241