A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems

With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propo...

Full description

Saved in:
Bibliographic Details
Main Authors: Harshvardhan, West, Brandon, Fidel, Adam, Amato, Nancy M., Rauchwerger, Lawrence
Format: Conference Proceeding
Language:English
Subjects:
Big data
Load modeling
Loading
Optimization
out-of-core graph algorithms
parallel graph processing
Parallel processing
Partitioning algorithms
Random access memory
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.
ISSN:1530-2075
DOI:10.1109/IPDPS.2015.28