Data Flow Lifecycles for Optimizing Workflow Coordination

A critical performance challenge in distributed scientific workflows is coordinating tasks and data flows on distributed resources. To guide these decisions, this paper introduces data flow lifecycle analysis. Workflows are commonly represented using directed acyclic graphs (DAGs). Data flow lifecyc...

Full description

Saved in:
Bibliographic Details
Main Authors: Lee, Hyungro, Guo, Luanzheng, Tang, Meng, Firoz, Jesun, Tallent, Nathan, Kougkas, Anthony, Sun, Xian-He
Format: Conference Proceeding
Language:English
Subjects:
caterpillar tree
data flow lifecycles
Data visualization
Distributed databases
distributed workflows
General and reference > Cross-computing tools and techniques > Measurement
General and reference > Cross-computing tools and techniques > Metrics
General and reference > Cross-computing tools and techniques > Performance
Information systems > Information storage systems > Storage architectures > Distributed storage
Information systems > Information storage systems > Storage management > Hierarchical storage management
Information systems > Information storage systems > Storage management > Information lifecycle management
Instruments
performance analysis
Processor scheduling
Quality of service
storage bottlenecks
Throughput
Volume measurement
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:A critical performance challenge in distributed scientific workflows is coordinating tasks and data flows on distributed resources. To guide these decisions, this paper introduces data flow lifecycle analysis. Workflows are commonly represented using directed acyclic graphs (DAGs). Data flow lifecycles (DFL) enrich task DAGs with data objects and properties that describe data flow and how tasks interact with that flow. Lifecycles enable analysis from several important perspectives: task, data, and data flow. We describe representation, measurement, analysis, visualization, and opportunity identification for DFLs. Our measurement is both distributed and scalable, using space that is constant per data file. We use lifecycles and opportunity analysis to reason about improved task placement and reduced data movement for five scientific workflows with different characteristics. Case studies show improvements of 15×, 1.9×, and 10--30×. Our work is implemented in the DataLife tool.
ISSN:2167-4337
DOI:10.1145/3581784.3607104