Design and Implementation of GXP Make -- A Workflow System Based on Make

This paper describes the rational behind designing workflow systems based on the Unix make by showing a number of idioms useful for workflows comprising many tasks. It also demonstrates a specific design and implementation of such a workflow system called GXP make. GXP make supports all the features...

Full description

Saved in:
Bibliographic Details
Main Authors: Taura, K, Matsuzaki, T, Miwa, M, Kamoshida, Y, Yokoyama, D, Nan Dun, Shibata, T, Choi Sung Jun, Tsujii, J
Format: Conference Proceeding
Language:English
Subjects:
Dispatching
Graphical user interfaces
Java
Libraries
Supercomputers
Syntactics
Web services
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper describes the rational behind designing workflow systems based on the Unix make by showing a number of idioms useful for workflows comprising many tasks. It also demonstrates a specific design and implementation of such a workflow system called GXP make. GXP make supports all the features of GNU make and extends its platforms from single node systems to clusters, clouds, supercomputers, and distributed systems. Interestingly, it is achieved by a very small code base that does not modify GNU make implementation at all. While being not ideal for performance, it achieved a useful performance and scalability of dispatching one million tasks in approximately 16,000 seconds (60 tasks per second, including dependence analysis) on an 8 core Intel Nehalem node. For real applications, recognition and classification of protein-protein interactions from biomedical texts on a supercomputer with more than 8,000 cores are described.
DOI:10.1109/eScience.2010.43