A Job Sizing Strategy for High-Throughput Scientific Workflows

The user of a computing facility must make a critical decision when submitting jobs for execution: how many resources (such as cores, memory, and disk) should be requested for each job? If the request is too small, the job may fail due to resource exhaustion; if the request is too large, the job may...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems 2018-02, Vol.29 (2), p.240-253
Main Authors: Tovar, Benjamin, da Silva, Rafael Ferreira, Juve, Gideon, Deelman, Ewa, Allcock, William, Thain, Douglas, Livny, Miron
Format: Article
Language:English
Subjects:
automatic job sizing
automatic provision of resources
Bioinformatics
Digital archives
Exhaustion
Feedback loop
High throughput computing (HTC)
Monitoring
Physics
Random access memory
Resource management
resource monitoring and enforcement
Resource scheduling
Sizing
Throughput
throughput and waste optimization
Workflow
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The user of a computing facility must make a critical decision when submitting jobs for execution: how many resources (such as cores, memory, and disk) should be requested for each job? If the request is too small, the job may fail due to resource exhaustion; if the request is too large, the job may succeed, but resources will be wasted. This decision is especially important when running hundreds of thousands of jobs in a high throughput workflow, which may exhibit complex, long tailed distributions of resource consumption. In this paper, we present a strategy for solving the job sizing problem: (1) applications are monitored and measured in user-space as they run; (2) the resource usage is collected into an online archive; and (3) jobs are automatically sized according to historical data in order to maximize throughput or minimize waste. We evaluate the solution analytically, and present case studies of applying the technique to high throughput physics and bioinformatics workflows consisting of hundreds of thousands of jobs, demonstrating an increase in throughput of 10-400 percent compared to naive approaches.
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2017.2762310