Job life cycle management libraries for CMS workflow management projects

Scientific analysis and simulation requires the processing and generation of millions of data samples. These tasks are often comprised of multiple smaller tasks divided over multiple (computing) sites. This paper discusses the Compact Muon Solenoid (CMS) workflow infrastructure, and specifically the...

Full description

Saved in:
Bibliographic Details
Published in:Journal of physics. Conference series 2010-04, Vol.219 (4), p.042024
Main Authors: Lingen, Frank van, Evans, Dave, Metson, Simon, Wakefield, Stuart, Wilkinson, Rick, Jackson, James, Spiga, Daniele, Foulkes, Stephen, Afaq, Anzar, Kuznetsov, Valentin, Vaandering, Eric, Ryu, Seangchan, Farina, Fabio, Codispoti, Giuseppe, Cinquilli, Mattia
Format: Article
Language:English
Subjects:
Data management
Infrastructure
Large Hadron Collider
Libraries
Maintenance costs
Physics
Project management
Solenoids
Specifications
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Scientific analysis and simulation requires the processing and generation of millions of data samples. These tasks are often comprised of multiple smaller tasks divided over multiple (computing) sites. This paper discusses the Compact Muon Solenoid (CMS) workflow infrastructure, and specifically the Python based workflow library which is used for so called task lifecycle management. The CMS workflow infrastructure consists of three layers: high level specification of the various tasks based on input/output data sets, life cycle management of task instances derived from the high level specification and execution management. The workflow library is the result of a convergence of three CMS sub projects that respectively deal with scientific analysis, simulation and real time data aggregation from the experiment. This will reduce duplication and hence development and maintenance costs.
ISSN:1742-6596
1742-6588
1742-6596
DOI:10.1088/1742-6596/219/4/042024