C3PO - A Dynamic Data Placement Agent for ATLAS Distributed Data Management

This paper introduces a new dynamic data placement agent for the ATLAS distributed data management system. This agent is designed to pre-place potentially popular data to make it more widely available. It therefore incorporates information from a variety of sources. Those include input datasets and...

Full description

Saved in:
Bibliographic Details
Published in:Journal of physics. Conference series 2017-10, Vol.898 (6), p.62012
Main Authors: Beermann, T, Lassnig, M, Barisits, M, Serfon, C, Garonne, V
Format: Article
Language:English
Subjects:
Algorithms
Data management
Physics
Placement
Workload
Workloads
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper introduces a new dynamic data placement agent for the ATLAS distributed data management system. This agent is designed to pre-place potentially popular data to make it more widely available. It therefore incorporates information from a variety of sources. Those include input datasets and sites workload information from the ATLAS workload management system, network metrics from different sources like FTS and PerfSonar, historical popularity data collected through a tracer mechanism and more. With this data it decides if, when and where to place new replicas that then can be used by the WMS to distribute the workload more evenly over available computing resources and then ultimately reduce job waiting times. This paper gives an overview of the architecture and the final implementation of this new agent. The paper also includes an evaluation of the placement algorithm by comparing the transfer times and the new replica usage.
ISSN:1742-6588
1742-6596
DOI:10.1088/1742-6596/898/6/062012