Rucio, the next-generation Data Management system in ATLAS

Rucio is the next-generation of Distributed Data Management (DDM) system benefiting from recent advances in cloud and ”Big Data” computing to address HEP experiments scaling requirements. Rucio is an evolution of the ATLAS DDM system Don Quixote 2 (DQ2), which has demonstrated very large scale data...

Full description

Saved in:
Bibliographic Details
Published in:Nuclear and particle physics proceedings 2016-04, Vol.273-275, p.969-975
Main Authors: Serfon, C., Barisits, M., Beermann, T., Garonne, V., Goossens, L., Lassnig, M., Nairz, A., Vigne, R.
Format: Article
Language:English
Subjects:
Distributed Data Management
DQ2
Rucio
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Rucio is the next-generation of Distributed Data Management (DDM) system benefiting from recent advances in cloud and ”Big Data” computing to address HEP experiments scaling requirements. Rucio is an evolution of the ATLAS DDM system Don Quixote 2 (DQ2), which has demonstrated very large scale data management capabilities with more than 160 petabytes spread worldwide across 130 sites, and accesses from 1,000 active users. However, DQ2 is reaching its limits in terms of scalability, requiring a large number of support staff to operate and being hard to extend with new technologies. Rucio addresses these issues by relying on new technologies to ensure system scalability, cover new user requirements and employ new automation framework to reduce operational overheads. This paper shows the key concepts of Rucio, details the Rucio design, and the technology it employs, the tests that were conducted to validate it and finally describes the migration steps that were conducted to move from DQ2 to Rucio.
ISSN:2405-6014
2405-6022
DOI:10.1016/j.nuclphysbps.2015.09.151