Managing very large distributed data sets on a data grid

In this work we address the management of very large data sets, which need to be stored and processed across many computing sites. The motivation for our work is the ATLAS experiment for the Large Hadron Collider (LHC), where the authors have been involved in the development of the data management m...

Full description

Saved in:
Bibliographic Details
Published in:Concurrency and computation 2010-08, Vol.22 (11), p.1338-1364
Main Authors: Branco, Miguel, Zaluska, Ed, de Roure, David, Lassnig, Mario, Garonne, Vincent
Format: Article
Language:English
Subjects:
Computation
Concurrency
data management
distributed systems
grid computing
Hadrons
Infrastructure
Middleware
modelling
simulation
Uncertainty
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this work we address the management of very large data sets, which need to be stored and processed across many computing sites. The motivation for our work is the ATLAS experiment for the Large Hadron Collider (LHC), where the authors have been involved in the development of the data management middleware. This middleware, called DQ2, has been used for the last several years by the ATLAS experiment for shipping petabytes of data to research centres and universities worldwide. We describe our experience in developing and deploying DQ2 on the Worldwide LHC computing Grid, a production Grid infrastructure formed of hundreds of computing sites. From this operational experience, we have identified an important degree of uncertainty that underlies the behaviour of large Grid infrastructures. This uncertainty is subjected to a detailed analysis, leading us to present novel modelling and simulation techniques for Data Grids. In addition, we discuss what we perceive as practical limits to the development of data distribution algorithms for Data Grids given the underlying infrastructure uncertainty, and propose future research directions. Copyright © 2009 John Wiley & Sons, Ltd.
ISSN:1532-0626
1532-0634
1532-0634
DOI:10.1002/cpe.1489