Hiding the Complexity: Building a Distributed ATLAS Tier-2 with a Single Resource Interface using ARC Middleware

• Published in: Journal of physics. Conference series 2011-12, Vol.331 (7), p.072029-5
• Main Authors: Purdie, S, Stewart, G, Kenyon, M, Skipsey, S, Washbrook, A, Bhimji, W, Filipčič, A
• Format: Article
• Language: English
• Subjects: Clusters; Complex systems; Complexity; Consumers; Data management; Data storage; Management; Mathematical models; Middleware; Physics; Response time (computers); Stacks
• ISSN: 1742-6596; 1742-6588; 1742-6596
• DOI: 10.1088/1742-6596/331/7/072029

Since their inception, Grids for high energy physics have found management of data to be the most challenging aspect of operations. This problem has generally been tackled by the experiment's data management framework controlling in fine detail the distribution of data around the grid and the careful brokering of jobs to sites with co-located data. This approach, however, presents experiments with a difficult and complex system to manage as well as introducing a rigidity into the framework which is very far from the original conception of the grid. In this paper we describe how the ScotGrid distributed Tier-2, which has sites in Glasgow, Edinburgh and Durham, was presented to ATLAS as a single, unified resource using the ARC middleware stack. In this model the ScotGrid 'data store' is hosted at Glasgow and presented as a single ATLAS storage resource. As jobs are taken from the ATLAS PanDA framework, they are dispatched to the computing cluster with the fastest response time. An ARC compute element at each site then asynchronously stages the data from the data store into a local cache hosted at each site. The job is then launched in the batch system and accesses data locally. We discuss the merits of this system compared to other operational models and consider, from the point of view of the resource providers (sites), and from the resource consumers (experiments); and consider issues involved in transitions to this model.