GRACC: GRid ACcounting Collector

The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As t...

Full description

Saved in:
Bibliographic Details
Published in:EPJ Web of conferences 2019, Vol.214, p.3032
Main Authors: Weitzel, Derek, Bockelman, Brian, Zvada, Marian, Retzke, Kevin, Bhat, Shreyas
Format: Article
Language:English
Subjects:
Data storage
Scientific visualization
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As the OSG aged, the size of the database grew very large. It became too large for the database technology to efficiently query to generate detailed reports. The design of a replacement requires data storage that could be queried efficiently to generate multi-year reports. Additionally, it requires flexibilityto add new attributes to the collected data. In this paper we will describe updates to the GRACC architecture in the last 18 months. GRACC uses modern web technologies that were designed for large data storage, query, and visualization. That includes the open source database Elasticsearch, message broker software RabbitMQ, and Grafana and Kibana as data visualization platforms. It uses multiple agents that perform operations on the data to transform it for easier querying and summarization.
ISSN:2100-014X
2101-6275
2100-014X
DOI:10.1051/epjconf/201921403032