GRACC: GRid ACcounting Collector

• Published in: EPJ Web of conferences 2019, Vol.214, p.3032
• Main Authors: Weitzel, Derek, Bockelman, Brian, Zvada, Marian, Retzke, Kevin, Bhat, Shreyas
• Format: Article
• Language: English
• Subjects: Data storage; Scientific visualization; Visualization
• ISSN: 2100-014X; 2101-6275; 2100-014X
• DOI: 10.1051/epjconf/201921403032

The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As the OSG aged, the size of the database grew very large. It became too large for the database technology to efficiently query to generate detailed reports. The design of a replacement requires data storage that could be queried efficiently to generate multi-year reports. Additionally, it requires flexibilityto add new attributes to the collected data. In this paper we will describe updates to the GRACC architecture in the last 18 months. GRACC uses modern web technologies that were designed for large data storage, query, and visualization. That includes the open source database Elasticsearch, message broker software RabbitMQ, and Grafana and Kibana as data visualization platforms. It uses multiple agents that perform operations on the data to transform it for easier querying and summarization.