Hybrid analysis pipelines in the REANA reproducible analysis platform

We introduce the feasibility of running hybrid analysis pipelines in the REANA reproducible analysis platform. The REANA platform allows researchers to specify declarative computational workflow steps describing the analysis process and to execute analysis workload on remote containerised compute cl...

Full description

Saved in:
Bibliographic Details
Main Authors: Rodríguez, Diego, Mačiulaitis, Rokas, Okraska, Jan, Šimko, Tibor
Format: Conference Proceeding
Language:English
Subjects:
Cloud computing
Control systems design
Particle physics
Pipelines
Workflow
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73
cites cdi_FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73
container_end_page
container_issue
container_start_page 6041
container_title
container_volume 245
creator Rodríguez, Diego
Mačiulaitis, Rokas
Okraska, Jan
Šimko, Tibor
description We introduce the feasibility of running hybrid analysis pipelines in the REANA reproducible analysis platform. The REANA platform allows researchers to specify declarative computational workflow steps describing the analysis process and to execute analysis workload on remote containerised compute clouds. We have designed an abstract job controller component permitting to execute different parts of the analysis workflow on different compute backends, such as HTCondor, Kubernetes and SLURM. We have prototyped the designed solution including the job execution, job monitoring, and input/output file staging mechanism between the various compute backends. We have tested the prototype using several particle physics model analyses. The present work introduces support for hybrid analysis workflows in the REANA reproducible analysis platform and paves the way towards studying underlying performance advantages and challenges associated with hybrid analysis patterns in complex particle physics data analyses.
doi_str_mv 10.1051/epjconf/202024506041
format conference_proceeding
fullrecord <record><control><sourceid>proquest_doaj_</sourceid><recordid>TN_cdi_doaj_primary_oai_doaj_org_article_e7ade15ec8d54813b37ccb99ffe2f34f</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><doaj_id>oai_doaj_org_article_e7ade15ec8d54813b37ccb99ffe2f34f</doaj_id><sourcerecordid>2465749298</sourcerecordid><originalsourceid>FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73</originalsourceid><addsrcrecordid>eNpNUE1LAzEQDaJgqf0HHhY8r83n7uZYSrWFoiAK3kI2mWiW7WZNtof-e7e2SGcOMwyP9-Y9hO4JfiRYkDn0jQmdm1M8Nhe4wJxcoQklGOeY8M_ri_0WzVJq8FhMSiaKCVqtD3X0NtOdbg_Jp6z3PbS-g5T5Lhu-IXtbLV4WWYQ-Brs3vm7hAtzqwYW4u0M3TrcJZuc5RR9Pq_flOt--Pm-Wi21uWIGHnGlLGUhnMaEEeMkkq8e3BFhtRcUqTsFR6ngFgMvaMEuNMFYaEETW1JZsijYnXht0o_rodzoeVNBe_R1C_FI6Dt60oKDUFogAU1nBK8JqVhpTS-kcUMe4G7keTlyjsZ89pEE1YR9HZ0lRXoiSSyqrEcVPKBNDShHcvyrB6pi_OuevLvNnv2Ileng</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>conference_proceeding</recordtype><pqid>2465749298</pqid></control><display><type>conference_proceeding</type><title>Hybrid analysis pipelines in the REANA reproducible analysis platform</title><source>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</source><source>Free Full-Text Journals in Chemistry</source><creator>Rodríguez, Diego ; Mačiulaitis, Rokas ; Okraska, Jan ; Šimko, Tibor</creator><contributor>Kamleh, W. ; Stewart, G.A. ; Kim, D. ; Doglioni, C. ; Silvestris, L. ; Jackson, P.</contributor><creatorcontrib>Rodríguez, Diego ; Mačiulaitis, Rokas ; Okraska, Jan ; Šimko, Tibor ; Kamleh, W. ; Stewart, G.A. ; Kim, D. ; Doglioni, C. ; Silvestris, L. ; Jackson, P.</creatorcontrib><description>We introduce the feasibility of running hybrid analysis pipelines in the REANA reproducible analysis platform. The REANA platform allows researchers to specify declarative computational workflow steps describing the analysis process and to execute analysis workload on remote containerised compute clouds. We have designed an abstract job controller component permitting to execute different parts of the analysis workflow on different compute backends, such as HTCondor, Kubernetes and SLURM. We have prototyped the designed solution including the job execution, job monitoring, and input/output file staging mechanism between the various compute backends. We have tested the prototype using several particle physics model analyses. The present work introduces support for hybrid analysis workflows in the REANA reproducible analysis platform and paves the way towards studying underlying performance advantages and challenges associated with hybrid analysis patterns in complex particle physics data analyses.</description><identifier>ISSN: 2100-014X</identifier><identifier>ISSN: 2101-6275</identifier><identifier>EISSN: 2100-014X</identifier><identifier>DOI: 10.1051/epjconf/202024506041</identifier><language>eng</language><publisher>Les Ulis: EDP Sciences</publisher><subject>Cloud computing ; Control systems design ; Particle physics ; Pipelines ; Workflow</subject><ispartof>EPJ Web of Conferences, 2020, Vol.245, p.6041</ispartof><rights>2020. This work is licensed under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and conditions, you may use this content in accordance with the terms of the License.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73</citedby><cites>FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2465749298?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>309,310,314,780,784,789,790,4024,23930,23931,25140,25753,27923,27924,27925,37012,44590</link.rule.ids></links><search><contributor>Kamleh, W.</contributor><contributor>Stewart, G.A.</contributor><contributor>Kim, D.</contributor><contributor>Doglioni, C.</contributor><contributor>Silvestris, L.</contributor><contributor>Jackson, P.</contributor><creatorcontrib>Rodríguez, Diego</creatorcontrib><creatorcontrib>Mačiulaitis, Rokas</creatorcontrib><creatorcontrib>Okraska, Jan</creatorcontrib><creatorcontrib>Šimko, Tibor</creatorcontrib><title>Hybrid analysis pipelines in the REANA reproducible analysis platform</title><title>EPJ Web of Conferences</title><description>We introduce the feasibility of running hybrid analysis pipelines in the REANA reproducible analysis platform. The REANA platform allows researchers to specify declarative computational workflow steps describing the analysis process and to execute analysis workload on remote containerised compute clouds. We have designed an abstract job controller component permitting to execute different parts of the analysis workflow on different compute backends, such as HTCondor, Kubernetes and SLURM. We have prototyped the designed solution including the job execution, job monitoring, and input/output file staging mechanism between the various compute backends. We have tested the prototype using several particle physics model analyses. The present work introduces support for hybrid analysis workflows in the REANA reproducible analysis platform and paves the way towards studying underlying performance advantages and challenges associated with hybrid analysis patterns in complex particle physics data analyses.</description><subject>Cloud computing</subject><subject>Control systems design</subject><subject>Particle physics</subject><subject>Pipelines</subject><subject>Workflow</subject><issn>2100-014X</issn><issn>2101-6275</issn><issn>2100-014X</issn><fulltext>true</fulltext><rsrctype>conference_proceeding</rsrctype><creationdate>2020</creationdate><recordtype>conference_proceeding</recordtype><sourceid>PIMPY</sourceid><sourceid>DOA</sourceid><recordid>eNpNUE1LAzEQDaJgqf0HHhY8r83n7uZYSrWFoiAK3kI2mWiW7WZNtof-e7e2SGcOMwyP9-Y9hO4JfiRYkDn0jQmdm1M8Nhe4wJxcoQklGOeY8M_ri_0WzVJq8FhMSiaKCVqtD3X0NtOdbg_Jp6z3PbS-g5T5Lhu-IXtbLV4WWYQ-Brs3vm7hAtzqwYW4u0M3TrcJZuc5RR9Pq_flOt--Pm-Wi21uWIGHnGlLGUhnMaEEeMkkq8e3BFhtRcUqTsFR6ngFgMvaMEuNMFYaEETW1JZsijYnXht0o_rodzoeVNBe_R1C_FI6Dt60oKDUFogAU1nBK8JqVhpTS-kcUMe4G7keTlyjsZ89pEE1YR9HZ0lRXoiSSyqrEcVPKBNDShHcvyrB6pi_OuevLvNnv2Ileng</recordid><startdate>2020</startdate><enddate>2020</enddate><creator>Rodríguez, Diego</creator><creator>Mačiulaitis, Rokas</creator><creator>Okraska, Jan</creator><creator>Šimko, Tibor</creator><general>EDP Sciences</general><scope>AAYXX</scope><scope>CITATION</scope><scope>8FE</scope><scope>8FG</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>P5Z</scope><scope>P62</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>DOA</scope></search><sort><creationdate>2020</creationdate><title>Hybrid analysis pipelines in the REANA reproducible analysis platform</title><author>Rodríguez, Diego ; Mačiulaitis, Rokas ; Okraska, Jan ; Šimko, Tibor</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73</frbrgroupid><rsrctype>conference_proceedings</rsrctype><prefilter>conference_proceedings</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Cloud computing</topic><topic>Control systems design</topic><topic>Particle physics</topic><topic>Pipelines</topic><topic>Workflow</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Rodríguez, Diego</creatorcontrib><creatorcontrib>Mačiulaitis, Rokas</creatorcontrib><creatorcontrib>Okraska, Jan</creatorcontrib><creatorcontrib>Šimko, Tibor</creatorcontrib><collection>CrossRef</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>Advanced Technologies &amp; Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest advanced technologies &amp; aerospace journals</collection><collection>ProQuest Advanced Technologies &amp; Aerospace Collection</collection><collection>Publicly Available Content Database (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>DOAJ Directory of Open Access Journals</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Rodríguez, Diego</au><au>Mačiulaitis, Rokas</au><au>Okraska, Jan</au><au>Šimko, Tibor</au><au>Kamleh, W.</au><au>Stewart, G.A.</au><au>Kim, D.</au><au>Doglioni, C.</au><au>Silvestris, L.</au><au>Jackson, P.</au><format>book</format><genre>proceeding</genre><ristype>CONF</ristype><atitle>Hybrid analysis pipelines in the REANA reproducible analysis platform</atitle><btitle>EPJ Web of Conferences</btitle><date>2020</date><risdate>2020</risdate><volume>245</volume><spage>6041</spage><pages>6041-</pages><issn>2100-014X</issn><issn>2101-6275</issn><eissn>2100-014X</eissn><abstract>We introduce the feasibility of running hybrid analysis pipelines in the REANA reproducible analysis platform. The REANA platform allows researchers to specify declarative computational workflow steps describing the analysis process and to execute analysis workload on remote containerised compute clouds. We have designed an abstract job controller component permitting to execute different parts of the analysis workflow on different compute backends, such as HTCondor, Kubernetes and SLURM. We have prototyped the designed solution including the job execution, job monitoring, and input/output file staging mechanism between the various compute backends. We have tested the prototype using several particle physics model analyses. The present work introduces support for hybrid analysis workflows in the REANA reproducible analysis platform and paves the way towards studying underlying performance advantages and challenges associated with hybrid analysis patterns in complex particle physics data analyses.</abstract><cop>Les Ulis</cop><pub>EDP Sciences</pub><doi>10.1051/epjconf/202024506041</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2100-014X
ispartof EPJ Web of Conferences, 2020, Vol.245, p.6041
issn 2100-014X
2101-6275
2100-014X
language eng
recordid cdi_doaj_primary_oai_doaj_org_article_e7ade15ec8d54813b37ccb99ffe2f34f
source Publicly Available Content Database (Proquest) (PQ_SDU_P3); Free Full-Text Journals in Chemistry
subjects Cloud computing
Control systems design
Particle physics
Pipelines
Workflow
title Hybrid analysis pipelines in the REANA reproducible analysis platform
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-27T10%3A54%3A10IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_doaj_&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=proceeding&rft.atitle=Hybrid%20analysis%20pipelines%20in%20the%20REANA%20reproducible%20analysis%20platform&rft.btitle=EPJ%20Web%20of%20Conferences&rft.au=Rodr%C3%ADguez,%20Diego&rft.date=2020&rft.volume=245&rft.spage=6041&rft.pages=6041-&rft.issn=2100-014X&rft.eissn=2100-014X&rft_id=info:doi/10.1051/epjconf/202024506041&rft_dat=%3Cproquest_doaj_%3E2465749298%3C/proquest_doaj_%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c360t-3ad23e9fd0121e47393b0145edad583842ef22f48ee07bc3d2c5cd9ce519b2d73%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2465749298&rft_id=info:pmid/&rfr_iscdi=true