Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.

StreamFlow: cross-breeding cloud with HPC

Colonnelli I.
First
;
Cantalupo B.;Aldinucci M.
Last
2021-01-01

Abstract

Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.
2021
9
4
1723
1737
https://ieeexplore.ieee.org/document/9177340
Cloud computing, Task analysis, Containers, Tools, Computer architecture, DSL, Pipelines
Colonnelli I.; Cantalupo B.; Merelli I.; Aldinucci M.
File in questo prodotto:
File Dimensione Formato  
2002.01558.pdf

Accesso aperto

Descrizione: Articolo principale
Tipo di file: PREPRINT (PRIMA BOZZA)
Dimensione 588.4 kB
Formato Adobe PDF
588.4 kB Adobe PDF Visualizza/Apri
StreamFlow_Cross-Breeding_Cloud_With_HPC.pdf

Accesso riservato

Descrizione: PDF Editoriale
Tipo di file: PDF EDITORIALE
Dimensione 1.38 MB
Formato Adobe PDF
1.38 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2318/1756840
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 33
  • ???jsp.display-item.citation.isi??? 21
social impact