CLOUDFLOW - Enabling Faster Biomedical Pipelines with Mapreduce and Spark (CROSBI ID 227776)
Prilog u časopisu | izvorni znanstveni rad | međunarodna recenzija
Podaci o odgovornosti
Forer, Lukas ; Afgan, Enis ; Weissenteiner, Hansi ; Davidović, Davor ; Specht, Guenther ; Kronenberg, Florian ; Schoenherr, Sebastian
engleski
CLOUDFLOW - Enabling Faster Biomedical Pipelines with Mapreduce and Spark
For many years Apache Hadoop has been used as a synonym for processing data in the MapReduce fashion. However, due to the complexity of developing MapReduce applications, adoption of this paradigm in genetics has been limited. To alleviate some of the issues, we have previously developed Cloudflow - a high-level pipeline framework that allows users to create sophisticated biomedical pipelines using predefined code blocks while the framework automatically translates those into the MapReduce execution model. With the introduction of the YARN resource management layer, new computational processing models such as Apache Spark are now plugable into the Hadoop ecosystem. In this paper we describe the extension of Cloudflow to support Apache Spark without any adaptions to already implemented pipelines. The described performance evaluation demonstrates that Spark can bring an additional boost for analysing next generation sequencing (NGS) data to the field of genetics. The Cloudflow framework is open source and freely available at https://github.com/genepi/cloudflow.
Apache YARN ; Pipeline Framework ; Spark ; Cloud Computing
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
Podaci o izdanju
17 (2)
2016.
103-114
objavljeno
1895-1767
10.12694/scpe.v17i2.1159
Povezanost rada
Računarstvo