Naslov
Towards Enabling Big Data and Federated Computing in the Cloud

Autori
Kowsar, Yousef ; Afgan, Enis

Vrsta, podvrsta i kategorija rada
Sažeci sa skupova, sažetak, znanstveni

Skup
Bioinformatics Open Source Conference (BOSC 2013)

Mjesto i datum
Berlin, Njemačka, 19.07.2013. - 20.07.2013

Vrsta sudjelovanja
Predavanje

Vrsta recenzije
Međunarodna recenzija

Ključne riječi
Distributed computing; federated computing; big data

Sažetak
As the information age continues, the rate at which data is produced is continuing its exponential growth. Although often primarily described as a challenge and an obstacle, the reality is that the availability of the increasing data volume presents enormous opportunities. The real power of the data will not come just from the sheer volume, but from the ability to analyze it. It is thus vital to provide flexible yet accessible solutions that enable researchers to move beyond the data collection and step into the world of data analytics. Simultaneously, Infrastructure-as-a-Service (IaaS) compute infrastructure model (i.e., the cloud) has showcased its ability to transform how access to compute resources is realized ; it delivered on the notion of Infrastructure-as-Code and enabled a new wave of compute adaptability. Over the past few years, we have been developing CloudMan as a versatile solution for enabling and managing compute clusters in cloud environments via a simple web interface or an API. However, CloudMan only supported batch processing workloads. As the magnitude of the data produced and processed in digital form grows, the need to support more types of applications in the cluster-in-the-cloud model is becoming more evident. We have thus extended the batch processing capability of CloudMan and added support for different types of analysis workloads to the created cloud environment. This was accomplished by adopting and utilizing a well established big data platform component: Hadoop [9] and adding support for federated computing using HTCondor. With these additions, CloudMan provides support for three types of workloads: batch, Hadoop, and federated. This talk will discuss the new opportunities behind this solution, the developed architecture, and showcase usability of the available implementation.

Izvorni jezik
Engleski

Znanstvena područja
Biologija, Računarstvo

POVEZANOST RADA