engleski

Towards Enabling Big Data and Federated Computing in the Cloud

As the information age continues, the rate at which data is produced is continuing its exponential growth. Although often primarily described as a challenge and an obstacle, the reality is that the availability of the increasing data volume presents enormous opportunities. The real power of the data will not come just from the sheer volume, but from the ability to analyze it. It is thus vital to provide flexible yet accessible solutions that enable researchers to move beyond the data collection and step into the world of data analytics. Simultaneously, Infrastructure-as-a-Service (IaaS) compute infrastructure model (i.e., the cloud) has showcased its ability to transform how access to compute resources is realized ; it delivered on the notion of Infrastructure-as-Code and enabled a new wave of compute adaptability. Over the past few years, we have been developing CloudMan as a versatile solution for enabling and managing compute clusters in cloud environments via a simple web interface or an API. However, CloudMan only supported batch processing workloads. As the magnitude of the data produced and processed in digital form grows, the need to support more types of applications in the cluster-in-the-cloud model is becoming more evident. We have thus extended the batch processing capability of CloudMan and added support for different types of analysis workloads to the created cloud environment. This was accomplished by adopting and utilizing a well established big data platform component: Hadoop [9] and adding support for federated computing using HTCondor. With these additions, CloudMan provides support for three types of workloads: batch, Hadoop, and federated. This talk will discuss the new opportunities behind this solution, the developed architecture, and showcase usability of the available implementation.

Distributed computing; federated computing; big data

nije evidentirano