engleski

Benchmarking Apache Beam for IoT Applications

The pervasiveness of computational and communication devices, coupled with innovative Internet of Things (IoT) application scenarios, led to a massive increase in the number of available data sources and the volume of data produced per unit of time. This contributed to the emergence of many open-source streaming data processing systems of different characteristics and diverging performances in specific usage scenarios. The heterogeneity of platforms, programming languages, and models used in such systems resulted in the prohibitively complex effort required to quickly and efficiently test their suitability for specific use cases. Apache Beam framework aims to introduce the unifying programming model for data processing systems, tackling the heterogeneity problem and allowing for fast testing of performance in specific usage scenarios and migration between different platforms. In order to test the maturity of the Apache Beam framework and its performance in processing data from the IoT domain, we constructed a benchmarking environment employing data sets with significant spatio-temporal properties along with a representative set of streaming operations for such data. We used Apache Kafka as data source and result collection within different computational resource configurations hosting execution engines. For each combination of the computational resource configuration, execution engine, and streaming operation, we measured and compared performance in terms of average throughput and its variance. The results show that the evaluated Flink and Spark runners, deployed on the single machine, manifest the law of diminishing returns rather quickly with regards to the number of available cores. While Spark runner core throughput significantly outperforms Flink runner’s core throughput, Spark runner’s system throughput is consistently lower than Flink ones.

Apache Beam ; streaming ; benchmarking

nije evidentirano