Naslov
Benchmarking Apache Beam for IoT Applications

Autori
Žaja, Mateo ; Lipić, Tomislav ; Čavrak, Igor

Vrsta, podvrsta i kategorija rada
Radovi u zbornicima skupova, cjeloviti rad (in extenso), znanstveni

Izvornik
44rd MIPRO 2021 International Convention - Proceedings / Skala, Karolj - Opatija : Hrvatska udruga za informacijsku i komunikacijsku tehnologiju, elektroniku i mikroelektroniku - MIPRO, 2021, 298-303

Skup
44th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO 2021)

Mjesto i datum
Opatija, Hrvatska, 27.09.2021. - 01.10.2021

Vrsta sudjelovanja
Predavanje

Vrsta recenzije
Međunarodna recenzija

Ključne riječi
Apache Beam ; streaming ; benchmarking

Sažetak
The pervasiveness of computational and communication devices, coupled with innovative Internet of Things (IoT) application scenarios, led to a massive increase in the number of available data sources and the volume of data produced per unit of time. This contributed to the emergence of many open-source streaming data processing systems of different characteristics and diverging performances in specific usage scenarios. The heterogeneity of platforms, programming languages, and models used in such systems resulted in the prohibitively complex effort required to quickly and efficiently test their suitability for specific use cases. Apache Beam framework aims to introduce the unifying programming model for data processing systems, tackling the heterogeneity problem and allowing for fast testing of performance in specific usage scenarios and migration between different platforms. In order to test the maturity of the Apache Beam framework and its performance in processing data from the IoT domain, we constructed a benchmarking environment employing data sets with significant spatio-temporal properties along with a representative set of streaming operations for such data. We used Apache Kafka as data source and result collection within different computational resource configurations hosting execution engines. For each combination of the computational resource configuration, execution engine, and streaming operation, we measured and compared performance in terms of average throughput and its variance. The results show that the evaluated Flink and Spark runners, deployed on the single machine, manifest the law of diminishing returns rather quickly with regards to the number of available cores. While Spark runner core throughput significantly outperforms Flink runner’s core throughput, Spark runner’s system throughput is consistently lower than Flink ones.

Izvorni jezik
Engleski

Znanstvena područja
Računarstvo

POVEZANOST RADA