DEBS grand challenge: real-time detection of air quality improvement with Apache Flink (CROSBI ID 709752)
Prilog sa skupa u zborniku | izvorni znanstveni rad | međunarodna recenzija
Podaci o odgovornosti
Marić, Josip ; Pripužić, Krešimir ; Antonić, Martina
engleski
DEBS grand challenge: real-time detection of air quality improvement with Apache Flink
The topic of the DEBS Grand Challenge 2021 is to develop a solution for detecting areas in which the air quality index (AQI) improved the most when compared to the previous year. The solution must run two given continuous queries in parallel on the incoming sensor data stream which must return the following: 1) a top 50 cities in terms of AQI improvement with their current AQIs and 2) a histogram of the longest streaks of good AQI. The incoming data is accessed through an API which provides streaming sensor measurements in batches. We present our solution based on Apache Flink, a distributed stream processing framework for the cluster. We opted for Flink since its applications can easily be scaled horizontally and vertically by adding computation nodes or increasing available resources, respectively. Flink allows us to divide the given queries into smaller tasks which can be run concurrently on different nodes in order to reduce the overall processing time and thus improve the performance of our solution. In more detail, the following performance intensive tasks are run in parallel on distributed nodes: 1) retrieving measurement batches, 2) assigning a city to each measurement and 3) calculating air quality index per city. We also discuss the main optimizations we have used to improve the performance and present an experimental evaluation of our solution.
sensor streams, big data, fast data, geospatial data streams
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
Podaci o prilogu
148-153.
2021.
objavljeno
10.1145/3465480.3466930
Podaci o matičnoj publikaciji
DEBS '21: Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems
The Association for Computing Machinery (ACM)
9781450385558
Podaci o skupu
15th ACM International Conference on Distributed and Event-based Systems
ostalo
28.06.2021-02.07.2021
Milano, Italija