Delta View Generation for Incremental Loading of Large Dimensions in a Data Warehouse (CROSBI ID 624936)
Prilog sa skupa u zborniku | izvorni znanstveni rad | međunarodna recenzija
Podaci o odgovornosti
Mekterović, Igor ; Brkić, Ljiljana
engleski
Delta View Generation for Incremental Loading of Large Dimensions in a Data Warehouse
Incremental load is the preferred approach in efficient ETL processes. Fact tables are the ones who benefit the most from this approach, since they are large in terms of row count. For the sake of simplicity, dimension tables are often ignored and populated in a full reload manner. However, big dimensions (e.g. Client) can also have a significant impact on the ETL process and should also be considered for incremental load. Although they have much smaller cardinality than a typical fact table, it usually takes much more resources to calculate one dimension table row than to calculate one fact table row. Large dimension tables are based on multiple source tables, and it is not trivial to determine the changed records that should be considered for the incremental load because changes in any and all of underlying source tables must be considered. In this paper, we present an algorithm for the dimension’s delta view generation. Delta view for a dimension encompasses all its source tables and produces a set of keys (e.g. ClientIds) that should be incrementally processed. We have employed this approach in a real world project and have noticed a significant reduction in the loading time of big dimensions.
data warehouse; incremental etl; delta view; algorithm
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
nije evidentirano
Podaci o prilogu
1417-1422.
2015.
objavljeno
Podaci o matičnoj publikaciji
Hrvatska udruga za informacijsku i komunikacijsku tehnologiju, elektroniku i mikroelektroniku - MIPRO
978-9-5323-3082-3
1847-3946
Podaci o skupu
MIPRO 2015
predavanje
25.05.2015-29.05.2015
Opatija, Hrvatska