High level architectural modelling for representing the extract, transform and load process

Abstract

This work in progress paper identifies the necessity for high level architectural modelling when designing Extract, Transform and Load (ETL) processes for updating Data Warehouses (DW). ETL processes have become quite complex when they operate over the prevalent heterogeneous systems and the myriad of deployed technologies. The focus of previous works has been mainly on process modelling at a low level ignoring the architectural specification. The many solutions given in literature, and presented here in the related works part, are tied to very specific scenarios. In this paper the problem definition is formulated and a basic practical solution for conceptually modelling an ETL process using UML class diagrams is given. The approach that is being presented here will allow designers of ETL processes to create top-down and bottom-up solutions that are more implementable in respect to the resources, technologies, and available data. This solution should simplify the architectural representation required to describe the data sources and the ETL processes. Various key problems that remain unresolved are mentioned and discussed.