During the 1990s with the rise of business intelligence and decision support there was a big push for loading data from disparate source systems into a consolidated data warehouse. Along with that, if you wanted to extract data, transform it, and load it somewhere else, you were in the market for an ETL or data integration tool: Ab Initio, IBM DataStage, Informatica PowerCenter, DMExpress, Data Transformation Services to name a few.
These tools provided a visual way to develop processes in which data could be conformed, combined, and loaded from many different sources and stored together in a unified database. However, as data volumes continued to grow and advancements in database technology around data warehousing concepts allowed for more powerful analytic processing, the database proved to be a powerful tool in performing those data transformations. This paradigm of extracting data, loading it, and then transforming it (ELT) lessened the need for a traditional data integration tool because the database itself could handle the load and transformation.
Fast forward to today, and in the world of data integration, the need for an ETL tool is back in full force. It’s still solving the same problem, but the equation has gotten a lot more complex. No longer are we only extracting data from mainframe systems, flat files, and other relational databases. We’re now dealing with real-time data from web services, structured and unstructured data, a myriad of data management platforms, and—to top it all off—this data may be stored locally or in the cloud. Additionally, as the growth rate in technology increases, we’re seeing both source and target systems change a lot more frequently so there is an increased need to decouple the business logic from the underlying storage system. This is where a data integration tool saves the day.
Enterprise data integration tools allow for the development of complex data transformation logic irrespective of the source of the data or where it will ultimately land. Add in the ability to push down that logic into other data management platforms, such as a relational database or Hadoop, and you’re ready to take advantage of all the changes in technology without re-developing your solution. So, while we’ve seen some shifts away from ETL tools in past, it looks like they are here to stay.