software etl a thing of the past

AWS Aim to Make Extract Transform and Load a Thing of the Past

No more ETL

Another announcement from AWS re:Invent reveals that Amazon is eliminating the need to perform Extract Transform and Load (ELT) operations. After years of development, AWS say that with Aurora zero-ETL users can combine data from multiple sources into a single data store completely ETL-free. Aurora zero-ETL is designed to integrate with Amazon Redshift and a similar integration is available for Redshift and Apache Spark.

When talking about the new capabilities at the conference, Adam Selipsky AWS CEO said:

You may have data in a number of different places like your application usage data in a database and customer reviews in your data lake. Putting them together has been a significant challenge up until now.

As the range of database types, data formats and size of data for enterprises has exponentially grown over the years, performing ETL has become more complex and one of the hardest workloads to automate. Engineers often have to write code to establish connections, build pipelines, schedule data and perform O&M.

As a solution, over the last few years AWS has been working to make integration between services easier. With the aim to completely eliminate ELT for analytics and machine learning.

Adam Selipsky went on to say,

What if we could eliminate ETL entirely? That would be a world we would all love. This is our vision, what we’re calling a zero ETL future. And in this future, data integration is no longer a manual effort. So today I’m excited to announce the preview of a fully managed new ETL-free integration between Aurora and Redshift.

The Amazon Redshift and Apache spark zero-ETL integration is currently available across all regions. What are you going to do with all your extra time?

James Capell

Technical editor and journalist. I have a particularly strong interest in NLP, AI ethics and cyber crime. Not too fond of cats.