Is there a difference between ETL and Data Engineering?

1. Big Data Analytics: From
ETL to Data Engineering
By Dmitry Tolpeko – November 2016

2. Is Data Engineering Just a New Term?
 Is there any difference between ETL and Data Engineering? Or do they mean
the same thing?
 I think they are very different, and I will try to explain.

3. What is ETL?
 ETL means Extract-Transform-Load, and it is a very important part of building
and maintaining a Data Warehouse
 Extract: Export data from one or many sources.
 Transform (at various stages): Clean, transform, aggregate etc. data
 Load (at various stages): Load data into the data warehouse, data marts i.e.
make data available for BI tools.

4. ETL Specifics
 Transform: Mostly SQL and Procedural SQL
 SQL Analytic (Window) functions such as LEAD/LAG for complex calculations
 Data in Data Warehouse: Often redundant, duplicated and pre-aggregated
for performance reasons. But in most cases data is in first normal form i.e.
columns contain atomic values.
Also data is in Star schema: a set of dimension and fact tables.
 Data Consumers: BI tools that mostly visualize data in interactive (filter,
drill-down etc.) reports and dashboards.

5. Big Data Analytics
 Data Scientists are the first consumers of data now.
 Feature engineering is the major step of building statistical and machine
learning models for advanced analytics:
 Feature vectors containing hundreds of elements with each element
containing a list, map, nested structs, buckets, list of maps, map of maps
and so on.
 Complex Calculations to create feature vectors within a data window (reduce
phase) require a non-SQL approach.

6. Skills: ETL Developer vs Data Engineer
 ETL Developer:
 SQL and procedural SQL
 Analytic SQL functions
 Data warehouse design and modelling
 Data Engineer:
 Python (Scala, Java), Spark, Pig, SQL
 Distributed data processing concepts (MapReduce, Spark)
 Statistics and machine learning concepts

7. Thank you!
Dmitry Tolpeko
dmtolpeko@gmail.com
http://www.dmtolpeko.com
@dmtolpeko