This document discusses AWS Glue, a fully managed extract, transform, and load (ETL) service that makes it easy to move data between data sources like databases and Amazon S3. It automates the time-consuming tasks of data preparation and transformation so customers can focus on analyzing their data and gaining insights. The document outlines the key components of AWS Glue including the data catalog, job execution, and job authoring capabilities and how it addresses the challenges of hand-coding and maintaining ETL jobs over time. It also provides an overview of how to populate the data catalog using crawlers, author jobs that are run in a serverless manner, and monitor job executions.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cassandra Bonner
Solutions Architect, Amazon Web Services
Level 200
Connecting your Data Analytics Pipeline

2. Ingest
Data Analytics Landscape
Serving
Amazon
Redshift
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Speed (Near real-time)
Scale (Batch)
AWS Glue
Sources
Flat
Files
Amazon
S3
Amazon
RDS
On Prem DB

3. We Have Lots of ETL Partners
Amazon Redshift Partner Page for Data Integration

4. Why Would Anyone Hand-code?
Brittle Error-prone Laborious

5. You Also Need to Maintain this Code
► As data formats change
► As target schemas change
► As you add sources
► As data volume grows

6. Code is Flexible Code is Powerful
You can unit test You can deploy with other code You know your dev tools

7. ETL is the Most Time-consuming Part of Analytics
ETL Data Warehousing Business Intelligence
Large % of time
spent here
Amazon Redshift Amazon QuickSight

8. The Data Gap
This Leads to Dark Data
1990 2000 2010 2020
Generated Data
Data
Volume
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Centre Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares

9. Glue Automates The Undifferentiated
Heavy-lifting Of ETL

10. AWS Glue: High-Level View

11. AWS Glue: Components
Data Catalog
 Hive metastore compatible metadata repository of data sources.
 Crawls data source to infer table, data type, partition format.
Job Execution
 Runs jobs in Spark containers – automatic scaling based on SLA.
 Glue is serverless - only pay for the resources you consume.
Job Authoring
 Generates Python code to move data from source to destination.
 Edit with your favourite IDE; share code snippets using Git.

12. Glue Data Catalog
Discover and organise your data sets

13. Glue Data Catalog
We added a few extensions:
 Search metadata
 Connection info
 Classification
 Versioning

14. Crawlers: Populate Data Catalog
Crawler
Custom Classifiers
Built—in Classifiers
Amazon S3 Amazon
RDS
Amazon
Redshift
JDBC
Data Stores
3
1
2
Connection
4
5

15. POPULATE DATA CATALOG DEMO

16. Job Authoring in Glue
Make ETL job authoring like code development using your own tools

17. Authoring Jobs
1 Source
2 Target
3
4
Transform Data
6
Generates
5
Triggers
PySpark script

18. Automatic Code Generation
Domain Driven Design, Eric Evans

19. Orchestration & Resource Management
Fully managed, serverless job execution

20. Running & Monitoring Jobs
JDBC
Amazon S3
Amazon
RDS
Amazon
Redshift
Data Sources
JDBC
Amazon S3
Amazon
RDS
Amazon
Redshift
Data Targets
Triggers
1
2 Extracts data
Job
4 Loads data
5
3
Transforms
Runs

21. AUTHORING & RUNNING JOBS DEMO

22. Sign Up For Glue Preview
You can sign up for a preview at aws.amazon.com/glue.

23. Thank you!