Presto @ Netflix (Hadoop Summit'15)

1. Presto @ Netflix: Interactive
Queries at Petabyte Scale
Nezih Yigitbasi and Zhenxiao Luo

2. Outline
Our big data platform
Presto @ Netflix
Netflix integration
Our contributions
What’s next?

3. Cloud
Apps
S3
Suro/Kafka Ursula
SSTable
s
Cassandra Aegisthus
Event Data
500 bn/day, 15m
Daily
Dimension Data
Netflix Data Pipeline

4. Data
Warehouse
Service
Tool
s
Gateways
Our Big Data Platform
Prod
Clients
Clusters
Query Prod TestProd
Big Data API/Portal
Metacat

5. Our Use Cases
Batch jobs (Pig, Hive)
ETL jobs
reporting and other analysis
Interactive jobs

6. Presto @ Netflix

7. What is Presto?
An open source distributed SQL engine
for running interactive queries against
large datasets

8. Why we love Presto?
Fast

9. 0
200
400
600
800
Group By Join + Group By Needle in
Haystack
Presto Hive
QueryCompletionTime[s]

10. Why we Love Presto?
Fast
Scalable
ANSI SQL
Open source
Works well on AWS
Hadoop friendly

11. presto-cli, Python, R, BI tools (ODBC/JDBC), etc.
Our Deployment
v 0.100
Java 8™
1 coordinator (r3.4xlarge)
~220 workers (r3.4xlarge)
Clients

12. 15+ PB
Total data size
2.5K
Queries/day
300+
Presto users
Data Size
100MB 1GB 1TB 10TB
0
20
40
60
80
100
%ofQueries
Query Runtime
0
20
40
60
80
100
4s 1m 5m 10m
%ofQueries

13. Netflix Integration

14. S3
Atlas
Sidecar
PrestoAmazon EMR
Amazon
RDS
HCat
Server
Coordinator
Worker

15. S3
Atlas
Sidecar
PrestoAmazon EMR
Data Lineage
query
completion
events

16. S3
Atlas
Sidecar
PrestoAmazon EMR
Monitoring
metrics

17. S3
Suro
Atlas
Sidecar
PrestoAmazon EMR
BI Tools

18. Our Contributions

19. S3 Filesystem
Query Optimizer
Parquet File Format
Complex Types
Multipart upload
Instance credentials
Role support
Reliability
Single distinct => Group By
Joins with similar subqueries
Schema evolution
Parquet 1.6
Various new
functions
Comparability

20. presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
Worker
Worker
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Functions
Type
System
1
2 3
4
5
6
7

21. Single Distinct => Group By
select
count(distinct c)
from t
select count(*)
from (select c
from t
group by c)
Output
Count
Aggregation
masks = {column$distinct}
Distinct
marker = column$distinct
Table Scan
Output
Count
Aggregation
masks = {}
Group By
Aggregation
count
Table Scan

22. Joins with Similar Subqueries
select *
from (select k,
agg1,
agg2
from t
group by k) a
join (select k,
agg3,
agg4
from t
group by k) b
on ( a.k = b.k )
Output
Table Scan
table = t
Join
key= k
Group By
Aggregation
key= k
agg1, agg2
Group By
Aggregation
key= k
agg3, agg4
Table Scan
table = t

23. Output
Table Scan
table = t
Group By
Aggregation
key= k
agg1, agg2, agg3, agg4
select k, agg1,
agg2, agg3,
agg4
from t
group by k
Joins with Similar Subqueries

24. presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
Worker
Worker
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Functions
Type
System
1
2 3
4
5
6
7

25. Complex Type Support
map_agg()
map_keys()
map_values()
map<K,V> row(F T)
=, !=
bug fixes
array<T>
array_join()
sort_array()
concat()
=, !=, <, >

26. presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
S3
Filesystem
Worker
Worker
S3
Filesystem
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Functions
Type
System
1
2 3
4
5
6
7

27. Presto S3 FileSystem
(multipart upload, instance/static credentials,
assume role, reliability, etc.)
S3
open() seek() list()
Get Object Get Object
Metadata
List Objects

28. presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
S3
Filesystem
Worker
Worker
S3
Filesystem
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Functions
Type
System
1
2 3
4
5
6
7
Parquet
Cursor
Parquet
Cursor

29. RowGroup Metadata
codec, encoding, etc.
Column Chunk
Page
Page
Page
Column Chunk
Page
Page
Page
Column Chunk
Page
Page
Page
RowGroup
Footer
schema, version, etc.
Column Metadata
value count
size,
min, max
Column Metadata
value count
size,
min, max
Column Metadata
value count
size,
min, max

30. What’s next?
Parquet optimizations
vectorized reader
predicate pushdown
lazy load
lazy decompression/decoding
Better resource management
Better BI tool integration

31. THANK YOU