Presto is a distributed SQL query engine used at Facebook for analytics workloads like warehouse ETL, dashboards, and A/B testing. Over the past year there have been improvements to workload management, security, execution performance, and language features. Upcoming work includes improving scalability, adding more connectors, and advanced query planning techniques like dynamic filtering. Future projects aim to enhance failure recovery, temporary tables, and introduce user-defined functions.

1. State of the Project
Martin Traverso July 16, 2018

2. Presto turns 6!

3. Presto @ Facebook
• Multiple uses
• Warehouse ETL and ad-hoc analytics
• Dashboards
• Analytics backend for A/B testing system
• Analytics backed for user facing products
• 1000s of nodes across several clusters
• 100s of PBs and quadrillions of rows processed/day
• > 70% of new warehouse ETL workloads on Presto

5. Since last year's meetup...
• 30 releases (0.176 to 0.205)
• ~3,500 commits (13,800 total)
• ~140 contributors (330 total)

6. Workload management
• Fairness in local scheduler
• Resource group improvements
• Query runtime limits
Since last year's meetup...

7. Security
• Authentication
• Client certificates
• Password
• JSON Web Tokens (RFC 7519)
• Worker-to-worker encryption
Since last year's meetup...

8. Security
• Column-level access control
public interface ConnectorAccessControl
{
...
default void checkCanSelectFromColumns(
ConnectorTransactionHandle transaction,
Identity identity,
SchemaTableName table,
Set<String> columns)
}
Since last year's meetup...

9. Execution
• Partitioned (a.k.a "bucket by bucket") query execution
• Distributed sort
• Dynamic writer scaling
• Support for JOIN spilling
• Improved memory accounting
• Java 9 support
• ... many, many performance and reliability improvements
Since last year's meetup...

10. Language
• Geospatial functions and optimizations (SQL/MM Part 3)
• Language constructs
• GROUPING
• CURRENT_USER
• New data types
• IPADDRESS
• SET_DIGEST
Since last year's meetup...

11. Language
• Decimal literals
123.45678 DECIMAL(8,5)
.12345 DECIMAL(5,5)
1e5 DOUBLE
0.3e-4 DOUBLE
123.456e2 DOUBLE
Since last year's meetup...

12. Language
• Ordered aggregates
SELECT id, array_agg(value ORDER BY index)
FROM t,
UNNEST(t.elements) WITH ORDINALITY
AS u(index, value)
GROUP BY id
Since last year's meetup...

13. Language
• LATERAL join
SELECT orderkey, total
FROM orders o, LATERAL (
SELECT sum(totalprice) total
FROM orders
WHERE custkey = o.custkey
AND orderdate <= o.orderdate
AND orderpriority > o.orderpriority
) t
Since last year's meetup...

14. Connectors
• TPC-DS
• Redshift
• Thrift
• Cassandra
• INSERT support
Since last year's meetup...

15. What's next

16. Scalability
• Larger clusters (> ~1000 nodes)
• HTTP/2
• Multiple query coordinators
What's next

17. Language
• TIMESTAMP semantics
• (c1, c2, ..., cN) IN <subquery>
• Improved correlated subqueries
• Equality for complex types containing NULLs
• Function namespaces (SQL path)
What's next

18. Connectors
• Elasticsearch
• Apache Phoenix
• Kudu
What's next

19. and more..
• Cost-based join type selection and reordering
• Dynamic filtering
• Warnings framework
• Role management
What's next

20. Future Projects

21. Execution
• Failure recovery
• Automatic skew handling
• Workload re-balancing
• Temporary tables
• Multi-phase query executions
Future Projects

22. Language
• User-defined functions
• SQL 2016 features
• Polymorphic table functions
• JSON syntax
• Row pattern matching
Future Projects

23. Others
• Client protocol overhaul
• Plan IR overhaul
• Extensible optimizer rules
• User-defined functions
Future Projects