This is a talk that I gave at the Data Council Berlin Meetup on May 16th, 2019 Abstract: Stream processing is being rapidly adopted by the enterprise. While in the past, stream processing frameworks mostly provided Java- or Scala-based APIs, stream processing with SQL is growing increasingly popular because it makes stream processing accessible to non-programmers and significantly reduces the effort to solve common tasks. About three years ago, the Apache Flink community started adding SQL support to process static and streaming data in a unified fashion. Today, Flink SQL powers production systems at Alibaba, Huawei, Lyft, and Uber. Fabian Hueske discusses the current state of Flink’s SQL support and explains the importance of Flink’s unified approach to process static and streaming data. After covering the basics, he shares common real-world use cases ranging from low-latency ETL to pattern detection and demonstrates how easily they can be addressed with Flink SQL.

1. © 2019 Ververica
Fabian Hueske – Software Engineer
Apache Flink® SQL in Action

2. © 2019 Ververica2
About Me
• Apache Flink PMC member & ASF member
‒Contributing since day 1 at TU Berlin
‒Focusing on Flink’s relational APIs since ~3.5 years
• Co-author of “Stream Processing with Apache Flink”
• Co-founder of Ververica (formerly data Artisans)

3. © 2019 Ververica3
What is Apache Flink?
Stateful computations over streams
real-time and historic
fast, scalable, fault tolerant, in-memory
event time, large state, exactly-once

4. © 2019 Ververica4
Hardened at Scale
Streaming Platform Service
billions messages per day
A lot of Stream SQL
Streaming Platform as a Service
3700+ container running Flink,
1400+ nodes, 22k+ cores, 100s of jobs,
3 trillion events / day, 20 TB state
Fraud detection
Streaming Analytics Platform
1000s jobs, 100.000s cores,
10 TBs state, metrics, analytics,
real time ML,
Streaming SQL as a platform

5. © 2019 Ververica5
Powered by Apache Flink

6. © 2019 Ververica6
Flink’s Powerful Abstractions
Process Function (events, state, time)
DataStream API (streams, windows)
SQL / Table API (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Layered abstractions to
navigate simple to complex use cases

7. © 2019 Ververica7
Flink’s Relational APIs
Unified APIs for batch & streaming data
A query specifies exactly the same result
regardless whether its input is
static batch data or streaming data.
tableEnvironment
.scan("clicks")
.groupBy('user)
.select('user, 'url.count as 'cnt)
SELECT user, COUNT(url) AS cnt
FROM clicks
GROUP BY user
LINQ-style Table APIANSI
SQL

8. © 2019 Ververica8
Query Translation
tableEnvironment
.scan("clicks")
.groupBy('user)
.select('user, 'url.count)
SELECT user, COUNT(url)
FROM clicks
GROUP BY user
Input data is
bounded
(batch)
Input data is
unbounded
(streaming)

9. © 2019 Ververica9
What if “Clicks” is a File?
Clicks
user cTime url
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
user cnt
Mary 2
Bob 1
Liz 1
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
Input data is
read at once
Result is produced
at once

10. © 2019 Ververica10
What if “Clicks” is a Stream?
user cTime url
user cnt
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
Clicks
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
Bob 1
Liz 1
Mary 1Mary 2
Input data is
continuously read
Result is
continuously updated
The result is the same!

11. © 2019 Ververica11
Why is Stream-Batch Unification Important?
• Usability
‒ ANSI SQL syntax: No custom “StreamSQL” syntax.
‒ ANSI SQL semantics: No stream-specific result semantics.
• Portability
‒ Run the same query on bounded & unbounded data
‒ Run the same query on recorded & real-time data
‒ Bootstrapping query state or backfilling results from historic data
now
bounded query
unbounded query
past future
bounded query
start of the stream
unbounded query

12. © 2019 Ververica12
Database Systems Run Queries on Streams
• Materialized views (MV) are similar to regular views,
but persisted to disk or memory
‒Used to speed-up analytical queries
‒MVs need to be updated when the base tables change
• MV maintenance is very similar to SQL on streams
‒Base table updates are a stream of DML statements
‒MV definition query is evaluated on that stream
‒MV is query result and continuously updated

13. © 2019 Ververica13
Continuous Queries in Flink
• Core concept is a “Dynamic Table”
‒ Dynamic tables are changing over time
• Queries on dynamic tables
‒ produce new dynamic tables (which are updated based on input)
‒ do not terminate
• Stream ↔ Dynamic table conversions

14. © 2019 Ververica14
Stream ↔ Dynamic Table Conversions
•A stream is the changelog of a dynamic table
–As change messages are ingested from a stream, a table evolves
–As a table evolves, change messages are emitted to a stream
•Different changelog interpretations
–Append-only change messages
–Upsert change messages
–Add/Retract change messages

15. © 2019 Ververica15
How Can I Use It?
• Embed SQL queries in regular Flink applications
‒ Tight integration with DataStream and DataSet APIs
‒ Mix and match with other libraries (CEP, ProcessFunction, Gelly)
‒ Package and operate queries like any other Flink application
• Run SQL queries via Flink’s SQL CLI Client
‒ Interactive mode: Submit query and inspect results
‒ Detached mode: Submit query and write results to sink system

16. © 2019 Ververica16
Show Results
Enter Query
SELECT
user,
COUNT(url)
FROM clicks
GROUP BY user
Database /
HDFS
Event Log
Query
State
ResultsChangelog
or Table SQL CLI Client
Catalog
Optimizer
CLI
Result Server
Submit Query Job
SQL CLI Client – Interactive Queries

17. © 2019 Ververica17
The New York Taxi Rides Data Set
• A public data set about taxi rides in New York City
• Rides are ingested as append-only (streaming) table
‒ Each ride is represented by a start and an end event
• Table: Rides
rideId: BIGINT // ID of the taxi ride
taxiId: BIGINT // ID of the taxi
isStart: BOOLEAN // flag for pick-up (true) or drop-off (false) event
lon: DOUBLE // longitude of pick-up or drop-off location
lat: DOUBLE // latitude of pick-up or drop-off location
rideTime: TIMESTAMP // time of pick-up or drop-off event
psgCnt: INT // number of passengers

18. © 2019 Ververica18
Compute Basic Statistics
SELECT
psgCnt,
COUNT(*) as cnt
FROM Rides
WHERE isStart
GROUP BY
psgCnt
▪ Count rides per number of passengers.

19. © 2019 Ververica19
Identify Popular Pick-Up / Drop-Off Locations
SELECT
area,
isStart,
TUMBLE_END(rideTime, INTERVAL '5' MINUTE) AS cntEnd,
COUNT(*) AS cnt
FROM (SELECT rideTime, isStart, toAreaId(lon, lat) AS area
FROM Rides)
GROUP BY
area,
isStart,
TUMBLE(rideTime, INTERVAL '5' MINUTE)
▪ Compute every 5 minutes for each area the
number of departing and arriving taxis.

20. © 2019 Ververica20
Average Tip Per Hour of Day
SELECT
HOUR(r.rideTime) AS hourOfDay,
AVG(f.tip) AS avgTip
FROM
Rides r,
Fares f
WHERE
NOT r.isStart AND
r.rideId = f.rideId AND
f.payTime BETWEEN r.rideTime - INTERVAL '5' MINUTE AND r.rideTime
GROUP BY
HOUR(r.rideTime);
▪ Compute the average tip per hour of day. Fare data is
stored in a separate table Fares that needs to be joined.

21. © 2019 Ververica21
SQL Feature Set in Flink 1.8.0
STREAMING ONLY
• OVER / WINDOW
‒ UNBOUNDED / BOUNDED
PRECEDING
• INNER JOIN with time-versioned
table
• MATCH_RECOGNIZE
‒ Pattern Matching/CEP (SQL:2016)
BATCH ONLY
• UNION / INTERSECT / EXCEPT
• ORDER BY
STREAMING & BATCH
• SELECT FROM WHERE
• GROUP BY [HAVING]
‒ Non-windowed
‒ TUMBLE, HOP, SESSION windows
• JOIN
‒ Time-Windowed INNER + OUTER
JOIN
‒ Non-windowed INNER + OUTER JOIN
• User-Defined Functions
‒ Scalar
‒ Aggregation
‒ Table-valued

22. © 2019 Ververica22
What Can I Build With That?
• Data Pipelines & Low-latency ETL
‒ Transform, aggregate, and move events in real-time
‒ Write streams to file systems, DBMS, K-V stores, …
‒ Ingest appearing files to produce streams
• Stream & Batch Analytics
‒ Run analytical queries over bounded and unbounded data
‒ Query and compare historic and real-time data
• Power Live Dashboards
‒ Compute and update data to visualize in real-time

23. © 2019 Ververica23
SQL CLI Client – Detached Queries
SQL CLI Client
Catalog
Optimizer
Database /
HDFS
Event Log
Query
Submit Query Job
State
CLI
Result Server
Database /
HDFS
Event Log
Enter Query
INSERT INTO
sinkTable
SELECT
user,
COUNT(url) AS cnt
FROM clicks
GROUP BY user
Cluster & Job ID

24. © 2019 Ververica24
Serving a Dashboard
Elastic
Search
Kafka
INSERT INTO AreaCnts
SELECT
toAreaId(lon, lat) AS areaId,
COUNT(*) AS cnt
FROM TaxiRides
WHERE isStart
GROUP BY toAreaId(lon, lat)

25. © 2019 Ververica25
Try Flink SQL Yourself!
•The demo setup is available as a free online training
–Slides to learn the basics
–Exercises on streaming data
https://github.com/ververica/sql-training

26. © 2019 Ververica26
There’s a Lot More To Come!
• Alibaba is contributing features of its Flink fork Blink back
• Major improvements for SQL and Table API
–Better coverage of SQL features: Full TPC-DS support
–Competitive performance: 10x compared to current state
–Improved connectivity: External catalogs (Hive) and connectors
• Extending the scope of Table API
–Expand support for user-defined functions
–Add support for machine-learning learning pipelines & algorithm library

27. © 2019 Ververica27
Summary
• Unification of stream and batch is important.
• Flink’s SQL solves many streaming and batch use cases.
• In production at Alibaba, Huawei, Lyft, Uber, and others.
• Query deployment as application or via CLI
• Expect major improvements for batch SQL soon!

28. © 2019 Ververica

29. © 2019 Ververica
@VervericaDatawww.ververica.com

30. © 2019 Ververica30
Average Ride Duration Per Pick-Up Location
SELECT pickUpArea,
AVG(timeDiff(s.rowTime, e.rowTime) / 60000) AS avgDuration
FROM (SELECT rideId, rowTime, toAreaId(lon, lat) AS pickUpArea
FROM TaxiRides
WHERE isStart) s
JOIN
(SELECT rideId, rowTime
FROM TaxiRides
WHERE NOT isStart) e
ON s.rideId = e.rideId AND
e.rowTime BETWEEN s.rowTime AND s.rowTime + INTERVAL '1' HOUR
GROUP BY pickUpArea
▪ Join start ride and end ride events on rideId and
compute average ride duration per pick-up location.