Jingwei Lu and Jason Zhang (Airbnb)
AirStream is a realtime stream computation framework built on top of Spark Streaming and HBase that allows our engineers and data scientists to easily leverage HBase to get real-time insights and build real-time feedback loops. In this talk, we will introduce AirStream, and then go over a few production use cases.
7. Stateful
Jingwei Lu, Liyin Tang, Jason Zhang
Computatio
n
Source
DStream DF DF
Sink1
Sink2
Sink N
State Storage
RDD
8. Multiple Streams
Jingwei Lu, Liyin Tang, Jason Zhang
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…
DataFrame
Sink1
Process
N
Sink2
Sink3
SinkN
…
Source
DStream
Align by Time
DataFram
e
DataFram
e
State
Storage
Source
DStream
…
9. Streaming + Batch
Jingwei Lu, Liyin Tang, Jason Zhang
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…
DataFram
e
State
Storage
Sourc
e
DStream
Sourc
e
…
Align by Time
…
DataFrame
Sink1
Process
A
Sink2
Sink3
SinkN
…
14. Jingwei Lu, Liyin Tang, Jason Zhang
State Store
• Merge changes
• Provide fast lookup
• Fast persistent storage across
streaming and batch jobs
14
15. Why HBase
Jingwei Lu, Liyin Tang, Jason Zhang
Rich Functionalities
Rich Integration with Hadoop EcoSystem
Easy Management
Strong Community
Reliable and Scalable
16. HBase State Store
Operators in Airstream
Jingwei Lu, Liyin Tang, Jason Zhang
16
Full Table Scan
Simple Aggregation
Bulk Upload
Key/Prefix Lookup
Update
17. Jingwei Lu, Liyin Tang, Jason Zhang
Computation DAG
17
Input Data
Left Outer Join Result
Key Lookup
18. Jingwei Lu, Liyin Tang, Jason Zhang
Key Space Design
• Hash partition key space
for load balance
• Composite key for K -> V
• Support full key lookup
• Prefix lookup supported for
all keys used in hash
function
Hash key1 key2 key3
Hash based on key prefix
Hash key1 key2
Lookup based on key prefix
key1 = ‘value1’ and key2 = ‘value2’
18
19. • Partition based on key before write
• Use bulk upload for large volume update
Write Performance
Jingwei Lu, Liyin Tang, Jason Zhang
19
20. Case Study
Jingwei Lu, Liyin Tang, Jason Zhang
Experiment realtime feedback
Update
Experiment Assignment
Event
Lookup
HBase
with TTL
Booking Event
Druid
Datado
g
20
one airstream job
22. Realtime Ingestion on HBase
Data Infrastructure
MySQL
Analytica
l Events
Kafka
Spark
Streamin
g
HBase
HDFS
Presto/Hive/Spar
k
Source
Inges
t
Realtime
Query
Snapsh
ot
Batch
Query
Jingwei Lu, Liyin Tang, Jason Zhang
22
23. Access Data in HBase
Jingwei Lu, Liyin Tang, Jason Zhang
HBase
Hive Presto
Spark
SQL
Spark
Streaming
Batch Jobs Interactive Query Streaming
HDFS
Snapshot
Table Mapping/Unifed View on realtime data
23
25. Case Study 1: Events Ingestion
Jingwei Lu, Liyin Tang, Jason Zhang
Kafka
topic
…
topic
topic
Spark
Executor
1
…
Executor
2
Executor
K
HBase
DeDup
HDFS
Region1…Region2R
egion M
Daily
Snapshot
Realtime
Query
Hive
Presto
Events
Partition
25
26. Case Study 2: Streaming DB Export
KafkaRDS
Table1
…
Spinalta
p.
Table1
…
Table2
TableN
Spinaltap.
Table2
Spinaltap.
TableN
Spark
Executor
1
…
Executor2
Executor K
HBase
Region1
…
Region2
Region M
HDFS
Region1…Region2R
egion M
Daily Snapshot
Realtime Query
Jingwei Lu, Liyin Tang, Jason Zhang
26
27. Case Study: Streaming DB Export
Rows CF: Colums Version Value
<ShardKey><DB_TABLE_#1><PK_a=A> id Fri May 19 00:33:19 2016 101
<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 19 00:33:19 2016 San Francisco
<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 10 00:34:19 2016 New York
<ShardKey><DB_TABLE_#2><PK_a=A’> id Fri May 19 00:33:19 2016 1
Jingwei Lu, Liyin Tang, Jason Zhang
27
28. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
TXN 2
Commit_TS:
102
TXN 3
Commit_TS:
103
TXN N
Commit_TS:
N’
Binlog Order
Jingwei Lu, Liyin Tang, Jason Zhang
28
29. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
TXN 2
Commit_TS:
103
TXN 3
Commit_TS:
102
TXN N
Commit_TS:
N’
NTP
Binlog Order
Jingwei Lu, Liyin Tang, Jason Zhang
29
30. Case Study: Streaming DB Export
TXN 1
Commit_TS:
101
…
Binlog Order
TXN 2
Commit_TS:
103
TXN 3
Commit_TS:
102
TXN N
Commit_TS:
N’
Point-in-Time Restore on TS 102
Jingwei Lu, Liyin Tang, Jason Zhang
30
31. Case Study: Streaming DB Export
Rows CF: Colums Version Value
<ShardKey><DB_TABLE_#1><PK_a=A> id bin100 101
<ShardKey><DB_TABLE_#1><PK_a=A> city bin101 San Francisco
<ShardKey><DB_TABLE_#1><PK_a=A> city bin102 New York
<ShardKey><DB_TABLE_#2><PK_a=A’> id bin100 1
Jingwei Lu, Liyin Tang, Jason Zhang
31
32. Case Study: Streaming DB Export
Rows Version (Logical Offset) Value
<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100
<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101
<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103
<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102
Jingwei Lu, Liyin Tang, Jason Zhang
32
33. Case Study: Streaming DB Export
Rows Version (Logical Offset) Value
<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100
<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101
<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103
<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102
Jingwei Lu, Liyin Tang, Jason Zhang
33
34. Summary
Jingwei Lu, Liyin Tang, Jason Zhang
Scalable and Reliable
Rich Stateful Computation
Rich Integration with Hadoop EcoSystem
Easy Operation
Realtime ingestion provides fast feedback loop.
Advanced monitoring infrastructure
Tracking changes instead of full snapshot for RDS dump
What is the goal of realtime ingestion:
*fast feedback loop for experiment to reduce testing cycle
*provide realtime view of production database for many offline workload(for example, machine learning)
Table mapping provide a unified view to access realtime ingested data.
For snapshot using scan it takes 10-30 minutes per table. This does not scale.
Take 10 minutes to do the link and restore. All tables can be accessed afterward.
Backup based db export restore takes 9 - 12 hours and it is subject to AWS network situation. Long latency and fragile.
We just need to track changes and apply to snapshot.
Provide near realtime snapshot of db. Unify across mysql and dynamodb