Running Flink on resource constrained edge devices is hard. There are two major obstacles: a) It consumes too much memory; b) Deploying a Flink cluster on remote device is not easy. This talk we will present a lightweight runtime implementation of Flink, which lowers down the memory consumption of the framework below 10 MB; Jobs are statically compiled to native binary, and can be run independently without any dependency, easy to deploy

1. Extend Flink to edge computing
with much lower footprint
Youjun Yuan(袁尤军)

2. Goal
Run Flink streaming job on most of the edge devices

3. Problems to run Flink job on edge
• Limited resources
‣ Most edge devices have < 500MB free memory
• Not easy to deploy
‣ Requires java
‣ A full Flink Cluster needed to be running

4. Solution
• A new memory efficient runtime framework Creek
‣ Golang was selected
‣ Including: windowing, time and watermark, system operators, built-
in functions, fault-tolerant, type system, connectors, deser, etc
• Independent and executable job
‣ Compile in the cloud, run on the edge
‣ Statically linked exe
Cloud
Edge
Build
Run

5. Architecture - job generation
Sql job Parser
Validate
Optimize
CodeGen
for Go
Composite
main.go
Creek
Framework
Build
Exe
Flink

6. Architecture - inside a job
Src1
Src2
Join Window Sink
Checkpoint
coordinator
Storage
Routine Channel Read write

7. Architecture - UDF
Sql job Parser
Validate
Optimize
CodeGen
for Go
Composite
main.go
Creek
Framework
Build
Exe
CatalogUdf.go
Parse &
validate
Gen java
udf

8. Summary - Functionality
• Flink compatible stream SQL API
• Golang UDF (scalar, aggregate and table function)
• Connectors: Kafka, File, MQTT, Stdin/Stdout
• State Backend: FileSystem

9. Summary - Performance
• Memory usage: <10MB (exclude state)
• Throughput: 410k msg/s (Raspberry Pi 4)
• Job file size: <15MB (~4MB after gzip)
• Zero dependency (download and run)

10. In the future - parallelism
Op1 Op2 Op3
Op1_0 Op2_0 Op3_0
Op1_1 Op2_1 Op3_1
Now: no parallelism Future: multiple parallelism
Support parallelism to fully utilize computing power

11. Q&A
Try creek job generator at http://creek.baidubce.com (Chinese UI)
Contact: wdyuanyoujun@163.com