Robust Stream Processing with Apache Flink

•Télécharger en tant que PPTX, PDF•

0 j'aime•907 vues

Slides from my OSCON 2016 talk. Not super useful without the accompanying video and code. I'll post that when it's available.

Logiciels

Robust
Stream Processing
with
Apache Flink
Jamie Grier
@jamiegrier
jamie@data-artisans.com

Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

Overview
• What is Apache Flink?
• What is Stateful Stream Processing?
• Windowed computation over streams
• Robust Time Handling (Event Time vs Processing Time)
• Robust Failure Handling
• Robust Planned Downtime Handling
• Robust Reprocessing

What is
Apache Flink?
s an open source platform for distributed stream and batch da

Stream Processing
Your
Code
Data Stream Data Stream

Stateful
Stream Processing
Your
Code
Data Stream Data Stream
State

More Complex
Example
Kafka
Files
Rabb
itMQ
Filter
Map
Join /
Sum
Influx
DB
C*

Distributed and Parallel
Deployment
Kafka
Files
Rabb
it
MQ
Filter
Pars
e
Join /
Sum
Influx
DB
C*

Robust Stream Processing
with Apache Flink

Windowing in Processing
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time

Windowing in Event
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time

We’re Hiring!
http://data-artisans.com/careers

Flink Forward 2016, Berlin
Submission deadline: June 30, 2016
Early bird deadline: July 15, 2016
www.flink-forward.org

Recommandé

Counting Elements in StreamsJamie Grier

Robust Stream Processing With Apache FlinkJamie Grier

Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...Flink Forward

Flink 1.0-slidesJamie Grier

Apache Flink Community Updates November 2016 @ Berlin MeetupRobert Metzger

Extending the Yahoo Streaming BenchmarkJamie Grier

Apache Flink @ Alibaba - Seattle Apache Flink MeetupBowen Li

Aljoscha Krettek - The Future of Apache FlinkFlink Forward

Recommandé

Counting Elements in StreamsJamie Grier

Robust Stream Processing With Apache FlinkJamie Grier

Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...Flink Forward

Flink 1.0-slidesJamie Grier

Apache Flink Community Updates November 2016 @ Berlin MeetupRobert Metzger

Extending the Yahoo Streaming BenchmarkJamie Grier

Apache Flink @ Alibaba - Seattle Apache Flink MeetupBowen Li

Aljoscha Krettek - The Future of Apache FlinkFlink Forward

data Artisans Product AnnouncementFlink Forward

Big Data WarsawMaximilian Michels

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...confluent

Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...Flink Forward

Stream Processing with Apache FlinkC4Media

Flink. Pure StreamingIndizen Technologies

Stream processing with Apache Flink @ OfferUpBowen Li

A look at Flink 1.2Stefan Richter

Portable Streaming Pipelines with Apache Beamconfluent

Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming APIFlink Forward

Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4Flink Forward

Deploying Confluent Platform for Productionconfluent

Tips & Tricks for Apache Kafka®confluent

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...Flink Forward

Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...Flink Forward

Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...Flink Forward

Berlin Apache Flink Meetup May 2015, Community UpdateRobert Metzger

Using Spark at VungleVungle

RedisConf17 - Pain-free PipeliningRedis Labs

Introduction to Apache Flinkdatamantra

Jamie Grier - Robust Stream Processing with Apache FlinkFlink Forward

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Contenu connexe

Tendances

data Artisans Product AnnouncementFlink Forward

Big Data WarsawMaximilian Michels

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...confluent

Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...Flink Forward

Stream Processing with Apache FlinkC4Media

Flink. Pure StreamingIndizen Technologies

Stream processing with Apache Flink @ OfferUpBowen Li

A look at Flink 1.2Stefan Richter

Portable Streaming Pipelines with Apache Beamconfluent

Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming APIFlink Forward

Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4Flink Forward

Deploying Confluent Platform for Productionconfluent

Tips & Tricks for Apache Kafka®confluent

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...Flink Forward

Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...Flink Forward

Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...Flink Forward

Berlin Apache Flink Meetup May 2015, Community UpdateRobert Metzger

Using Spark at VungleVungle

RedisConf17 - Pain-free PipeliningRedis Labs

Introduction to Apache Flinkdatamantra

Tendances (20)

data Artisans Product Announcement

Big Data Warsaw

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...

Stream Processing with Apache Flink

Flink. Pure Streaming

Stream processing with Apache Flink @ OfferUp

A look at Flink 1.2

Portable Streaming Pipelines with Apache Beam

Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API

Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4

Deploying Confluent Platform for Production

Tips & Tricks for Apache Kafka®

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...

Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...

Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...

Berlin Apache Flink Meetup May 2015, Community Update

Using Spark at Vungle

RedisConf17 - Pain-free Pipelining

Introduction to Apache Flink

Similaire à Robust Stream Processing with Apache Flink

Jamie Grier - Robust Stream Processing with Apache FlinkFlink Forward

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Robust stream processing with Apache FlinkAljoscha Krettek

Big Data Analytics Platforms by KTH and RISE SICSBig Data Value Association

A Practical Guide to Selecting a Stream Processing Technology confluent

Flink in actionArtem Semenenko

Streaming in the Wild with Apache FlinkKostas Tzoumas

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

Near real-time anomaly detection at Lyftmarkgrover

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...ScyllaDB

Stateful Stream Processing at In-Memory SpeedJamie Grier

Cloud lunch and learn real-time streaming in azureTimothy Spann

Streaming in the Wild with Apache FlinkDataWorks Summit/Hadoop Summit

Hail hydrate! from stream to lake using open sourceTimothy Spann

Stinger.Next by Alan Gates of HortonworksData Con LA

Data Stream Processing with Apache FlinkFabian Hueske

Flink forward-2017-netflix keystones-paasMonal Daxini

The Purpose of Cloud-technology for Video ProductionMaarten Verwaest

Reintroducing the Stream Processor: A universal tool for continuous data anal...Paris Carbone

How Tencent Applies Apache Pulsar to Apache InLong —— A Streaming Data Integr...StreamNative

Similaire à Robust Stream Processing with Apache Flink (20)

Jamie Grier - Robust Stream Processing with Apache Flink

Trend Micro Big Data Platform and Apache Bigtop

Robust stream processing with Apache Flink

Big Data Analytics Platforms by KTH and RISE SICS

A Practical Guide to Selecting a Stream Processing Technology

Flink in action

Streaming in the Wild with Apache Flink

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

Near real-time anomaly detection at Lyft

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...

Stateful Stream Processing at In-Memory Speed

Cloud lunch and learn real-time streaming in azure

Streaming in the Wild with Apache Flink

Hail hydrate! from stream to lake using open source

Stinger.Next by Alan Gates of Hortonworks

Data Stream Processing with Apache Flink

Flink forward-2017-netflix keystones-paas

The Purpose of Cloud-technology for Video Production

Reintroducing the Stream Processor: A universal tool for continuous data anal...

How Tencent Applies Apache Pulsar to Apache InLong —— A Streaming Data Integr...

Dernier

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent

Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol

What is Advanced Excel and what are some best practices for designing and cre...Technogeeks

Unveiling Design Patterns: A Visual Guide with UML DiagramsAhmed Mohamed

Powering Real-Time Decisions with Continuous Data StreamsSafe Software

Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz

Software Coding for software engineeringssuserb3a23b

Automate your Kamailio Test Calls - Kamailio World 2024Andreas Granig

React Server Component in Next.js by Hanief UtamaHanief Utama

Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...OnePlan Solutions

How to submit a standout Adobe Champion ApplicationBradBedford3

Ahmed Motair CV April 2024 (Senior SW Developer)Ahmed Mater

SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa

Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfkalichargn70th171

Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions

Implementing Zero Trust strategy with AzureDinusha Kumarasiri

GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfAlina Yurenko

A healthy diet for your Java application Devoxx France.pdfMarharyta Nedzelska

Advantages of Odoo ERP 17 for Your BusinessEnvertis Software Solutions

Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López

Dernier (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha

What is Advanced Excel and what are some best practices for designing and cre...

Unveiling Design Patterns: A Visual Guide with UML Diagrams

Powering Real-Time Decisions with Continuous Data Streams

Folding Cheat Sheet #4 - fourth in a series

Software Coding for software engineering

Automate your Kamailio Test Calls - Kamailio World 2024

React Server Component in Next.js by Hanief Utama

Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...

How to submit a standout Adobe Champion Application

Ahmed Motair CV April 2024 (Senior SW Developer)

SpotFlow: Tracking Method Calls and States at Runtime

Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf

Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...

Implementing Zero Trust strategy with Azure

GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf

A healthy diet for your Java application Devoxx France.pdf

Advantages of Odoo ERP 17 for Your Business

Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...

Robust Stream Processing with Apache Flink

1. Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier jamie@data-artisans.com

2. Who am I? • Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing

3. Overview • What is Apache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing

4. What is Apache Flink? s an open source platform for distributed stream and batch da

5. What is Apache Flink?

6. Stream Processing Your Code Data Stream Data Stream

7. Stateful Stream Processing Your Code Data Stream Data Stream State

8. More Complex Example Kafka Files Rabb itMQ Filter Map Join / Sum Influx DB C*

9. Distributed and Parallel Deployment Kafka Files Rabb it MQ Filter Pars e Join / Sum Influx DB C*

10. Robust Stream Processing with Apache Flink

11. Code Example!

12. Windowing

13. Processing Time vs Event Time

14. Windowing in Processing Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time

15. Windowing in Event Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time