Kafka Summit 2021 - Apache Kafka meets workflow engines
•
0 j'aime•501 vues
Slides from my talk at Kafka Summit Americas 2021 on September 15th
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Kafka Summit 2021 - Apache Kafka meets workflow engines
Similaire à Kafka Summit 2021 - Apache Kafka meets workflow engines (20)
Plus de Bernd Ruecker
Plus de Bernd Ruecker (18)
Dernier
Dernier (20)
Kafka Summit 2021 - Apache Kafka meets workflow engines
- 1. Apache Kafka Meets Workflow Engines Bernd Ruecker Co-Founder and Chief Technologist @Camunda @berndruecker
- 3. There are pretty amazing use cases @berndruecker Order Management in Telco 5,000 process instances/s Money Transfer 200 process instances/s Perseverance Mars Rover Nasa
- 6. Scale Onboarding you might think of Real-life onboarding use case in banking Money transfer or trading Process Instances per second @berndruecker
- 7. Making phone calls Scale Telco Onboarding Process Instances per second @berndruecker
- 8. A process Steps In a certain sequence Which might be long running @berndruecker Time-controlled events
- 9. A workflow engine Or more scalable: @berndruecker
- 10. A workflow engine Or more scalable: @berndruecker Camunda Platform Camunda Cloud Two workflow engines: - RDBMS / traditional & on-prem - Event-Sourcing / cloud-native
- 11. https://blog.bernd-ruecker.com/zeebe- io-a-horizontally-scalable-distributed- workflow-engine-45788a90d549 “Zeebe” is the workflow engine offered within Camunda Cloud
- 12. Workflow engines at scale https://camunda.com/blog/2019/08/zeebe-horizontal-scalability/ @berndruecker
- 14. Technical Example Java Spring Boot Application Camunda Cloud Confluent Cloud Kafka Topic Zeebe Worker Via subscription Publish record Kafka Listener Via subscription Correlate to process instance https://github.com/berndruecker/kafka-camunda-spring-simple @berndruecker
- 15. Kafka Connect
- 21. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 22. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 23. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 24. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment @berndruecker
- 25. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
- 26. What we wanted Photo by Lijian Zhang, under Creative Commons SA 2.0 License and Wikimedia Commons / CC BY-SA 4.0 @berndruecker Choreography
- 27. Why is it so tempting? Service A Event Bus A B Service B C Service C D E F Service D G Service … Service … Service … Service … Service … Service … Service … @berndruecker
- 28. Why is it so tempting? Service A Event Bus A B Service B C Service C D E F Service D G Service … Service … Service … Service … Service … Service … Service … Adding is easy! You can „buy“ a shorter initial time-to-value by choreography. It yields in technical debt. @berndruecker
- 30. Order Fulfillment Decide about responsibility Checkout Payment Inventory Shipment Payment received Order placed Retrieve payment @berndruecker This is choreography This is orchestration
- 31. Order Fulfillment Decide about responsibility Checkout Payment Inventory Shipment Order placed Retrieve payment It can still be Apache Kafka! @berndruecker
- 32. ? Event Command Message Record Event Fact, happened in the past, immutable Intend, Want s.th. to happen, The intention itself is a fact @berndruecker
- 33. ? Event Command Message Record Event @berndruecker
- 34. Commands in disguise The Customer Needs To Be Sent A Message To Confirm Address Change Event Send Message Wording of Sender Wording of recipient
- 35. Checkout Order Payment Event-driven: Decision to couple is on the receiving side Command-driven Decision to couple is on the sending side Direction of dependency Retrieve Payment Order placed Payment received Direction of dependency @berndruecker
- 36. Order Checkout Payment Inventory Shipment Stateful orchestration This orchestration requires state @berndruecker
- 38. Now it is easy to change the process flow @berndruecker
- 40. Customer created Sam Newman: Building Microservices @berndruecker
- 41. Mix orchestration and choreography Orchestration Orchestration Orchestration Choreography @berndruecker
- 42. Another use case: Transactions Service Kafka RDMS 1. Receive 4. Send additional events 2. Business Logic 3. Send response @berndruecker
- 43. Another use case example: Vehicle Maintenance
- 44. Action Insights Measurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Call driver to stop and inspect … @berndruecker
- 45. Action Insights Measurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Workflows Event Streams @berndruecker Custom Connector
- 48. Summary • A workflow engine is complementary to Apache Kafka • The technical integration is easy • There are tools that scale • Typical use cases • Microservices orchestration (balanced with choreography) • Business transactions • Actions resulting from streams @berndruecker
- 50. Want to learn more? https://ProcessAutomationBook.com/ Free electronic version available Featuring code examples:
- 52. Berlin, Germany http://berndruecker.io/ mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda