Two of the main software architectural trends in software development this decade has been the move to streaming data processing, and the move to microservice architecture. Both of these architectures are driven by the needs of managing and mining knowledge from ever increasing volumes of data in a close to real-time fashion—all while being reactive: responsive under failure, and responsive under load. I'm here to tell you that these two trends are converging, and a fusion of the two is both logical and inevitable. In this session we will talk about what a fused approach to stream processing and microservices could look like, what opportunities exist—what software development for business software can look like in the following decade.

1. The convergence of stream processing
& microservice architecture
√Deputy CTO, Lightbend
@viktorklang / v@lightbend.com / viktorklang.com

3. Overview

4. History of Data Processing
4 Offline Batch processing
4 MapReduce
4 Lambda Architecture

5. History of Services
4 Client-Server
4 CORBA & DCOM
4 EJB & N-Tier
4 SOA & ESB
4 HTTP-API & REST

6. Today

7. Catch of the day
Live1
Stream Processing
&
Microservice Architecture
1
Near real-time

8. Shared requirements:
4 Available
4 Elastically scalable
4 Resilient & self-healing
4 Loosely coupled & upgradeable

9. Reactive Systems2
Available (responsive), under failure (resilience) and
under load (elasticity), by being message-driven.
2
https://www.reactivemanifesto.org/

10. Reactive Streams3
Provides a standard for asynchronous stream
processing with non-blocking back pressure.
3
www.reactive-streams.com & java.util.concurrent.Flow (JDK9+)

11. Ingredients
4 Stream processing stages: Stateless & Stateful
4 Microservices: RPC-style & Evented
4 Domain Events with Schema/Codec

12. #define Events4
4 Immutable recordings of observations
4 Can be disregarded but not be retracted once accepted
4 Can be invalidated but (typically) not be deleted
4 Knowledge accrues with every processed event
4
Learn more: https://speakerdeck.com/jboner/designing-events-first-microservices

13. Services (evented)

14. Realizationevented services are stream processing stages

15. The near future:
Convergence

16. Is this the solution?
4 Alice: So we we lift evented services into the stream
processing pipeline …
4 Bob: … and then we're done!
4 <record scratch>
4 Narrator: They're forgetting the request-reply use-
case which RPC services excel at.

17. Services (rpc)

18. The improved solution
4 lift the service into a stream stage
4 lift input Events into Commands and submit
4 capture responses & convert into output Events

19. What if we could
derive microservices
from data streams?

20. Ingestion
endpoints

21. Simple
Projection
endpoints

22. Complex
Projection
endpoints

23. Unification
4 Microservices are a part of a streaming pipeline
4 A pipeline can now be exposed as Microservices
4 We can independently upgrade parts of the pipeline

24. Developing converged applications
4 Deployment Orchestration
4 Events-First DDD & EventStorming
4 (Meta)Data-driven Continuous Deployment
4 Use of the right tools for each component

25. Beyond

26. Extend to
the Edges

27. End of stream

28. Explore what we do at Lightbend:
4 Lightbend Fast Data Platform
4 https://www.lightbend.com/products/fast-data-
platform
4 Lightbend Reactive Platform
4 https://www.lightbend.com/products/reactive-platform
4 Lightbend Enterprise Suite
4 https://www.lightbend.com/products/enterprise-suite

29. Thank you for attending!
The convergence of stream processing
& microservice architecture
√
Deputy CTO, Lightbend
@viktorklang / v@lightbend.com / viktorklang.com