Dual write strategies for microservices
•
11 j'aime•4,288 vues
Patterns covered: Modular monolith Two-phase commit Orchestration Choreography Saga Event sourcing Outbox pattern Parallel pipelines Listen to Yourself Slides from Red Hat Summit 2021 talk https://events.summit.redhat.com/widget/redhat/sum21/sessioncatalog/session/1607126048915001oL4X
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Dual write strategies for microservices
Similaire à Dual write strategies for microservices (20)
Plus de Bilgin Ibryam
Plus de Bilgin Ibryam (11)
Dernier
Dernier (20)
Dual write strategies for microservices
- 1. Dual Write Strategies for Microservices Bilgin Ibryam @bibryam Product Manager Red Hat
- 2. About me 2 ▸ Product Manager at Red Hat ▸ Former Principal Architect ▸ Committer at Apache Camel ▸ Author ･ Camel Design Patterns ･ Kubernetes Patterns ▸ @ bibryam ▸ https://www.ofbizian.com
- 3. There is a catch 3 Challenges with legacy applications ▸ Frequent deployment is difficult ▸ Obstacles to scaling development ▸ Scaling the application can be difficult Modern application benefits ▸ Greater team autonomy ▸ Reliability, scalability, and other abilities ▸ Reduce time to market
- 4. The dual-write problem 4 Dual-write challenge behind the scene ▸ Updating more than one resources ▸ Reliable inter-service communication ▸ Coordinating long-running, business transactions ▸ Recovery from failed distributed transaction ▸ Implementing idempotent operations ▸ Scaling with increasing data volume
- 7. Modular monolith 7 Challenges ▸ Strong data consistency requirements ▸ Strict performance requirements ▸ Complex business transactions Solution ▸ Single process deployment unit ▸ Independent modules with clear responsibilities and well-defined interfaces/contracts ▸ Each module access only its tables
- 8. Implementation options 8 Diagram credits @axelfontaine
- 9. Modular monolith 9 Benefits ▸ Simple transaction semantics with local transactions ▸ ACID properties Drawbacks ▸ Shared runtime hinders independent deployment, scalability, and failure isolation ▸ Shared database can lead to coupling among modules
- 10. Two-phase commit
- 11. Two-phase commit 11 Challenges ▸ Strong data consistency requirements ▸ Heterogeneous data sources ▸ Integrating with a third-party or legacy system ▸ Exactly-once message processing Solution ▸ 2PC protocol and X/Open XA specification ▸ JTA and WS AtomicTransaction implementations
- 12. Implementation options 12 Distributed transaction managers ▸ Narayana, JOTM, BTM, Atomikos, MSDTC Datasources ▸ PostgreSQL, MySQL, Db2, Oracle, SQL Server ▸ ActiveMQ, HornetQ, MSMQ, IBM MQ, Solace ▸ Infinispan, Hazelcast Non 2PC/XA distributed transactions ▸ eBay’s GRIT protocol
- 13. Two-phase commit 13 Benefits ▸ An out-of-the-box, standard-based solution ▸ Implemented by many traditional datasources ▸ ACID properties Drawbacks ▸ A blocking protocol with database locks and performance penalty ▸ All participating services must be available ▸ Manual recovery from controller crashes
- 16. Challenges ▸ Business process spanning several services ▸ Business functions lasting for hours or days ▸ A single location for management and monitoring of distributed transactions Solution ▸ Implement each business transaction that spans multiple services as a sequence of local transactions Saga 16
- 17. Saga implementation options 17 Coordination approaches ▸ Orchestration - a controller tells the participants what local transactions to execute ▸ Choreography - business transaction coordination logic is spread among all participants Communication mechanisms ▸ Synchronous, for example HTTP, gRPC ▸ Asynchronous, for example Apache ActiveMQ, Apache Kafka Orchestration Choreography
- 18. Orchestration based Saga 18 Benefits ▸ Doesn’t require all participants to be available at the same time, nor to have knowledge about each other ▸ Single place to define and monitor the transaction flow Drawbacks ▸ Complex programing model, requires coordination, compensation logic, and idempotency implementations ▸ Lacks Isolation from AC*D properties which can cause dirty reads, lost updates, non-repeatable reads
- 19. Choreography
- 20. Choreography 20 Challenges ▸ Business process spanning several services ▸ Business functions lasting for hours or days ▸ Highly scalable and available system Solution ▸ Implement each business transaction that spans multiple services as a sequence of local transactions with distributing decision making in each service
- 21. Choreography 21 Benefits ▸ Doesn't require a transaction coordinator service ▸ Better performance compared to orchestration approach due to smaller number of interactions Drawbacks ▸ No single place to define and monitor the business transaction flow ▸ Coupling among participant with point to point interactions ▸ Lacks Isolation from AC*D properties
- 22. Implementation options 22 Publish, then local-commit Local-commit, then publish Event sourcing Outbox pattern Two-phase commit
- 23. Outbox Pattern 23 Offers an approach for services to update their data store and notify other services in a reliable and eventually consistent manner. Benefits ▸ Addresses the dual-write problem ▸ Offers “read your own writes" semantics Drawbacks ▸ Requires specialized tools, such as Debezium
- 25. Parallel pipelines 25 Challenges ▸ Avoid dual writes to a local database and a messaging system Solution ▸ Publish a message to other services and yourself in the same messaging system
- 26. Listen to Yourself 26 Benefits ▸ Simple, scalable architecture with parallel processing capabilities Drawbacks ▸ Requires temporal dismantling, not commonly applicable ▸ Hard to reason about the global system state ▸ Lacks “read your own writes" semantics
- 27. Summary
- 29. Dual write strategies for microservices 29 Modular Monolith Two-phase Commit Orchestration Choreography Parallel Pipeline Service runtime Single process Single/Multiple Multiple Multiple Multiple Datasources Single Heterogeneous (requires XA) Heterogeneous Heterogeneous Heterogeneous Point of control Centralized Centralized Centralized Distributed Distributed Steps/Flow execution At once At once (for happy paths), highly coupled Sequential, temporal decoupling Sequential, temporal decoupling Parallel, temporal decoupling Properties ACID, blocking, synchronous ACID, blocking, synchronous ACD, non-blocking, (a)synchronous, ACD, non-blocking, asynchronous, ACD, non-blocking, asynchronous, Examples Local transactions XA, JTA, WS-AT Saga/Outbox, Debezium, Kafka Saga/Outbox, Debezium, Kafka Listen to yourself pattern
- 30. 30 Red Hat OpenShift Streams for Apache Kafka a fully managed Apache Kafka service by Red Hat http://red.ht/TryKafka Try Apache Kafka in seconds