Real-World Pulsar Architectural Patterns
•Télécharger en tant que PPTX, PDF•
6 j'aime•696 vues
This presentation covers Real-World Pulsar Architectural Patterns involving Distributed Caching and Distributed Tracing. We also cover the use of Apache Ignite, Jaeger, Apache Flink, and many other technologies, as well as industry best-practices.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Real-World Pulsar Architectural Patterns
Similaire à Real-World Pulsar Architectural Patterns (20)
Plus de Devin Bost
Plus de Devin Bost (6)
Dernier
Dernier (20)
Real-World Pulsar Architectural Patterns
- 1. Real-World Pulsar Architectural Patterns Every pattern shown here has been developed and implemented with my team at Overstock Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/ By Devin Bost, Senior Data Engineer at Overstock Distributed Caching + Distributed Tracing +
- 2. Missing Stephen Bourke in Ireland and Sam Lowen, our new product manager.
- 3. Disclaimer: I’m assuming you already know that Apache Pulsar is the future of real-time messaging
- 4. = Pulsar topic = Kafka topic = Apache Bookkeeper Tech Legend
- 5. Starting simple with Pub/Sub
- 6. Starting simple with Pub/Sub
- 7. Starting simple with Pub/Sub
- 8. Starting simple with Pub/Sub
- 9. Producer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Passthrough function /ingest /feeds
- 10. Diagram courtesy of Thor Sigurjonsson
- 11. Producer Producer Producer Consumer Consumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Passthrough function Higher-availability: /ingest /feeds
- 12. Producer Producer Producer Consumer Consumer Consumer Passthrough function Is this a safe approach to caching? /ingest /feeds
- 13. Producer Producer Producer Consumer Consumer Consumer Passthrough function What happens if Redis goes down? X /ingest /feeds
- 14. Producer Producer Producer Consumer Consumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite /ingest /feeds
- 15. Producer Producer Producer Consumer Consumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence /ingest /feeds
- 16. Producer Producer Producer Consumer Consumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis /ingest /feeds
- 17. Producer Producer Producer Consumer Consumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis Supports tables with backing cache /ingest /feeds
- 18. Producer Producer Producer Consumer Consumer Consumer Passthrough function It’s much safer to use a distributed cache technology like Ignite Smart Persistence Faster than Redis Supports tables with backing cache Supports transactions /ingest /feeds
- 19. What if you have a business- critical service that can’t lose messages?
- 20. “But, doesn’t Pulsar already guarantee that messages won’t be lost?”
- 21. There’s a difference between Pulsar losing messages and your application losing messages!
- 22. So, we introduce a backfill path.
- 23. Notice I avoided putting a SQL database on the list Replication Persistent storage Producer Producer Producer Consumer Consumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Passthrough function Batch job Backfill Topic Consumer Consumer Consumer Alerts to end users (e.g. Email, SMS, Twilio call, etc.) OR Etc Message delivered yet? Message delivered yet? Kappa + Lambda architectures /ingest /feeds
- 24. Can we simplify it?
- 25. Replication Persistent storage Pulsar Function Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Alerts to end users (e.g. Email, SMS, Twilio call, etc.) Batch job OR Etc Producer Producer Producer Passthrough function Backfill Topic Pulsar Function Message delivered yet? Message delivered yet? You could add another passthrough function and topic if you want more isolation. /ingest /feeds
- 26. How about just for ingesting data into a cache with a backfill?
- 27. Option 1: Web Service Passthrough Function Persistent Append Only Storage Cache Sink Batch Engine (e.g. Spark, NiFi, etc.) Read All Data Loads into existing topic OR Starts Job Etc Replication /ingest /feeds
- 28. Option 2: Achieves separation of concerns and prevents QoS problems with live traffic when running a backfill Web Service Passthrough Function Cache Sink Batch Engine (e.g. Spark, NiFi, etc.) Loads into backfill topic OR Starts Job Cache Sink Persistent Append Only Storage Read All Data Etc Replication /ingest /feeds
- 29. Topic with Retention Function Cache Sink Function (stopped until needing to backfill) Exclusive Mode (Subscription stores in Bookkeeper automatically.) Tiered Storage in S3 or Google Cloud Backfill Cache Sink Starts Function Passthrough FunctionOption 3: Note: You need to ensure the Bookkeeper cluster is fast enough to keep up with the brokers or your brokers’ memory will fill up Also, this approach will only give you a single backfill run unless you have additional replication. /feeds
- 30. What about adding caching to a legacy application?
- 31. Legacy SQL DB Website Enrich from cached data Extract Relevant Data Filter to desired clicks (Raw Clicks) Store in cache Web Application Emitting specific events (Omitting passthrough details for simplicity.)
- 32. Legacy SQL DB Website Enrich from cached data Extract Relevant Data Filter to desired clicks Store in cache Web Application Emitting specific events You can also emit directly to Pulsar as a producer. It’s simpler if you have the ability to touch the website code.
- 33. Legacy SQL DB Website Enrich from cached data Extract Relevant Data Filter to desired clicks Store in cache Web Application Emitting specific events You can also emit directly to Pulsar as a producer. It’s simpler if you have the ability to touch the website code. However, the raw clicks flow still gets messy.
- 34. Legacy SQL DB Website Enrich from cached data Store in cache Web Application Emitting specific events Event A Event B Event C Cleaner & better separation of concerns to have purposeful topics... Easier to debug & maintain.
- 35. What if you’re using a graph engine for more complex query logic but need that data in real-time?
- 36. If you don’t make it synchronous, you will get race conditions when updating and querying the graph! Web Application Emitting specific deltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function
- 37. What if you need a more robust verification that deltas are in order and aren’t duplicates? (e.g. financially impacting increment/decrement values or state variables)
- 38. Usually, it’s best to separate concerns into separate functions, like this. Web Application Emitting specific deltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function Gate Keeper Filter Check if message has been seen already Duplicate or late? Drop messageYes No But, is that the right approach here?
- 39. What happens when you turn up parallelism on the functions?
- 40. Web Application Emitting specific deltas (e.g. state change, increment, etc.) (2) Return on completion (1) Write Change, Wait for Success Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter Gate Keeper Filter
- 41. In this case, the right approach is to consolidate your logic to leverage the transactional guarantees of your database. Web Application (2) Return on completion (1) Check if duplicate or outdated. If not, write Change, Wait for Success Get full record Complex graph query Synchronous Update Function
- 42. Leverage transactional guarantees of your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function
- 43. Leverage transactional guarantees of your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Always be mindful of how the behavior might change when function parallelism is turned up!
- 44. Leverage transactional guarantees of your database! (Your function will need to retry if a transaction fails.) Web Application Get full record Complex graph query Synchronous Update Function Synchronous Update Function Synchronous Update Function Synchronous Update Function Always be mindful of how the behavior might change when function parallelism is turned up! If making state changes, be sure that you get timestamps on your upstream data contract so you can verify if the messages are in order!
- 45. Now, what happens when you need to debug a large or complex function flow?
- 46. FunctionFunction Function Web Application Function Web Service Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function
- 47. FunctionFunction Function Web Application Function Web Service Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination?
- 48. FunctionFunction Function Web Application Function Web Service Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing?
- 49. FunctionFunction Function Web Application Function Web Service Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing? What if we can’t modify the function code (since it’s a multi-tenant application)?
- 50. FunctionFunction Function Web Application Function Web Service Web Application Function Function Web Service Web Application FunctionFunction Function Function Function Function What happens if some messages seem to not be reaching their destination? What happens if a message isn’t getting transformed correctly at some point or null values are appearing? What if we can’t modify the data contracts either for the same reason? What if we can’t modify the function code (since it’s a multi-tenant application)?
- 51. Function Function Function Function Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 52. Function Function Function Function Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 53. Function Function Function Function Span 1 Span 2 Span 3 Span 4 Span 1 Trace
- 54. Jaeger is based on the OpenTracing standard Check out the book, “Mastering Distributed Tracing” (2019) by Yuri Shkuro Trace Span Span Span Span Span Span Span Span Span Span SpanSpan Tags (Key/Value) Logs (Key/Value + Timestamp) Tags (Key/Value) Logs (Key/Value + Timestamp) Tags (Key/Value) Logs (Key/Value + Timestamp) . . .Span Span Tags (Key/Value) Logs (Key/Value + Timestamp)
- 57. 4 5 7 3 Count By Key e.g. Over a session-based window
- 58. + + + OR
- 59. Metadata parameters for the TapFunction’s envelope are specified in its config Message1 Message2 Message3 Message1’ Message2’ Message3’ Message1’’ Message2’’ Message3’’ . . .Function1 Function1 JoinerFunction Message2 Message2’ Message2’’ CorrelationId=productId-784 Message1 Message1’ Message1’’ CorrelationId=productId-142 Message3 Message3’ Message3’’CorrelationId=productId-923 Jaeger Sink Taps wrap message with header containing: CorrelationId, Tenant, Namespace, Name, timestamp, etc. TapFunction TapFunction TapFunction SamplerFunction CorrelationId=productId-784 Message1 Message1’ Message1’’ StartTimestamp EndTimestamp Span CorrelationId=productId-784 StartTimestamp EndTimestamp Span Message1 StartTimestamp EndTimestamp Span Message1’ StartTimestamp EndTimestamp Span Message1’’Message1’ Message1’’ Message1’’’
- 60. CorrelationId is derived and put into the envelope produced by the TapFunction. { "correlationKey": "productId", "correlationValue": "20603199", "correlationId": "productId-20603199”, . . . } The TapFunction defines the correlationKey in its Pulsar Config. You can tap ANY topic! Message1 Message2 Message3 Message1’ Message2’ Message3’ Message1’’ Message2’’ Message3’’ . . .Function1 Function1 Taps wrap message with header containing: CorrelationId, Tenant, Namespace, Name, timestamp, etc. TapFunction TapFunction TapFunction
- 61. Uses Flink’s stateful join capability. It all happens in a keyed stream! JoinerFunction Message2 Message2’ Message2’’ CorrelationId=productId-784 Message1 Message1’ Message1’’ CorrelationId=productId-142 Message3 Message3’ Message3’’CorrelationId=productId-923
- 62. Allows us to specify rate to limit how many messages are sampled. For dev, this is set to 100% to allow all. This is just a simple Pulsar filter function. SamplerFunction CorrelationId=productId-784 Message1 Message1’ Message1’’
- 63. These Spans emit to Jaeger and can be stored in a Cassandra or Elasticsearch backend for production. OR Jaeger Sink StartTimestamp EndTimestamp Span CorrelationId=productId-784 StartTimestamp EndTimestamp Span Message1 StartTimestamp EndTimestamp Span Message1’ StartTimestamp EndTimestamp Span Message1’’Message1’ Message1’’ Message1’’’
- 65. If fields are omitted from the tap’s config, we capture all that we can.
- 66. Another trick is to provide alternative representations of a value to make search/analytics easier downstream.
- 67. Questions?
- 68. Real-World Pulsar Architectural Patterns Every pattern shown here has been developed and implemented with my team at Overstock Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/ By Devin Bost, Senior Data Engineer at Overstock Distributed Caching + Distributed Tracing