Streaming Trend Discovery: Real-Time Discovery in a Sea of Events with Scott Haines
•
3 j'aime•865 vues
Time is the one thing we can never get in front of. It is rooted in everything, and “timeliness” is now more important than ever especially as we see businesses automate more and more of their processes. This presentation will scratch the surface of streaming discovery with a deeper dive into the telecommunications space where it is normal to receive billions of events a day from globally distributed sub-systems and where key decisions “must” be automated. We’ll start out with a quick primer on telecommunications, an overview of the key components of our architecture, and make a case for the importance of “ringing”. We will then walk through a simplified solution for doing windowed histogram analysis and labeling of data in flight using Spark Structured Streaming and mapGroupsWithState. I will walk through some suggestions for scaling up to billions of events, managing memory when using the spark StateStore as well as how to avoid pitfalls with the serialized data stored there. What you’ll learn: 1. How to use the new features of Spark 2.2.0 (mapGroupsWithState / StateStore) 2. How to bucket and analyze data in the streaming world 3. How to avoid common Serialization mistakes (eg. how to upgrade application code and retain stored state) 4. More about the telecommunications space than you’ll probably want to know! 5. Learn a new approach to building applications for enterprise and production. Assumptions: 1. You know Scala – or want to know more about it. 2. You have deployed spark to production at your company or want to 3. You want to learn some neat tricks that may save you tons of time! Take Aways: 1. Fully functioning spark app – with unit tests!
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Streaming Trend Discovery: Real-Time Discovery in a Sea of Events with Scott Haines
Similaire à Streaming Trend Discovery: Real-Time Discovery in a Sea of Events with Scott Haines (20)
Plus de Databricks
Plus de Databricks (20)
Dernier
Dernier (20)
Streaming Trend Discovery: Real-Time Discovery in a Sea of Events with Scott Haines
- 1. Scott Haines, Twilio Streaming Trend Discovery: real-time discovery in a sea of events #DDSAIS16 @newfront
- 2. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Agenda • Intro and use cases for Spark at Twilio (5 min) • Elements of a Streaming Discovery Engine (5 min) • Principals of Valuable Data Sets (4 min) • Refresher on Histograms and Percentiles (2 min) • Building a Discovery Engine on Spark: Part One (15 min) • Break?: 10 min • Building a Discovery Engine on Spark: Part Two (15 min) • Lessons learned (5 min) • Questions (5 min) Project Files on Github @ https://bit.ly/2ISI3BS 2
- 3. #DDSAIS16 Github https://bit.ly/2ISI3BS Scott Haines • 6+ years working with Streaming Systems • Yahoo! Games • Recommendations, Personalization • Daily Fantasy Growth / Analytics • Mobile Campaigns, Fraud, Trends • Flurry Analytics • Statistics based notification system • Stats / Data Science Enthusiast • Drummer / Snowboarder / Hot Sauce Maker / Teacher 3 Principal Engineer, Voice Insights @ Twilio @newfront
- 4. #DDSAIS16 Github https://bit.ly/2ISI3BS Twilio We are a public, large-scale, globally available & distributed communications as-a-service company. • Founded in 2008. IPO'd in 2016 • Telecommunications is our bread and butter • Along with Video / SMS / MMS / Wireless Sim / Authy / Sync • Find out more and sign up @ twilio.com 4 Where I work twilio.com
- 5. #DDSAIS16 Github https://bit.ly/2ISI3BS Use cases for Spark @Twilio • Real-Time Dashboards and Debugging • Fraud Detection • Carrier Behavior and Network Anomaly Detection • Streaming ETL, Metrics and Aggregations • ASR and Sentiment 5
- 6. #DDSAIS16 Github https://bit.ly/2ISI3BS Voice Insights @Twilio • Processing >3 TB /d • >4 billion raw events /d • >2 billion metric & sessions aggregations /d • Powers Voice Insights / Carrier Ops / Support and more 6
- 7. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Next up • Elements of the Streaming Trend Discovery Architecture 7
- 8. #DDSAIS16 Github https://bit.ly/2ISI3BS 8 Building a Discovery Engine End to End Architecture API raw events Streaming Aggregation metric aggs S3 ElasticTrend Processing Recent Metrics Historic Metrics events Historic Statistics <50ms response time *Generalized ESD or Grubbs & Distance Function
- 9. #DDSAIS16 Github https://bit.ly/2ISI3BS Core Principals of Valuable Data Sets • Data must be Structured. No JSON! 9
- 10. #DDSAIS16 Github https://bit.ly/2ISI3BS Core Principals of Valuable Data Sets • Data must be Structured. No JSON! • JSON ~= Object 10
- 11. #DDSAIS16 Github https://bit.ly/2ISI3BS Core Principals of Valuable Data Sets • Data must be Structured. No JSON! • JSON ~= Object • Avro, Protobuf, Parquet are widely used and can support backwards compatible updates. 11
- 12. #DDSAIS16 Github https://bit.ly/2ISI3BS Core Principals Establish a Data Contract 12 Structured Data has Types and are part of the Data Contract • Properties of the Data must be documented. • Data must be Accurate and Complete* • Data must support backwards and forwards compatibility. • Data should be easily flattened.
- 13. #DDSAIS16 Github https://bit.ly/2ISI3BS Core Principals Final Thoughts 13 • Structured Data allows for compile time guarantees • Write once. Compile across languages • Non-Java Object Serialization • External Serialization simplifies deploys given you can simply use the last stored session state* * don't corrupt your stored session data
- 14. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Building our mental model • Quick Refresher on Percentiles and Histograms 14
- 15. #DDSAIS16 Github https://bit.ly/2ISI3BS • Percentiles group numeric data in a set of observations. 15 Building a Discovery Engine Primer: What is a Percentile? S = {1,2,3,4,5,6,7,8,9,10}
- 16. #DDSAIS16 Github https://bit.ly/2ISI3BS • Percentiles group numeric data in a set of observations. • Common splits are min, 25%,50% (median),75% and max 16 Building a Discovery Engine Primer: What is a Percentile? S = {1,2,3,4,5,6,7,8,9,10} min:1 p25: 3.25 median: 5.50 p75: 7.75 max: 10.0
- 17. #DDSAIS16 Github https://bit.ly/2ISI3BS • Histograms are bucketed distributions of numeric datasets 17 Building a Discovery Engine Primer: What is a Histogram?
- 18. #DDSAIS16 Github https://bit.ly/2ISI3BS • Histograms are bucketed distributions of numeric datasets • In a Normal Distribution, we talk about the Area Under the Curve 18 Building a Discovery Engine Primer: What is a Histogram?
- 19. #DDSAIS16 Github https://bit.ly/2ISI3BS • Histograms are bucketed distributions of numeric datasets • A Normal Distribution, fit into the Area Under the Curve • Histograms allow us to understand the shape of a distribution 19 Building a Discovery Engine Primer: What is a Histogram? 68-95-99.7 Rule - 68%@1sd->mean,95@2x,99.7@3x)
- 20. #DDSAIS16 Github https://bit.ly/2ISI3BS 20 t1 t2 t3 t4 Building a Discovery Engine Goal: Time Series Histogram Analysis
- 21. #DDSAIS16 Github https://bit.ly/2ISI3BS • Limitations: Not easily Aggregated or hard to extract 21 Building a Discovery Engine Deep Dive: Quantiles / Histograms in Spark
- 22. #DDSAIS16 Github https://bit.ly/2ISI3BS Define a new UDF based off of the MutableAggregationBuffer's in Spark 22 Building a Discovery Engine Deep Dive: Aggregation Extension Options
- 23. #DDSAIS16 Github https://bit.ly/2ISI3BS • Upside: Things are human readable 23 Building a Discovery Engine Deep Dive: Aggregation Extension Options
- 24. #DDSAIS16 Github https://bit.ly/2ISI3BS • Upside: Things are human readable • Downside: Performance* / Control* 24 Building a Discovery Engine Deep Dive: Aggregation Extension Options
- 25. #DDSAIS16 Github https://bit.ly/2ISI3BS • Ditch the DataFrame's unstructured API and move over to Datasets typed API. • This enables the use of flatMapGroupsWithState • Now with a little effort you have full control over the aggregation strategy 25 Building a Discovery Engine Deep Dive: Aggregation Extension Options : Final Thoughts
- 26. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Next up: after the break Building a Discovery Engine: Part 2 1. Define the Problem (1m) 2. Build a plan of Attack (1m) 3. Model the Data (2m) 5. Build the Streaming Discovery Engine (~20m) 6. Hopefully time for lessons learned and Questions! 26
- 27. #DDSAIS16 Github https://bit.ly/2ISI3BS Global Telephony is tough. 27
- 28. #DDSAIS16 Github https://bit.ly/2ISI3BS Simply Connecting two endpoints is ripe for failure. 28
- 29. #DDSAIS16 Github https://bit.ly/2ISI3BS 29 X Given Networks can go down at anytime.
- 30. #DDSAIS16 Github https://bit.ly/2ISI3BS 30 We should build a system to auto-discover trends in (~real-time) for our KPMs and automatically report them...
- 31. #DDSAIS16 Github https://bit.ly/2ISI3BS 31 Building a Discovery Engine End to End Architecture API raw events Streaming Aggregation metric aggs S3 ElasticTrend Processing Recent Metrics Historic Metrics events Historic Statistics <50ms response time *Generalized ESD or Grubbs & Distance Function
- 32. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 32 Define the raw event • Must provide the information necessary for aggregation
- 33. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 33 Define the raw event • Must provide the information necessary for aggregation • id must be idempotent. Brownie points if it can be reused in Hash Partitioner
- 34. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 34 Define the raw event • Must provide the information necessary for aggregation • id must be idempotent. *dedup key - Brownie points if it can be reused in Hash Partitioner in spark and kafka! • event_dimensions + event_group can create great grouping hash
- 35. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 35 Define the raw event • Allows for event evolution while supporting backwards compatibility.
- 36. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 36 Define the raw event • Allows for event evolution while supporting backwards compatibility. • Enum ordinals can be used in ML jobs without the need for string encoding
- 37. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 37 Define the aggregation • Includes logical time series Window • Stores all Dimensional and Statistic Information
- 38. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 38 Define the aggregation • Includes logical time series Window • Stores all Dimensional and Statistic Information • Can be used together as a strong Binary Group Key
- 39. #DDSAIS16 Github https://bit.ly/2ISI3BS Building a Discovery Engine 39 Why hash the dimensions? • Hashing the dimensions of the aggregation simplifies the groupBy or groupByKey mechanics. • Use guava since it ships with Spark and works well at scale for Hashing.
- 40. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Next up • Putting together what we've learned with Spark • Available in the content @ https://bit.ly/2ISI3BS • Walk through of critical components of the Application 40
- 41. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 41 Building a Discovery Engine Creates the pipeline * process(KDSR), inject SparkSession (implicit*) * write back to Kafka * checkpoint, etc
- 42. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 42 Building a Discovery Engine 1. Transform (protobuf - > Product) 2. Actual Aggregation * groupedBy BinaryType * fMGwS keyed by BinaryType * EventTimeTimeout*
- 43. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 43 Building a Discovery Engine • StateSpec takes a StateFunc • This is for Key/Value pair grouped data • Key is [A]. It can be Binary. Cool • Take an Iterator[B] • Stores a GroupState[C] • Emits Iterator[D]
- 44. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 44 Building a Discovery Engine metricAggregation
- 45. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 45 Building a Discovery Engine • Data Sketches from Yahoo* provides serializable Quantile Sketches with Histogram capabilities. • Data Sketches are a light weight representation of a complete set of data (on a diet - think HyperLogLog) • We use this on our team to enable performant aggregation of complicated statistics without storing all rows in Spark memory.
- 46. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 46 Building a Discovery Engine • Data Sketches from Yahoo* provides serializable Quantile Sketches with Histogram capabilities. • Data Sketches are a light weight representation of a complete set of data (on a diet - think HyperLogLog) • We use this on our team to enable performant aggregation of complicated statistics without storing all rows in Spark memory.
- 47. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 47 Building a Discovery Engine • Testing should feel good. • Ensure you solve Encoder issues early • Make sure you don't add regression's into your 24/7/365 apps!
- 48. #DDSAIS16 Github https://bit.ly/2ISI3BS 48 Building a Discovery Engine End to End Architecture API raw events Streaming Aggregation metric aggs S3 ElasticTrend Processing Recent Metrics Historic Metrics events Historic Statistics <50ms response time *Generalized ESD or Grubbs & Distance Function
- 49. #DDSAIS16 Github https://bit.ly/2ISI3BS#DDSAIS16 Building the Continuous Trend Detector 1. Read from Kafka (raw events) 2. Look up *historic statistics from Elastic Search or Redis 3. Generate an anomaly stream based off of distance to statistical norm 4. Emit to Kafka for downstream use 49 Building a Discovery Engine Anatomy of the application: Streaming Aggregation Query
- 50. #DDSAIS16 Github https://bit.ly/2ISI3BS Discovering trends in massive data sets doesn't come for free. There are core principals and ideologies that must be followed in order to extract interesting patterns from your data. 50
- 51. #DDSAIS16 Github https://bit.ly/2ISI3BS Questions? / Thanks 51 @newfront github@newfront shaines@twilio.com Thanks to Marek Radonsky and Rohit Gupta from the Voice Insights team who helped make these patterns possible.