Fast real-time approximations using Spark streaming
By Kevin Schmidt (Head of Data Science at Mind Candy) Luis Vicente (Senior Data Engineer at Mind Candy) For mobile games, constant tweaks are the difference between success and failure. Data and analytics have to be available in real-time, but calculating, for example, uniqueness or newness of a data point requires a list of seen data points - both memory intensive and tricky when using real-time stream processing like Spark Streaming. Probabilistic data structures allow approximation of these properties with a fixed memory representation, and are very well suited for this kind of stream processing. Getting from the theory of approximation to a useful metric at a low error rate even for many millions of users is another story. In our talk we will look at practical ways of achieving this: which approximation we used for selection of useful metrics, why we picked a specific probabilistic data structure, how we stored it in Cassandra as a time series and how we implemented it in Spark Streaming.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Fast real-time approximations using Spark streaming
Similaire à Fast real-time approximations using Spark streaming (20)
Plus de huguk
Plus de huguk (20)
Dernier
Dernier (20)
Fast real-time approximations using Spark streaming
- 1. Fast > Perfect Practical real-time approximations using Spark Streaming Kevin Schmidt @kevinschmidtbiz LuisVicente @lvicentesanchez
- 2. A Bit of Context: Mind Candy
- 3. A Bit of Context: Free To Play Sum ArbitraryValues Count Uniques It’s Complicated
- 4. A Bit of Context: Setup
- 5. A Bit of Context: Requirements • Constant storage space usage independent of number of users • Handle delayed or duplicate data • Error rate under 3%
- 6. Counting Users: Basics How To Count IDs Uniquely
- 7. Counting Users: HyperLogLog addIdentifier(value: String) merge(other: HyperLogLog): HyperLogLog zero(): HyperLogLog countUniques(): Long HyperLogLog Error Rate = 1.6% Fixed Size = 4KB 14Bit Size: 12Bit Size: Error Rate = 0.9% Fixed Size = 16KB
- 12. Counting Users: Some Scala https://github.com/lvicentesanchez/fast-gt-perfect
- 15. Counting Users: Result • Constant storage size usage for one day of data using 14bit HyperLogLogs: 288 * 16KB = 4608KB • HyperLogLogs count users only once even if data is duplicated or repeated • Time bucketing ensures delayed data is counted correctly • Difference of <1% between HyperLogLogs and real count
- 16. Counting Revenue: Basics How To Sum ArbitraryValues
- 17. Counting Revenue: BloomFilter BloomFilter Capacity = 10k Error Rate = 1% Size = 11.7KB Configurable Size: addIdentifier(value: String) merge(other: BloomFilter): BloomFilter zero(): BloomFilter contains(): Boolean
- 21. Counting Revenue: Some Scala https://github.com/lvicentesanchez/fast-gt-perfect
- 23. Counting Revenue: Result • Constant storage size usage for one day of data using a 10k BloomFilter: 288 * 11.7KB = 3370KB • BloomFilter eliminates sales already counted • Time bucketing ensures delayed data is counted correctly and keeps BloomFilters small • Difference of <1% between approximated and real revenue
- 24. Trending: Basics How To Find the Top K
- 25. Trending: StreamSummary StreamSummary Configurable Size: addIdentifier(value: String) merge(other: SS): SS topK(k: Int): Seq[(String, Long)] Capacity = 400 Max Size = 21.9KB Metwally,Agrawal & Abbadi: Efficient Computation of Frequent and Top-k Elements in Data Streams (2005)
- 29. Trending: Result • Constant storage size usage for one day of data using a Top400 StreamSummary: 288 * 21.9KB = 6307KB • StreamSummary will not eliminate duplicates • Time bucketing ensures delayed data is counted correctly • Difference of <2% between StreamSummary trending items and the real trending items