My Presentation in QCon SF 2015

1. STREAM
PROCESSING@
UBER
DANNY YUAN @ UBER

2. What is Uber

3. Transportation at your fingertips

5. Stream Data Allows Us To Feel The Pulse Of Cities

6. Marketplace Health

7. What’s Going on Now

8. What’s Happened?

9. Status Tracking

13. A Little Background

14. Uber’s Platform Is a Distributed State Machine
Rider States

15. Uber’s Platform Is a Distributed State Machine
Rider States Driver States

16. Applications can’t do everything

17. Instead, Applications Emit Events

18. Events Should Be Available In Seconds

19. Events Should Rarely Get Lost

20. Events Should Be Cheap And Scalable

22. Where are the challenges?

23. Many Dimensions
Dozens of fields per event

24. Granular Data

25. Granular Data

26. Granular Data
Over 10,000 hexagons in the city

27. Granular Data
7 vehicle types

28. Granular Data
1440 minutes in a day

29. Granular Data
13 driver states

30. Granular Data
300 cities

31. Granular Data
1 day of data: 300 x 10,000 x 7 x 1440 x 13 = 393 billion
possible combinations

32. Unknown Query Patterns
Any combination of dimensions

33. Variety of Aggregations
- Heatmap
- Top N
- Histogram
- count(), avg(), sum(), percent(), geo

34. Different Geo Aggregation

35. Large Data Volume
• Hundreds of thousands of events per
second, or billions of events per day
• At least dozens of fields in each event

36. Tight Schedule

37. Key: Generalization

38. Data Type
• Dimensional Temporal Spatial Data
Dimension Value
state driver_arrived
vehicle type uber X
timestamp 13244323342
lattitude 12.23
longitude 30.00

39. Data Query
• OLAP on single-table temporal-spatial data
SELECT
<agg
functions>,
<dimensions>
FROM
<data_source>
WHERE
<boolean
filter>
GROUP
BY
<dimensions>
HAVING
<boolean
filter>
ORDER
BY
<sorting
criterial>
LIMIT
<n>
DO
<post
aggregation>

40. Finding the Right Storage System

41. Minimum Requirements
• OLAP with geospatial and time series support
• Support large amount of data
• Sub-second response time
• Query of raw data

42. It can’t be a KV store

43. Challenges to KV Store
Pre-computing all keys is O(2n)
for both space
and time

44. It can’t be a relational database

45. Challenges to Relational DB
• Managing multiple indices is painful
• Scanning is not fast enough

46. A System That Supports
• Fast scan
• Arbitrary boolean queries
• Raw data
• Wide range of aggregations

47. Elasticsearch

48. Highly Efficient Inverted-Index For Boolean Query

49. Built-in Distributed Query

50. Fast Scan with Flexible Aggregations

51. Storage

52. Are We Done?

53. Transformation
e.g. (Lat, Long) -> (zipcode, hexagon)

54. Dynamic Pricing

55. Trend Prediction

56. Supply and Demand Distribution

57. Technically Speaking: Clustering & Pr(D, S, E)

58. New Use Cases —> New Requirements

59. Pre-aggregation

60. Joining Multiple Streams

61. Sessionization

62. Multi-Staged Processing

63. State Management

64. Apache Samza

65. Why Apache Samza?

66. DAG on Kafka

67. Excellent Integration with Kafka

68. Excellent Integration with Kafka

69. Built-in Checkpointing

70. Built-in State Management

71. Processing Storage

72. What If Storage Is Down?

73. What If Processing Takes Long?

74. Processing Storage

75. Are We Done?

78. Post Processing

79. Results Transformation and Smoothing

80. Scale of Post Processing
10,000 hexagons in a city

81. Scale of Post Processing
331 neighboring hexagons to look at

82. Scale of Post Processing
331 x 10,000 = 3.1 Million Hexagons to
Process for a Single Query

83. Scale of Post Processing
99%-ile Processing Time: 70ms

84. Post Processing
• Each processor is a pure function
• Processors can be composed by combinators

85. Post Processing
• Highly parallelized execution
• Pipelining

86. Post Processing
• Each processor is a pure function
• Processors can be composed by combinators
• Highly parallelized execution

87. Practical Considerations

88. Data Discovery

89. Elasticsearch Query Can Be Complex

90. /driverAcceptanceRate?
geo_dist(10,
[37,
22])&
time_range(2015-­‐02-­‐04,2015-­‐03-­‐06)&
aggregate(timeseries(7d))&
eq(msg.driverId,1)

91. Elasticsearch Query Can Be Optimized
• Pipelining
• Validation
• Throttling

92. Timeinseconds

93. Elasticsearch Can Be Replaced

94. Storage QueryProcessing

95. There’s one more thing

96. There are always patterns in streams

97. There is always need for quick exploration

98. How many drivers cancel a request 10 times in a
row within a 5-minute window?

99. Which riders request a pickup from 100 miles
apart within a half hour window?

101. Complex Event Processing
FROM
driver_canceled#window.time(10
min)
SELECT
clientUUID,
count(clientUUID)
as
cancelCount
GROUP
BY
clientUUID
HAVING
cancelCount
>
10
INSERT
INTO
hipchat(room);

102. Implementation Becomes Easy

103. Thank You!