2. Voldemort Intro
● Amazon Dynamo style NoSQL k-v store
○ get(k)
○ put(k,v)
○ getall(k1,k2,...)
○ delete(k)
● Tunable Consistency
● Highly Available
● Automatic Partitioning
3. Voldemort Intro
● Pluggable Storage
○ BDB-JE - Primary OLTP store
○ Read Only - Reliable serving layer for Hadoop datasets
○ MySQL - Good ‘ol MySQL without native replication
○ InMemory - Backed by Java ConcurrentHashMap
● Clients
○ Native Java Client
○ REST Coordinator Service
● Open source
● More at project-voldemort.com
5. Architecture
Coordinator
Service
Server 1
Server 2
Native Java
Client
get()
put()
getall()
Server 3
Server 4
bdb
bdb
bdb
bdb
“k1” p1 s1,s2
“k2” p2 s3,s4
“k3” p1 s1,s2
“k4” p2 s3,s4
Client
Service
Client
Service
Client
Service
Client
Service
Client
Service
6. Consistent Hashing
▪ Consistent Hashing Idea
▪ Divide key space into partitions
– Partitions: A,B,C,…,H
– hash(key) mod # partitions = pkey
▪ Randomly map partitions to servers
▪ Locate servers from keys
– K1 => A => S1,
– K2 => C => S3
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K
1
K
2
Voldemort Intro
7. Consistent Hashing with Replication
▪ Replication factor (RF)
– how many replicas to have
▪ Replica selection
– Find the primary partition
– Walk the ring to create preference list
▪ Find RF-1 additional servers
▪ Skip servers already in list
▪ Examples: RF = 3
– K1: S1, S2, S3
– K2: S3,S1,S4
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K1 [S1, S2,
S3]
K2
[S3,S1,S
4]
Voldemort Intro
8. Zone Aware Replication
▪ Servers divided into zones
▪ Zone = Data Center
▪ Per zone replication factor
▪ Local zone vs. remote zones
– Local zone (LZ) is where client is
▪ Two zones example:
– LZ = 1
– Zone1: S1 S3; RF=2
– Zone2: S2 S4; RF=1
– Preference lists:
▪ K1: Z1: S1, S3; Z2: S2
▪ K2: Z1: S3, S1; Z2: S4
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K1 [ Z1 [S1, S3], Z2
[S2] ]
K2 [ Z1 [S3, S1], Z2 [S4] ]
Voldemort Intro
11. Voldemort @ LinkedIn
● 17% of all LinkedIn Services
○ embed a direct client
● Fast (95th percentile < 20ms) for almost all clients
12. Voldemort @ LinkedIn
● Front Facing
○ Search (Recruiter + Site)
○ People You May Know
○ inShare
○ Media thumbnails
○ Notifications
○ Endorsements
○ Skills
○ Frequency capping Ads
○ Custom Segments
○ Who Viewed Your Profile
○ People You Want to Hire
● Internal Services
○ Email cache
○ Email delivery stack
○ Recommendation Services
○ Personalization Services
○ Mobile Auth
● Not exhaustive!
15. Storage Layer Rewrite
Where We Wanted To Be
● Predictable online performance
● Scan jobs
○ Non Intrusive, Fast
● Elastic
○ Recover failed nodes in minutes
○ Add hardware overnight
16. Storage Layer Rewrite
Where We Really Were
1. GC Issues
a. Unpredictable GC Churn
b. Scan jobs cause Full GCs
2. Slow Scans (even on SSDs)
a. Daily Retention Job/Slop Pusher
b. Not Partition Aware
3. Memory Management
a. 0-Control over a single store’s share
4. Managing Multiple Versions
a. Lock Contention
b. Additional bdb-delete() cost during put()
5. Weaker Durability on Crash
a. Dirty Writes in heap
17. Storage Layer Rewrite
BDB Cache
on JVM
Disk
Index
Index
IndexIndexIndex
Index
Index
...
... ...
Leaf Index Leaf Index Leaf Leaf
Server Thread
BDB-Checkpointer
BDB-Cleaner
BDB-JE
18. Storage Layer Rewrite
JVM Heap BDB Cache
Store A’s B+Tree
Store D’s B+Tree
Store C’s B+Tree
Store B’s B+Tree
Server Threads Cleaner-A Checkpointer-A
Cleaner-A
Cleaner-A
Cleaner-A
Checkpointer-A
Checkpointer-A
Checkpointer-A
Multi-Tenant Example
19. Storage Layer Rewrite
Road To Recovery
● Move data off heap
○ Only Index sits on heap
● Cache Control to reduce scan impact
● Partition Aware Storage
○ Range scans to the rescue
● Dynamic Cache Partitioning
○ Control how much heap goes to a single store
● SSD Aware Optimizations
○ Checkpointing
○ Cache Policy
● Manage versions directly
○ Treat BDB as plain k-v store
20. Storage Layer Rewrite
Moving Data Off Heap
● Much improved
GC
○ memory churn
○ promotions
● SSD Aware hit-
the-disk design
● Strong
Durability on
Crash
○ Runaway heap
SSD/Page
Cache
Index
put(k,v)
Leafold Leafnew
1
2
JVM Heap
21. Storage Layer Rewrite
Reducing Scan Impact
● Massive Cache
Pollution
○ Throttling not an option
● Exercise cursor level
control
● Sustained rates upto
30-40K/sec
22. Storage Layer Rewrite
Managing Versions Directly
● No more extra delete()
● No more separate duplicate tree
○ Much improved locking performance
● More compact storage
BIN
DIN
DBIN
V1 V2
BIN
V1,V2
23. Storage Layer Rewrite
SSD Aware Optimizations
● Checkpoints on SSD
○ Age-old recovery time vs performance
tradeoff
● Predictability
○ Level based policy
● Streaming Writes
○ Turn off checkpointer
● BDB5 Support
○ Much better compaction
○ Much less index metadata
Checkpointer Interval vs Recovery Time
25. Storage Layer Rewrite
Speed Up
Percentage Of Partitions Scanned
● Restore
○ 1 Day -> 1 hour ● Rebalancing
○ ~Week -> Hours
26. Storage Layer Rewrite
Dynamic Cache Partitioning
● Control share of heap per store
○ Dynamically add/reduce memory
○ Currently isolating bursty store
● Improve Capacity Model
○ More production validation?
○ Auto tuning mechanisms?
● Isolate at the JVM level?
○ Rethink deployment model
28. Cluster Expansion Rewrite
● Basis of scale-out philosophy
● Cluster Expansion
○ Add servers to existing cluster
● 0 Downtime operation
● Transparent to client
○ Functionality
○ Mostly Performance too
29. Cluster Expansion Rewrite
Types Of Clusters
● Zoned Read Write
○ Zone = DataCenter
● Non Zoned
○ Read-Write
○ Read-Only (Hadoop BuildAndPush)
30. Zone 1 Zone 2
Zone 1
Zone 2
Server 1 Server 1Server 2 Server 2
Server 2 Server 1Server 3 Server 2 Server 3Server 1
Expansion Example
P1
S2
P3
S4
S1
P2
S3
P4
P1
S2
P3
S4
S1
P2
S3
P4
P1S2
P3S4
S1P2
S3P4
P1S2
S4
P2
S3P4
S1
P3
Server 4
New New
Server 4
NewNew
31. Expansion In Action 1: Change Cluster
Topology
Cluster Expansion Rewrite
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
1
Zone 1 Zone 2
32. Expansion In Action 2: Setup Proxy Bridges
Cluster Expansion Rewrite
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Proxy Bridge
1
2
1. Change cluster
topology
Zone 1 Zone 2
33. Expansion In Action 3: Client Picks Up New
Topology
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Proxy Bridge
1
2
1. Change cluster
topology
2. Proxy request based
on old topology
3
Zone 1
Zone 2
34. Expansion In Action 4: Move Partitions
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Local
Move
Proxy Bridge
Client
Cross DC Move
1
3
4
41. Change cluster
topology
2. Proxy request based
on old topology
3. Client picks up
change
2
Zone 1
Zone 2
35. Expansion In Action
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Local
Move
Proxy Bridge
Client
Cross DC Move
1
3
4
41. Change cluster
topology
2. Client picks up
change
3. Proxy request based
on old topology
4. Move partitions
2
Zone 1
Zone 2
36. Problems
Cluster Expansion Rewrite
● One Ring Spanning Data Centers
○ Cross datacenter data moves/proxies
● Not Safely Abortable
○ Additional cleanup/consolidation
● Cannot Add New Data Centers
● Opaque Planner Code
○ No special treatment of Zones
● Lack of tools
○ Skew Analysis
○ Repartitioning/Balancing Utility
37. Zone 1 Zone 2
Server 1
Redesign: Zone N-ary Philosophy
P1
S1
Server 4
New New
Data Move
Old nth Replica of
P in Zone Z
New nth Replica of
P in Zone Z
Donor Stealer
Proxy Bridge
● Given a partition P, whose mapping has changed
Server 3Server 2 Server 1
P1
S1
Server 4Server 3Server 2
38. Redesign: Advantages
Cluster Expansion Rewrite
● Simple, yet powerful
● Feasible alternative to breaking the ring
○ Expensive to rewrite all of DR
● No more cross datacenter moves
● Aligns proxy bridges mechanism with planner logic
● Principally applied
○ Abortable Rebalances
○ Zone Expansion
39. Abortable Rebalance
Cluster Expansion Rewrite
● Plans go wrong
● Introducing proxy puts
○ Safely rollback to old
topology
● Avoid Data Loss &
adhoc repairs
● Double write load
during rebalance
Stealer Donor
put(k,v) proxy-get(k)
vold
local-
put(k,vold)
local-
put(k,v)
Success proxy-put(k,v)
40. Zone Expansion
Cluster Expansion Rewrite
● Builds upon Zone N-ary idea
● Fetch data from an existing zone
● No proxy bridges
○ No donors in same zone
● Cannot read from new zone until complete
41. New Rebalance Utilities
Cluster Expansion Rewrite
• PartitionAnalysis
○ Determine skewness of a cluster
● Repartitioner
○ Improve partition balance
○ Greedy-Random swapping
● RebalancePlanner
○ Incorporate Zone N-Ary logic
○ Operational Insights: storage overhead,probability client will pick up
new metadata
● Rebalance Controller
○ Cleaner reimplementation based on new planner/scheduler
42. Wins In Production
Cluster Expansion Rewrite
• 7 Zoned RW Clusters expanded into new zone
○ Hiccups resolved overnight
○ Abortability is handy
● Small Details -> Big Difference
○ Proxy Pause period
○ Accurate Progress reporting
○ Proxy get/getall optimization