SlideShare une entreprise Scribd logo
1  sur  42
Télécharger pour lire hors ligne
Recruiting SolutionsRecruiting SolutionsRecruiting Solutions
Voldemort : Prototype to Production
A Journey to 1M Operations/Sec
Voldemort Intro
●  Amazon Dynamo style NoSQL k-v store
○  get(k)
○  put(k,v)
○  getall(k1,k2,...)
○  delete(k)
●  Tunable Consistency
●  Highly Available
●  Automatic Partitioning
Voldemort Intro
●  Pluggable Storage
○  BDB-JE - Primary OLTP store
○  Read Only - Reliable serving layer for Hadoop datasets
○  MySQL - Good ‘ol MySQL without native replication
○  InMemory - Backed by Java ConcurrentHashMap
●  Clients
○  Native Java Client
○  REST Coordinator Service
●  Open source
●  More at project-voldemort.com
Agenda
○ High Level Overview
○  Usage At LinkedIn
○  Storage Layer
○  Cluster Expansion
Architecture
Coordinator
Service
Server 1
Server 2
Native Java
Client
get()
put()
getall()
Server 3
Server 4
bdb
bdb
bdb
bdb
“k1” p1 s1,s2
“k2” p2 s3,s4
“k3” p1 s1,s2
“k4” p2 s3,s4
Client
Service
Client
Service
Client
Service
Client
Service
Client
Service
Consistent Hashing
▪  Consistent Hashing Idea
▪  Divide key space into partitions
–  Partitions: A,B,C,…,H
–  hash(key) mod # partitions = pkey
▪  Randomly map partitions to servers
▪  Locate servers from keys
–  K1 => A => S1,
–  K2 => C => S3
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K
1
K
2
Voldemort Intro
Consistent Hashing with Replication
▪  Replication factor (RF)
–  how many replicas to have
▪  Replica selection
–  Find the primary partition
–  Walk the ring to create preference list
▪  Find RF-1 additional servers
▪  Skip servers already in list
▪  Examples: RF = 3
–  K1: S1, S2, S3
–  K2: S3,S1,S4
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K1 [S1, S2,
S3]
K2
[S3,S1,S
4]
Voldemort Intro
Zone Aware Replication
▪  Servers divided into zones
▪  Zone = Data Center
▪  Per zone replication factor
▪  Local zone vs. remote zones
–  Local zone (LZ) is where client is
▪  Two zones example:
–  LZ = 1
–  Zone1: S1 S3; RF=2
–  Zone2: S2 S4; RF=1
–  Preference lists:
▪  K1: Z1: S1, S3; Z2: S2
▪  K2: Z1: S3, S1; Z2: S4
A
B
C
DE
F
G
H
S
1
S
1
S
2
S
3
S
3
S
4
S
2
S
4
K1 [ Z1 [S1, S3], Z2
[S2] ]
K2 [ Z1 [S3, S1], Z2 [S4] ]
Voldemort Intro
Voldemort @ LinkedIn
385
Stores
238 R-
O Stores
147 R-
W Stores
3
Zones
14
Clusters
~200
TB
~750
Servers
Voldemort @ LinkedIn
~1M
Storage
ops/s
22%
R-O
78%
R-W
Voldemort @ LinkedIn
●  17% of all LinkedIn Services
○  embed a direct client
●  Fast (95th percentile < 20ms) for almost all clients
Voldemort @ LinkedIn
●  Front Facing
○  Search (Recruiter + Site)
○  People You May Know
○  inShare
○  Media thumbnails
○  Notifications
○  Endorsements
○  Skills
○  Frequency capping Ads
○  Custom Segments
○  Who Viewed Your Profile
○  People You Want to Hire
●  Internal Services
○  Email cache
○  Email delivery stack
○  Recommendation Services
○  Personalization Services
○  Mobile Auth
●  Not exhaustive!
Growth Since 2011
●  Berkeley DB Java Edition
○  Embedded
○  100% Java
○  ACID compliant
○  Log structured
●  Voldemort uses
○  Vanilla k-v apis
○  Cursors for scans
Storage Layer
Storage Layer Rewrite
Where We Wanted To Be
●  Predictable online performance
●  Scan jobs
○  Non Intrusive, Fast
●  Elastic
○  Recover failed nodes in minutes
○  Add hardware overnight
Storage Layer Rewrite
Where We Really Were
1.  GC Issues
a.  Unpredictable GC Churn
b.  Scan jobs cause Full GCs
2.  Slow Scans (even on SSDs)
a.  Daily Retention Job/Slop Pusher
b.  Not Partition Aware
3.  Memory Management
a.  0-Control over a single store’s share
4.  Managing Multiple Versions
a.  Lock Contention
b.  Additional bdb-delete() cost during put()
5.  Weaker Durability on Crash
a.  Dirty Writes in heap
Storage Layer Rewrite
BDB Cache
on JVM
Disk
Index
Index
IndexIndexIndex
Index
Index
...
... ...
Leaf Index Leaf Index Leaf Leaf
Server Thread
BDB-Checkpointer
BDB-Cleaner
BDB-JE
Storage Layer Rewrite
JVM Heap BDB Cache
Store A’s B+Tree
Store D’s B+Tree
Store C’s B+Tree
Store B’s B+Tree
Server Threads Cleaner-A Checkpointer-A
Cleaner-A
Cleaner-A
Cleaner-A
Checkpointer-A
Checkpointer-A
Checkpointer-A
Multi-Tenant Example
Storage Layer Rewrite
Road To Recovery
●  Move data off heap
○  Only Index sits on heap
●  Cache Control to reduce scan impact
●  Partition Aware Storage
○  Range scans to the rescue
●  Dynamic Cache Partitioning
○  Control how much heap goes to a single store
●  SSD Aware Optimizations
○  Checkpointing
○  Cache Policy
●  Manage versions directly
○  Treat BDB as plain k-v store
Storage Layer Rewrite
Moving Data Off Heap
●  Much improved
GC
○  memory churn
○  promotions
●  SSD Aware hit-
the-disk design
●  Strong
Durability on
Crash
○  Runaway heap
SSD/Page
Cache
Index
put(k,v)
Leafold Leafnew
1
2
JVM Heap
Storage Layer Rewrite
Reducing Scan Impact
●  Massive Cache
Pollution
○  Throttling not an option
●  Exercise cursor level
control
●  Sustained rates upto
30-40K/sec
Storage Layer Rewrite
Managing Versions Directly
●  No more extra delete()
●  No more separate duplicate tree
○  Much improved locking performance
●  More compact storage
BIN
DIN
DBIN
V1 V2
BIN
V1,V2
Storage Layer Rewrite
SSD Aware Optimizations
●  Checkpoints on SSD
○  Age-old recovery time vs performance
tradeoff
●  Predictability
○  Level based policy
●  Streaming Writes
○  Turn off checkpointer
●  BDB5 Support
○  Much better compaction
○  Much less index metadata
Checkpointer Interval vs Recovery Time
Storage Layer Rewrite
Partition Aware Storage
“Key” “Key”Partition-id
Root
Subtree
k5
k6
k7
k8
k1
k2
k3
k4
Subtree
k9
k10
k11
k12
k13
k14
k15
k16
Root
P1 SubtreeP0 Subtree
k1
k3
k5
k7
k9
k11
k13
k15
k2
k4
k6
k8
k10
k12
k14
k16
Storage Layer Rewrite
Speed Up
Percentage Of Partitions Scanned
●  Restore
○  1 Day -> 1 hour ●  Rebalancing
○  ~Week -> Hours
Storage Layer Rewrite
Dynamic Cache Partitioning
●  Control share of heap per store
○  Dynamically add/reduce memory
○  Currently isolating bursty store
●  Improve Capacity Model
○  More production validation?
○  Auto tuning mechanisms?
●  Isolate at the JVM level?
○  Rethink deployment model
Storage Layer Rewrite
Wins In Production
Rein in GC
Storage Latency Way Down
Cluster Expansion Rewrite
●  Basis of scale-out philosophy
●  Cluster Expansion
○  Add servers to existing cluster
●  0 Downtime operation
●  Transparent to client
○  Functionality
○  Mostly Performance too
Cluster Expansion Rewrite
Types Of Clusters
●  Zoned Read Write
○  Zone = DataCenter
●  Non Zoned
○  Read-Write
○  Read-Only (Hadoop BuildAndPush)
Zone 1 Zone 2
Zone 1
Zone 2
Server 1 Server 1Server 2 Server 2
Server 2 Server 1Server 3 Server 2 Server 3Server 1
Expansion Example
P1
S2
P3
S4
S1
P2
S3
P4
P1
S2
P3
S4
S1
P2
S3
P4
P1S2
P3S4
S1P2
S3P4
P1S2
S4
P2
S3P4
S1
P3
Server 4
New New
Server 4
NewNew
Expansion In Action 1: Change Cluster
Topology
Cluster Expansion Rewrite
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
1
Zone 1 Zone 2
Expansion In Action 2: Setup Proxy Bridges
Cluster Expansion Rewrite
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Proxy Bridge
1
2
1. Change cluster
topology
Zone 1 Zone 2
Expansion In Action 3: Client Picks Up New
Topology
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Proxy Bridge
1
2
1. Change cluster
topology
2. Proxy request based
on old topology
3
Zone 1
Zone 2
Expansion In Action 4: Move Partitions
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Local
Move
Proxy Bridge
Client
Cross DC Move
1
3
4
41. Change cluster
topology
2. Proxy request based
on old topology
3. Client picks up
change
2
Zone 1
Zone 2
Expansion In Action
Cluster Expansion Rewrite
Client
Server 1
New
Server
Server 2
New
Server
Server 2Server 1
Rebalance
Controller
Local
Move
Proxy Bridge
Client
Cross DC Move
1
3
4
41. Change cluster
topology
2. Client picks up
change
3. Proxy request based
on old topology
4. Move partitions
2
Zone 1
Zone 2
Problems
Cluster Expansion Rewrite
●  One Ring Spanning Data Centers
○  Cross datacenter data moves/proxies
●  Not Safely Abortable
○  Additional cleanup/consolidation
●  Cannot Add New Data Centers
●  Opaque Planner Code
○  No special treatment of Zones
●  Lack of tools
○  Skew Analysis
○  Repartitioning/Balancing Utility
Zone 1 Zone 2
Server 1
Redesign: Zone N-ary Philosophy
P1
S1
Server 4
New New
Data Move
Old nth Replica of
P in Zone Z
New nth Replica of
P in Zone Z
Donor Stealer
Proxy Bridge
● Given a partition P, whose mapping has changed
Server 3Server 2 Server 1
P1
S1
Server 4Server 3Server 2
Redesign: Advantages
Cluster Expansion Rewrite
●  Simple, yet powerful
●  Feasible alternative to breaking the ring
○  Expensive to rewrite all of DR
●  No more cross datacenter moves
●  Aligns proxy bridges mechanism with planner logic
●  Principally applied
○  Abortable Rebalances
○  Zone Expansion
Abortable Rebalance
Cluster Expansion Rewrite
●  Plans go wrong
●  Introducing proxy puts
○  Safely rollback to old
topology
●  Avoid Data Loss &
adhoc repairs
●  Double write load
during rebalance
Stealer Donor
put(k,v) proxy-get(k)
vold
local-
put(k,vold)
local-
put(k,v)
Success proxy-put(k,v)
Zone Expansion
Cluster Expansion Rewrite
●  Builds upon Zone N-ary idea
●  Fetch data from an existing zone
●  No proxy bridges
○  No donors in same zone
●  Cannot read from new zone until complete
New Rebalance Utilities
Cluster Expansion Rewrite
•  PartitionAnalysis
○  Determine skewness of a cluster
●  Repartitioner
○  Improve partition balance
○  Greedy-Random swapping
●  RebalancePlanner
○  Incorporate Zone N-Ary logic
○  Operational Insights: storage overhead,probability client will pick up
new metadata
●  Rebalance Controller
○  Cleaner reimplementation based on new planner/scheduler
Wins In Production
Cluster Expansion Rewrite
•  7 Zoned RW Clusters expanded into new zone
○  Hiccups resolved overnight
○  Abortability is handy
●  Small Details -> Big Difference
○  Proxy Pause period
○  Accurate Progress reporting
○  Proxy get/getall optimization

Contenu connexe

Tendances

Power of the Log: LSM & Append Only Data Structures
Power of the Log: LSM & Append Only Data StructuresPower of the Log: LSM & Append Only Data Structures
Power of the Log: LSM & Append Only Data Structuresconfluent
 
Delta lake and the delta architecture
Delta lake and the delta architectureDelta lake and the delta architecture
Delta lake and the delta architectureAdam Doyle
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta LakeDatabricks
 
Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security Mydbops
 
In memory databases presentation
In memory databases presentationIn memory databases presentation
In memory databases presentationMichael Keane
 
LinkedIn Data Infrastructure (QCon London 2012)
LinkedIn Data Infrastructure (QCon London 2012)LinkedIn Data Infrastructure (QCon London 2012)
LinkedIn Data Infrastructure (QCon London 2012)Sid Anand
 
When Apache Spark Meets TiDB with Xiaoyu Ma
When Apache Spark Meets TiDB with Xiaoyu MaWhen Apache Spark Meets TiDB with Xiaoyu Ma
When Apache Spark Meets TiDB with Xiaoyu MaDatabricks
 
Introduction to Databases
Introduction to DatabasesIntroduction to Databases
Introduction to DatabasesRam Kedem
 
Accelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks AutoloaderAccelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks AutoloaderDatabricks
 
Spark with Delta Lake
Spark with Delta LakeSpark with Delta Lake
Spark with Delta LakeKnoldus Inc.
 
Cloud DW technology trends and considerations for enterprises to apply snowflake
Cloud DW technology trends and considerations for enterprises to apply snowflakeCloud DW technology trends and considerations for enterprises to apply snowflake
Cloud DW technology trends and considerations for enterprises to apply snowflakeSANG WON PARK
 
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Databricks
 
The Basics of MongoDB
The Basics of MongoDBThe Basics of MongoDB
The Basics of MongoDBvaluebound
 
Talend Big Data Capabilities Overview
Talend Big Data Capabilities OverviewTalend Big Data Capabilities Overview
Talend Big Data Capabilities OverviewRajan Kanitkar
 
Application Monitoring using Open Source: VictoriaMetrics - ClickHouse
Application Monitoring using Open Source: VictoriaMetrics - ClickHouseApplication Monitoring using Open Source: VictoriaMetrics - ClickHouse
Application Monitoring using Open Source: VictoriaMetrics - ClickHouseVictoriaMetrics
 
OLTP+OLAP=HTAP
 OLTP+OLAP=HTAP OLTP+OLAP=HTAP
OLTP+OLAP=HTAPEDB
 

Tendances (20)

Power of the Log: LSM & Append Only Data Structures
Power of the Log: LSM & Append Only Data StructuresPower of the Log: LSM & Append Only Data Structures
Power of the Log: LSM & Append Only Data Structures
 
Key-Value NoSQL Database
Key-Value NoSQL DatabaseKey-Value NoSQL Database
Key-Value NoSQL Database
 
Delta lake and the delta architecture
Delta lake and the delta architectureDelta lake and the delta architecture
Delta lake and the delta architecture
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta Lake
 
MyRocks Deep Dive
MyRocks Deep DiveMyRocks Deep Dive
MyRocks Deep Dive
 
MongodB Internals
MongodB InternalsMongodB Internals
MongodB Internals
 
NoSQL
NoSQLNoSQL
NoSQL
 
Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security Achieving compliance With MongoDB Security
Achieving compliance With MongoDB Security
 
In memory databases presentation
In memory databases presentationIn memory databases presentation
In memory databases presentation
 
LinkedIn Data Infrastructure (QCon London 2012)
LinkedIn Data Infrastructure (QCon London 2012)LinkedIn Data Infrastructure (QCon London 2012)
LinkedIn Data Infrastructure (QCon London 2012)
 
When Apache Spark Meets TiDB with Xiaoyu Ma
When Apache Spark Meets TiDB with Xiaoyu MaWhen Apache Spark Meets TiDB with Xiaoyu Ma
When Apache Spark Meets TiDB with Xiaoyu Ma
 
Introduction to Databases
Introduction to DatabasesIntroduction to Databases
Introduction to Databases
 
Accelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks AutoloaderAccelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks Autoloader
 
Spark with Delta Lake
Spark with Delta LakeSpark with Delta Lake
Spark with Delta Lake
 
Cloud DW technology trends and considerations for enterprises to apply snowflake
Cloud DW technology trends and considerations for enterprises to apply snowflakeCloud DW technology trends and considerations for enterprises to apply snowflake
Cloud DW technology trends and considerations for enterprises to apply snowflake
 
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0
 
The Basics of MongoDB
The Basics of MongoDBThe Basics of MongoDB
The Basics of MongoDB
 
Talend Big Data Capabilities Overview
Talend Big Data Capabilities OverviewTalend Big Data Capabilities Overview
Talend Big Data Capabilities Overview
 
Application Monitoring using Open Source: VictoriaMetrics - ClickHouse
Application Monitoring using Open Source: VictoriaMetrics - ClickHouseApplication Monitoring using Open Source: VictoriaMetrics - ClickHouse
Application Monitoring using Open Source: VictoriaMetrics - ClickHouse
 
OLTP+OLAP=HTAP
 OLTP+OLAP=HTAP OLTP+OLAP=HTAP
OLTP+OLAP=HTAP
 

En vedette

Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesComposing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesVinoth Chandar
 
Hadoop Strata Talk - Uber, your hadoop has arrived
Hadoop Strata Talk - Uber, your hadoop has arrived Hadoop Strata Talk - Uber, your hadoop has arrived
Hadoop Strata Talk - Uber, your hadoop has arrived Vinoth Chandar
 
Introducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jIntroducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jInnova4j
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State DrivesVinoth Chandar
 
The inherent complexity of stream processing
The inherent complexity of stream processingThe inherent complexity of stream processing
The inherent complexity of stream processingnathanmarz
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016StampedeCon
 
Big data @ uber vu (1)
Big data @ uber vu (1)Big data @ uber vu (1)
Big data @ uber vu (1)Mihnea Giurgea
 
Node Architecture Implications for In-Memory Data Analytics on Scale-in Clusters
Node Architecture Implications for In-Memory Data Analytics on Scale-in ClustersNode Architecture Implications for In-Memory Data Analytics on Scale-in Clusters
Node Architecture Implications for In-Memory Data Analytics on Scale-in ClustersAhsan Javed Awan
 
Distributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingDistributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingCloudFundoo
 
Project Voldemort: Big data loading
Project Voldemort: Big data loadingProject Voldemort: Big data loading
Project Voldemort: Big data loadingDan Harvey
 
Story 06
Story 06Story 06
Story 06JooWan
 
Data Mining with R CH1 요약
Data Mining with R CH1 요약Data Mining with R CH1 요약
Data Mining with R CH1 요약Sung Yub Kim
 
Real time data ingestion and Hybrid Cloud
Real time data ingestion and Hybrid CloudReal time data ingestion and Hybrid Cloud
Real time data ingestion and Hybrid CloudNeeraj Sabharwal
 
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APIStreaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APICarol McDonald
 

En vedette (20)

Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesComposing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
 
Hadoop Strata Talk - Uber, your hadoop has arrived
Hadoop Strata Talk - Uber, your hadoop has arrived Hadoop Strata Talk - Uber, your hadoop has arrived
Hadoop Strata Talk - Uber, your hadoop has arrived
 
Introducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jIntroducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4j
 
Bluetube
BluetubeBluetube
Bluetube
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State Drives
 
Project Voldemort
Project VoldemortProject Voldemort
Project Voldemort
 
Project Voldemort
Project VoldemortProject Voldemort
Project Voldemort
 
The inherent complexity of stream processing
The inherent complexity of stream processingThe inherent complexity of stream processing
The inherent complexity of stream processing
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016
 
Big data @ uber vu (1)
Big data @ uber vu (1)Big data @ uber vu (1)
Big data @ uber vu (1)
 
Node Architecture Implications for In-Memory Data Analytics on Scale-in Clusters
Node Architecture Implications for In-Memory Data Analytics on Scale-in ClustersNode Architecture Implications for In-Memory Data Analytics on Scale-in Clusters
Node Architecture Implications for In-Memory Data Analytics on Scale-in Clusters
 
Distributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingDistributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent Hashing
 
Project Voldemort: Big data loading
Project Voldemort: Big data loadingProject Voldemort: Big data loading
Project Voldemort: Big data loading
 
ASPgems - kappa architecture
ASPgems - kappa architectureASPgems - kappa architecture
ASPgems - kappa architecture
 
Story 06
Story 06Story 06
Story 06
 
Data Mining with R CH1 요약
Data Mining with R CH1 요약Data Mining with R CH1 요약
Data Mining with R CH1 요약
 
Real time data ingestion and Hybrid Cloud
Real time data ingestion and Hybrid CloudReal time data ingestion and Hybrid Cloud
Real time data ingestion and Hybrid Cloud
 
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APIStreaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka API
 
UBER Data Mining
UBER Data MiningUBER Data Mining
UBER Data Mining
 

Similaire à Voldemort : Prototype to Production

Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQLPingCAP
 
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Rick Hwang
 
Build Dynamic DNS server from scratch in C (Part1)
Build Dynamic DNS server from scratch in C (Part1)Build Dynamic DNS server from scratch in C (Part1)
Build Dynamic DNS server from scratch in C (Part1)Yen-Kuan Wu
 
TiDB vs Aurora.pdf
TiDB vs Aurora.pdfTiDB vs Aurora.pdf
TiDB vs Aurora.pdfssuser3fb50b
 
5 levels of high availability from multi instance to hybrid cloud
5 levels of high availability  from multi instance to hybrid cloud5 levels of high availability  from multi instance to hybrid cloud
5 levels of high availability from multi instance to hybrid cloudRafał Leszko
 
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
5 Levels of High Availability: From Multi-instance to Hybrid Cloud5 Levels of High Availability: From Multi-instance to Hybrid Cloud
5 Levels of High Availability: From Multi-instance to Hybrid CloudRafał Leszko
 
Real-time analytics with Druid at Appsflyer
Real-time analytics with Druid at AppsflyerReal-time analytics with Druid at Appsflyer
Real-time analytics with Druid at AppsflyerMichael Spector
 
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022StreamNative
 
Netflix - Realtime Impression Store
Netflix - Realtime Impression Store Netflix - Realtime Impression Store
Netflix - Realtime Impression Store Nitin S
 
To Serverless and Beyond
To Serverless and BeyondTo Serverless and Beyond
To Serverless and BeyondScyllaDB
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDBPingCAP
 
Orchestrating Cassandra with Kubernetes: Challenges and Opportunities
Orchestrating Cassandra with Kubernetes: Challenges and OpportunitiesOrchestrating Cassandra with Kubernetes: Challenges and Opportunities
Orchestrating Cassandra with Kubernetes: Challenges and OpportunitiesRaghavendra Prabhu
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloudOVHcloud
 
Raft Engine Meetup 220702.pdf
Raft Engine Meetup 220702.pdfRaft Engine Meetup 220702.pdf
Raft Engine Meetup 220702.pdffengxun
 
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...Lucidworks
 
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017Alexey Grigorev
 
Large scale overlay networks with ovn: problems and solutions
Large scale overlay networks with ovn: problems and solutionsLarge scale overlay networks with ovn: problems and solutions
Large scale overlay networks with ovn: problems and solutionsHan Zhou
 

Similaire à Voldemort : Prototype to Production (20)

Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQL
 
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
 
Build Dynamic DNS server from scratch in C (Part1)
Build Dynamic DNS server from scratch in C (Part1)Build Dynamic DNS server from scratch in C (Part1)
Build Dynamic DNS server from scratch in C (Part1)
 
TiDB vs Aurora.pdf
TiDB vs Aurora.pdfTiDB vs Aurora.pdf
TiDB vs Aurora.pdf
 
5 levels of high availability from multi instance to hybrid cloud
5 levels of high availability  from multi instance to hybrid cloud5 levels of high availability  from multi instance to hybrid cloud
5 levels of high availability from multi instance to hybrid cloud
 
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
5 Levels of High Availability: From Multi-instance to Hybrid Cloud5 Levels of High Availability: From Multi-instance to Hybrid Cloud
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
 
Real-time analytics with Druid at Appsflyer
Real-time analytics with Druid at AppsflyerReal-time analytics with Druid at Appsflyer
Real-time analytics with Druid at Appsflyer
 
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
 
Netflix - Realtime Impression Store
Netflix - Realtime Impression Store Netflix - Realtime Impression Store
Netflix - Realtime Impression Store
 
Geode - Day 2
Geode - Day 2Geode - Day 2
Geode - Day 2
 
To Serverless and Beyond
To Serverless and BeyondTo Serverless and Beyond
To Serverless and Beyond
 
Welcome to icehouse
Welcome to icehouseWelcome to icehouse
Welcome to icehouse
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDB
 
Orchestrating Cassandra with Kubernetes: Challenges and Opportunities
Orchestrating Cassandra with Kubernetes: Challenges and OpportunitiesOrchestrating Cassandra with Kubernetes: Challenges and Opportunities
Orchestrating Cassandra with Kubernetes: Challenges and Opportunities
 
Running Cassandra in AWS
Running Cassandra in AWSRunning Cassandra in AWS
Running Cassandra in AWS
 
Logs @ OVHcloud
Logs @ OVHcloudLogs @ OVHcloud
Logs @ OVHcloud
 
Raft Engine Meetup 220702.pdf
Raft Engine Meetup 220702.pdfRaft Engine Meetup 220702.pdf
Raft Engine Meetup 220702.pdf
 
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...
Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...
 
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
Large Scale Vandalism Detection in Knowledge Bases: PyData Berlin 2017
 
Large scale overlay networks with ovn: problems and solutions
Large scale overlay networks with ovn: problems and solutionsLarge scale overlay networks with ovn: problems and solutions
Large scale overlay networks with ovn: problems and solutions
 

Dernier

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 

Dernier (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 

Voldemort : Prototype to Production

  • 1. Recruiting SolutionsRecruiting SolutionsRecruiting Solutions Voldemort : Prototype to Production A Journey to 1M Operations/Sec
  • 2. Voldemort Intro ●  Amazon Dynamo style NoSQL k-v store ○  get(k) ○  put(k,v) ○  getall(k1,k2,...) ○  delete(k) ●  Tunable Consistency ●  Highly Available ●  Automatic Partitioning
  • 3. Voldemort Intro ●  Pluggable Storage ○  BDB-JE - Primary OLTP store ○  Read Only - Reliable serving layer for Hadoop datasets ○  MySQL - Good ‘ol MySQL without native replication ○  InMemory - Backed by Java ConcurrentHashMap ●  Clients ○  Native Java Client ○  REST Coordinator Service ●  Open source ●  More at project-voldemort.com
  • 4. Agenda ○ High Level Overview ○  Usage At LinkedIn ○  Storage Layer ○  Cluster Expansion
  • 5. Architecture Coordinator Service Server 1 Server 2 Native Java Client get() put() getall() Server 3 Server 4 bdb bdb bdb bdb “k1” p1 s1,s2 “k2” p2 s3,s4 “k3” p1 s1,s2 “k4” p2 s3,s4 Client Service Client Service Client Service Client Service Client Service
  • 6. Consistent Hashing ▪  Consistent Hashing Idea ▪  Divide key space into partitions –  Partitions: A,B,C,…,H –  hash(key) mod # partitions = pkey ▪  Randomly map partitions to servers ▪  Locate servers from keys –  K1 => A => S1, –  K2 => C => S3 A B C DE F G H S 1 S 1 S 2 S 3 S 3 S 4 S 2 S 4 K 1 K 2 Voldemort Intro
  • 7. Consistent Hashing with Replication ▪  Replication factor (RF) –  how many replicas to have ▪  Replica selection –  Find the primary partition –  Walk the ring to create preference list ▪  Find RF-1 additional servers ▪  Skip servers already in list ▪  Examples: RF = 3 –  K1: S1, S2, S3 –  K2: S3,S1,S4 A B C DE F G H S 1 S 1 S 2 S 3 S 3 S 4 S 2 S 4 K1 [S1, S2, S3] K2 [S3,S1,S 4] Voldemort Intro
  • 8. Zone Aware Replication ▪  Servers divided into zones ▪  Zone = Data Center ▪  Per zone replication factor ▪  Local zone vs. remote zones –  Local zone (LZ) is where client is ▪  Two zones example: –  LZ = 1 –  Zone1: S1 S3; RF=2 –  Zone2: S2 S4; RF=1 –  Preference lists: ▪  K1: Z1: S1, S3; Z2: S2 ▪  K2: Z1: S3, S1; Z2: S4 A B C DE F G H S 1 S 1 S 2 S 3 S 3 S 4 S 2 S 4 K1 [ Z1 [S1, S3], Z2 [S2] ] K2 [ Z1 [S3, S1], Z2 [S4] ] Voldemort Intro
  • 9. Voldemort @ LinkedIn 385 Stores 238 R- O Stores 147 R- W Stores 3 Zones 14 Clusters ~200 TB ~750 Servers
  • 11. Voldemort @ LinkedIn ●  17% of all LinkedIn Services ○  embed a direct client ●  Fast (95th percentile < 20ms) for almost all clients
  • 12. Voldemort @ LinkedIn ●  Front Facing ○  Search (Recruiter + Site) ○  People You May Know ○  inShare ○  Media thumbnails ○  Notifications ○  Endorsements ○  Skills ○  Frequency capping Ads ○  Custom Segments ○  Who Viewed Your Profile ○  People You Want to Hire ●  Internal Services ○  Email cache ○  Email delivery stack ○  Recommendation Services ○  Personalization Services ○  Mobile Auth ●  Not exhaustive!
  • 14. ●  Berkeley DB Java Edition ○  Embedded ○  100% Java ○  ACID compliant ○  Log structured ●  Voldemort uses ○  Vanilla k-v apis ○  Cursors for scans Storage Layer
  • 15. Storage Layer Rewrite Where We Wanted To Be ●  Predictable online performance ●  Scan jobs ○  Non Intrusive, Fast ●  Elastic ○  Recover failed nodes in minutes ○  Add hardware overnight
  • 16. Storage Layer Rewrite Where We Really Were 1.  GC Issues a.  Unpredictable GC Churn b.  Scan jobs cause Full GCs 2.  Slow Scans (even on SSDs) a.  Daily Retention Job/Slop Pusher b.  Not Partition Aware 3.  Memory Management a.  0-Control over a single store’s share 4.  Managing Multiple Versions a.  Lock Contention b.  Additional bdb-delete() cost during put() 5.  Weaker Durability on Crash a.  Dirty Writes in heap
  • 17. Storage Layer Rewrite BDB Cache on JVM Disk Index Index IndexIndexIndex Index Index ... ... ... Leaf Index Leaf Index Leaf Leaf Server Thread BDB-Checkpointer BDB-Cleaner BDB-JE
  • 18. Storage Layer Rewrite JVM Heap BDB Cache Store A’s B+Tree Store D’s B+Tree Store C’s B+Tree Store B’s B+Tree Server Threads Cleaner-A Checkpointer-A Cleaner-A Cleaner-A Cleaner-A Checkpointer-A Checkpointer-A Checkpointer-A Multi-Tenant Example
  • 19. Storage Layer Rewrite Road To Recovery ●  Move data off heap ○  Only Index sits on heap ●  Cache Control to reduce scan impact ●  Partition Aware Storage ○  Range scans to the rescue ●  Dynamic Cache Partitioning ○  Control how much heap goes to a single store ●  SSD Aware Optimizations ○  Checkpointing ○  Cache Policy ●  Manage versions directly ○  Treat BDB as plain k-v store
  • 20. Storage Layer Rewrite Moving Data Off Heap ●  Much improved GC ○  memory churn ○  promotions ●  SSD Aware hit- the-disk design ●  Strong Durability on Crash ○  Runaway heap SSD/Page Cache Index put(k,v) Leafold Leafnew 1 2 JVM Heap
  • 21. Storage Layer Rewrite Reducing Scan Impact ●  Massive Cache Pollution ○  Throttling not an option ●  Exercise cursor level control ●  Sustained rates upto 30-40K/sec
  • 22. Storage Layer Rewrite Managing Versions Directly ●  No more extra delete() ●  No more separate duplicate tree ○  Much improved locking performance ●  More compact storage BIN DIN DBIN V1 V2 BIN V1,V2
  • 23. Storage Layer Rewrite SSD Aware Optimizations ●  Checkpoints on SSD ○  Age-old recovery time vs performance tradeoff ●  Predictability ○  Level based policy ●  Streaming Writes ○  Turn off checkpointer ●  BDB5 Support ○  Much better compaction ○  Much less index metadata Checkpointer Interval vs Recovery Time
  • 24. Storage Layer Rewrite Partition Aware Storage “Key” “Key”Partition-id Root Subtree k5 k6 k7 k8 k1 k2 k3 k4 Subtree k9 k10 k11 k12 k13 k14 k15 k16 Root P1 SubtreeP0 Subtree k1 k3 k5 k7 k9 k11 k13 k15 k2 k4 k6 k8 k10 k12 k14 k16
  • 25. Storage Layer Rewrite Speed Up Percentage Of Partitions Scanned ●  Restore ○  1 Day -> 1 hour ●  Rebalancing ○  ~Week -> Hours
  • 26. Storage Layer Rewrite Dynamic Cache Partitioning ●  Control share of heap per store ○  Dynamically add/reduce memory ○  Currently isolating bursty store ●  Improve Capacity Model ○  More production validation? ○  Auto tuning mechanisms? ●  Isolate at the JVM level? ○  Rethink deployment model
  • 27. Storage Layer Rewrite Wins In Production Rein in GC Storage Latency Way Down
  • 28. Cluster Expansion Rewrite ●  Basis of scale-out philosophy ●  Cluster Expansion ○  Add servers to existing cluster ●  0 Downtime operation ●  Transparent to client ○  Functionality ○  Mostly Performance too
  • 29. Cluster Expansion Rewrite Types Of Clusters ●  Zoned Read Write ○  Zone = DataCenter ●  Non Zoned ○  Read-Write ○  Read-Only (Hadoop BuildAndPush)
  • 30. Zone 1 Zone 2 Zone 1 Zone 2 Server 1 Server 1Server 2 Server 2 Server 2 Server 1Server 3 Server 2 Server 3Server 1 Expansion Example P1 S2 P3 S4 S1 P2 S3 P4 P1 S2 P3 S4 S1 P2 S3 P4 P1S2 P3S4 S1P2 S3P4 P1S2 S4 P2 S3P4 S1 P3 Server 4 New New Server 4 NewNew
  • 31. Expansion In Action 1: Change Cluster Topology Cluster Expansion Rewrite Server 1 New Server Server 2 New Server Server 2Server 1 Rebalance Controller 1 Zone 1 Zone 2
  • 32. Expansion In Action 2: Setup Proxy Bridges Cluster Expansion Rewrite Server 1 New Server Server 2 New Server Server 2Server 1 Rebalance Controller Proxy Bridge 1 2 1. Change cluster topology Zone 1 Zone 2
  • 33. Expansion In Action 3: Client Picks Up New Topology Cluster Expansion Rewrite Client Server 1 New Server Server 2 New Server Server 2Server 1 Rebalance Controller Proxy Bridge 1 2 1. Change cluster topology 2. Proxy request based on old topology 3 Zone 1 Zone 2
  • 34. Expansion In Action 4: Move Partitions Cluster Expansion Rewrite Client Server 1 New Server Server 2 New Server Server 2Server 1 Rebalance Controller Local Move Proxy Bridge Client Cross DC Move 1 3 4 41. Change cluster topology 2. Proxy request based on old topology 3. Client picks up change 2 Zone 1 Zone 2
  • 35. Expansion In Action Cluster Expansion Rewrite Client Server 1 New Server Server 2 New Server Server 2Server 1 Rebalance Controller Local Move Proxy Bridge Client Cross DC Move 1 3 4 41. Change cluster topology 2. Client picks up change 3. Proxy request based on old topology 4. Move partitions 2 Zone 1 Zone 2
  • 36. Problems Cluster Expansion Rewrite ●  One Ring Spanning Data Centers ○  Cross datacenter data moves/proxies ●  Not Safely Abortable ○  Additional cleanup/consolidation ●  Cannot Add New Data Centers ●  Opaque Planner Code ○  No special treatment of Zones ●  Lack of tools ○  Skew Analysis ○  Repartitioning/Balancing Utility
  • 37. Zone 1 Zone 2 Server 1 Redesign: Zone N-ary Philosophy P1 S1 Server 4 New New Data Move Old nth Replica of P in Zone Z New nth Replica of P in Zone Z Donor Stealer Proxy Bridge ● Given a partition P, whose mapping has changed Server 3Server 2 Server 1 P1 S1 Server 4Server 3Server 2
  • 38. Redesign: Advantages Cluster Expansion Rewrite ●  Simple, yet powerful ●  Feasible alternative to breaking the ring ○  Expensive to rewrite all of DR ●  No more cross datacenter moves ●  Aligns proxy bridges mechanism with planner logic ●  Principally applied ○  Abortable Rebalances ○  Zone Expansion
  • 39. Abortable Rebalance Cluster Expansion Rewrite ●  Plans go wrong ●  Introducing proxy puts ○  Safely rollback to old topology ●  Avoid Data Loss & adhoc repairs ●  Double write load during rebalance Stealer Donor put(k,v) proxy-get(k) vold local- put(k,vold) local- put(k,v) Success proxy-put(k,v)
  • 40. Zone Expansion Cluster Expansion Rewrite ●  Builds upon Zone N-ary idea ●  Fetch data from an existing zone ●  No proxy bridges ○  No donors in same zone ●  Cannot read from new zone until complete
  • 41. New Rebalance Utilities Cluster Expansion Rewrite •  PartitionAnalysis ○  Determine skewness of a cluster ●  Repartitioner ○  Improve partition balance ○  Greedy-Random swapping ●  RebalancePlanner ○  Incorporate Zone N-Ary logic ○  Operational Insights: storage overhead,probability client will pick up new metadata ●  Rebalance Controller ○  Cleaner reimplementation based on new planner/scheduler
  • 42. Wins In Production Cluster Expansion Rewrite •  7 Zoned RW Clusters expanded into new zone ○  Hiccups resolved overnight ○  Abortability is handy ●  Small Details -> Big Difference ○  Proxy Pause period ○  Accurate Progress reporting ○  Proxy get/getall optimization