Kafka is evolving to remove its dependency on Zookeeper. The Kafka Improvement Proposal 500 (KIP-500) aims to manage Kafka's metadata log with a self-managed Raft consensus algorithm and controller quorum rather than relying on Zookeeper. This will improve scalability, robustness, and make deployment easier. It will take multiple releases to fully implement KIP-500, beginning with removing Zookeeper from clients and ending with a release where Zookeeper is no longer required.
2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
kafka-zookeeper/
3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com
4. 2
● Kafka has gotten its mileage out of Zookeeper
● But it is still a second system
● KIP-500 has been adopted by the community
● This is not a 1-1 replacement
● We’ve been headed this direction for years
Introduction
86. 8484
Metadata as an
Event Log
- Each change becomes a
message
- Changes are propagated
to all brokers
...
924 Create topic ”foo”
925 Delete topic “bar”
926 Add node 4 to the cluster
927 Create topic “baz”
928 Alter ISR for “foo-0”
929 Add node 5 to the cluster
87. 8585
Metadata as an
Event Log
- Clear ordering
- Can send deltas
- Offset tracks consumer
position
- Easy to measure lag
...
924 Create topic ”foo”
925 Delete topic “bar”
926 Add node 4 to the cluster
927 Create topic “baz”
928 Alter ISR for “foo-0”
929 Add node 5 to the cluster
91. 89
Can we use the existing Kafka log
replication protocol?
- How do we elect the leader?
We need a self-managed quorum.
Implementing
the Controller
Log
92. 90
Can we use the existing Kafka log
replication protocol?
- How do we elect the leader?
We need a self-managed quorum.
Implementing
the Controller
Log
Enter Raft.
Leader election
is by simple
majority.
93. 91
Kafka Raft
Writes Single Leader Single Leader
Fencing Monotonically increasing
epoch
Monotonically increasing
term
Log reconciliation Offset and epoch Term and index
Push/Pull Pull Push
Commit Semantics ISR Majority
Leader Election From ISR through
Zookeeper
Majority
99. 97
Broker Registration
- Brokers send
heartbeats to the active
controller
- The controller uses this
to build a map of the
cluster
Controller
100. 98
Controller
Broker Registration
- Brokers send
heartbeats to the active
controller
- The controller uses this
to build a map of the
cluster
- The controller also tells
brokers if they should
be fenced or shut down
101. 99
Controller
Fencing
- Brokers need to be
fenced if they’re
partitioned from the
controller, or can’t keep
up
- Brokers self-fence if
they can’t talk to the
controller
123. 121
Apache ZooKeeper has served us well
- KIP-500 is not a 1:1 replacement, but a different
paradigm
We have already started removing ZK from clients
- Consumer, AdminClient
- Improved encapsulation, security, upgradability
124. 122
Metadata should be managed as a log
- Deltas, ordering, caching
- Controller Failover, Fencing
- Improved scalability, robustness, easier deployment
The metadata log must be self-managed
- Raft
- Controller quorum
125. 123
It will take a few releases to implement KIP-500
- Additional KIPs for APIs, Raft, Metadata, etc.
Rolling upgrades will be supported
- Bridge release
- Post-ZK release
Kafka needs no Keeper