4. (Don’t) store offsets in ZooKeeper
• Heavy write-load on ZooKeeper
• Especially an issue
– during 0.7 to 0.8 migration
– and before we switched to SSDs
• Non-ideal work-arounds
– Increase offset-commit intervals
– Filter commits if offsets have not moved
– Spread large offset commits over commit
interval
5. Offset management (ideals)
• Durable
• Support high write-load
• Consistent reads
• Atomic offset commits
• Fast commits/fetches
6. Store offsets in a replicated log
audit-consumer
PageViewEvent-0
240
audit-consumer
EmailBounceEvent-0
232
__consumer_offsets Next commit
Group
Offset
Partition
7. Store offsets in a replicated log
audit-consumer
PageViewEvent-0
240
audit-consumer
EmailBounceEvent-0
232
__consumer_offsets
audit-consumer
EmailBounceEvent-0
248
8. Store offsets in a replicated log
audit-consumer
PageViewEvent-0
240
audit-consumer
EmailBounceEvent-0
232
__consumer_offsets
audit-consumer
EmailBounceEvent-0
248
audit-consumer
PageViewEvent-0
323
9. Store offsets in a replicated log
audit-consumer
PageViewEvent-0
240
audit-consumer
EmailBounceEvent-0
232
__consumer_offsets
audit-consumer
EmailBounceEvent-0
248
audit-consumer
PageViewEvent-0
323
mirrormaker
ClickEvent-0
54543
29. Offset{Commit,Fetch} API
OffsetFetchRequest
o Group Id: String
o Partitions: List<Topic-partition>
OffsetFetchResponse
o Response map
§ Key è Topic-partition
§ Value è Partition-data
• Offset: Long
• Metadata: String
• Error code: Short
31. Offset{Commit,Fetch} API
KafkaConsumer<K, V> consumer = new KafkaConsumer<K, V>(properties);!
…!
TopicPartition partition1 = new TopicPartition("topic1", 0);!
TopicPartition partition1 = new TopicPartition("topic1", 1);!
!
consumer.subscribe(partition1, partition2);!
!
Map<TopicPartition, Long> offsets = new LinkedHashMap<TopicPartition,
Long>();!
offsets.put(partition1, 123L);!
offsets.put(partition2, 4320L);!
…!
// commit offsets!
consumer.commit(offsets, CommitType.SYNC);!
…!
// fetch offsets!
long committedOffset = consumer.committed(partition1);!
!
32. How to read the offsets topic
To read everything, use the console consumer!
./bin/kafka-console-consumer.sh --topic __consumer_offsets --
zookeeper localhost:2181 --formatter "kafka.server.OffsetManager
$OffsetsMessageFormatter" --consumer.config config/
consumer.properties!
(Must set exclude.internal.topics = false in consumer.properties)
!
To read a single partition, use the simple-
consumer-shell
./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets --
partition 12 --broker-list localhost:9092 --formatter
"kafka.server.OffsetManager$OffsetsMessageFormatter"!
34. How to migrate/roll-back
Migrate from ZooKeeper to Kafka:
• Config change
– offsets.storage=kafka
– dual.commit.enabled=true
• Rolling bounce
• Config change
– dual.commit.enabled=false
• Rolling bounce
35. How to migrate/roll-back
Migrate from Kafka to ZooKeeper:
• Config change
– dual.commit.enabled=true
• Rolling bounce
• Config change
– offsets.storage=zookeeper
– dual.commit.enabled=false
• Rolling bounce
36. Key metrics to monitor
• Consumer mbeans
– Kafka commit rate
– ZooKeeper commit rate (during migration)
• Broker mbeans
– Max-dirty ratio and other log cleaner metrics
– Offset cache size
– Group count
– {ConsumerMetadata, OffsetCommit, OffsetFetch}
request metrics
37. 0.8.3
• Support compression in compacted topics
(KAFKA-1734)
• Change offset commit “timestamp” to
mean retention period: KAFKA-1634
• Offset client