Intro

Hi, I'm Lucia Cerchie from Confluent, and in this video, we'll be discussing how consumer group IDs work, and their role in fault tolerance and work sharing.

What is a consumer?

First of all, what is a Kafka consumer? Kafka consumers read or consume data from Kafka producers, and do the work of reading event streams. They read events, or messages, from logs called topics. Topics are further split into partitions, which are append-only logs that store the messages. This enables each topic to be hosted and replicated across a number of brokers.

What is a consumer group?

A given consumer in the consumer group can read from multiple partitions, including multiple partitions housed in the same topic. Each partition can only be consumed by a single consumer in the group. Consumer groups play a key role in Kafka's compute layer, making it highly scalable.

Work Sharing

All right, so what role does a consumer group ID play in consumer group work sharing? Let's take a look at how consumers in a group share work. If there's only one partition, and you have multiple consumers in a group, then only one consumer can read from that partition at a time. If there are eight partitions, then each of those four consumers will then be assigned two partitions. Say there are nine partitions. That means the leftover partition will be sent to the first consumer in the group, so that one consumer reads from three partitions, and the rest of the consumers read from two partitions. In each case, the group leader, or the first consumer of the group, handles the distribution of work. This whole process is predicated on the presence of a group ID to unify the consumers. It's important to remember this while you're setting up your consumers.

Fault Tolerance

Now, what's the role of consumer group IDs in fault tolerance? Each consumer in the group sends heartbeat requests to the brokers at a set interval. If a consumer does not send heartbeat requests in time, the consumer is removed from the group, and a re-balance is triggered. How does a group ID play into re-balancing?

Rebalancing

Well, in either case, the broker's record of the associated offset of a message in a partition determines where the consumer will begin reading messages after a re-join. As long as the group ID remains the same, it can pick up exactly where it left off. Here, that's at message offset one, without any risk of data loss. Learn More If you enjoyed this video, see the link below for a blog post on the same topic. You can also head over to Confluent Developer to take a Kafka 101 course. Feel free to like and subscribe.